# StarRocks Documentation > StarRocks is an open-source, high-performance OLAP database for real-time analytics at scale. It supports Standard SQL, materialized views, data lakes (Iceberg, Delta Lake, Hudi), stream ingestion (Kafka, Flink), and cloud-native deployment. This documentation covers SQL reference, table design, data loading, query acceleration, administration, and release notes. ## Administration ### Cluster Snapshot Beta feature [Advice on use of Beta features](https://docs.starrocks.io/docs/introduction/maturity.md) This topic describes how to use Cluster Snapshot for disaster recovery on shared-data clusters. This feature is supported from v3.4.2 onwards and only available on shared-data clusters. #### Overview[​](#overview "Direct link to Overview") The fundamental idea of disaster recovery for shared-data clusters is to ensure that the full cluster state (including data and metadata) is stored in object storage. This way, if the cluster encounters a failure, it can be restored from the object storage as long as the data and metadata remain intact. Additionally, features like backups and cross-region replication offered by cloud providers can be used to achieve remote recovery and cross-region disaster recovery. In shared-data clusters, the CN state (data) is stored in object storage, but the FE state (metadata) remains local. To ensure that object storage has all the cluster state for restoration, StarRocks now supports Cluster Snapshot for both data and metadata in object storage. ##### Workflow[​](#workflow "Direct link to Workflow") ![Workflow](/assets/images/cluster_snapshot_workflow-6a61b5e029595a53a7072a54bd5c6e4f.png) ##### Terms[​](#terms "Direct link to Terms") * **Cluster snapshot** A cluster snapshot refers to a snapshot of the cluster state at a certain moment. It contains all the objects in the cluster, such as catalogs, databases, tables, users & privileges, loading tasks, and more. It does not include all external dependent objects, such as configuration files of external catalogs, and local UDF JAR packages. * **Generating cluster snapshot** The system automatically maintains a snapshot closely following the latest cluster state. Historical snapshots will be dropped right after the latest one is created, keeping only one snapshot available all the time. * **Cluster Restore** Restore the cluster from a snapshot. #### Automated cluster snapshot[​](#automated-cluster-snapshot "Direct link to Automated cluster snapshot") Automated Cluster Snapshot is disabled by default. Use the following statement to enable this feature: Syntax: ```sql ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON [STORAGE VOLUME ] ``` Parameter: `storage_volume_name`: Specifies the storage volume used to store the snapshot. If this parameter is not specified, the default storage volume will be used. For details on creating a storage volume, see [CREATE STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md). Each time FE creates a new metadata image after completing a metadata checkpoint, it automatically creates a snapshot. The name of the snapshot is generated by the system, following the format `automated_cluster_snapshot_{timestamp}`. Metadata snapshots are stored under `/{storage_volume_locations}/{service_id}/meta/image/automated_cluster_snapshot_timestamp`. Data snapshots are stored in the same location as the original data. FE configuration item `automated_cluster_snapshot_interval_seconds` controls the snapshot automation cycle. The default value is 600 seconds (10 minutes). ##### Disable automated cluster snapshot[​](#disable-automated-cluster-snapshot "Direct link to Disable automated cluster snapshot") Use the following statement to disable automated cluster snapshot: ```sql ADMIN SET AUTOMATED CLUSTER SNAPSHOT OFF ``` Once Automated Cluster Snapshot is disabled, the system will automatically purge the historical snapshot. #### View cluster snapshot[​](#view-cluster-snapshot "Direct link to View cluster snapshot") You can query the view `information_schema.cluster_snapshots` to view the latest cluster snapshot and the snapshots yet to be dropped. ```sql SELECT * FROM information_schema.cluster_snapshots; ``` Return: | Field | Description | | -------------------- | ----------------------------------------------------------------- | | snapshot\_name | The name of the snapshot. | | snapshot\_type | The type of the snapshot. Valid values: `automated` and `manual`. | | created\_time | The time at which the snapshot was created. | | fe\_journal\_id | The ID of the FE journal. | | starmgr\_journal\_id | The ID of the StarManager journal. | | properties | Applies to a feature not yet available. | | storage\_volume | The storage volume where the snapshot is stored. | | storage\_path | The storage path under which the snapshot is stored. | #### View cluster snapshot job[​](#view-cluster-snapshot-job "Direct link to View cluster snapshot job") You can query the view `information_schema.cluster_snapshot_jobs` to view the job information of cluster snapshots. ```sql SELECT * FROM information_schema.cluster_snapshot_jobs; ``` Return: | Field | Description | | -------------- | ----------------------------------------------------------------------------------------------------------------- | | snapshot\_name | The name of the snapshot. | | job\_id | The ID of the job. | | created\_time | The time at which the job was created. | | finished\_time | The time at which the job was finished. | | state | The state of the job. Valid values: `INITIALIZING`, `SNAPSHOTING`, `FINISHED`, `EXPIRED`, `DELETED`, and `ERROR`. | | detail\_info | The specific progress information of the current execution stage. | | error\_message | The error message (if any) of the job. | #### Restore the cluster[​](#restore-the-cluster "Direct link to Restore the cluster") Follow these steps to restore the cluster with the cluster snapshot. 1. **(Optional)** If you want to perform a cross-cluster recovery within the region, you MUST modify the configuration file **cluster\_snapshot.yaml** under the directory `fe/conf` of the Leader FE node. Otherwise, you can skip this step. To restore the data in a new cluster, all files under the original storage path must be copied to the new path. You must use the template provided in [Appendix](#appendix). 2) Start the Leader FE node. ```bash ./fe/bin/start_fe.sh --cluster_snapshot --daemon ``` 3) Start other FE nodes **after cleaning the `meta` directories**. ```bash ./fe/bin/start_fe.sh --helper : --daemon ``` Add the nodes to the cluster. ```sql -- Add the Follower node: ALTER SYSTEM ADD FOLLOWER ":"; -- Add the Observer node: ALTER SYSTEM ADD OBSERVER ":"; ``` 4) Start CN nodes **after cleaning the `storage_root_path` directories**. ```bash ./be/bin/start_cn.sh --daemon ``` Add the node to the cluster. ```sql ALTER SYSTEM ADD COMPUTE NODE ":"; ``` If you have modified **cluster\_snapshot.yaml** in the step 1, the node and storage volumes will be re-configured in the new cluster according to the information in the file. #### Appendix[​](#appendix "Direct link to Appendix") Template of **cluster\_snapshot.yaml** for cross-cluster recovery: ```yaml # Information of the cluster snapshot to be downloaded for restoration. cluster_snapshot: # The URI of the snapshot. # Example 1: s3://defaultbucket/test/f7265e80-631c-44d3-a8ac-cf7cdc7adec811019/meta/image/automated_cluster_snapshot_1704038400000 # Example 2: s3://defaultbucket/test/f7265e80-631c-44d3-a8ac-cf7cdc7adec811019/meta cluster_snapshot_path: # The name of the storage volume to store the snapshot. You must define it in the `storage_volumes` section. # NOTE: It must be identical with that in the original cluster. storage_volume_name: my_s3_volume # [Optional] Node information of the new cluster where the snapshot is to be restored. # If this section is not specified, the new cluster after recovery only has the Leader FE node. # CN nodes retain the information of the original cluster. # NOTE: DO NOT include the Leader FE node in this section. frontends: # FE host. - host: xxx.xx.xx.x1 # FE edit_log_port. edit_log_port: 9010 # The FE node type. Valid values: `follower` (Default) and `observer`. type: follower - host: xxx.xx.xx.x2 edit_log_port: 9010 type: observer compute_nodes: # CN host. - host: xxx.xx.xx.x3 # CN heartbeat_service_port. heartbeat_service_port: 9050 - host: xxx.xx.xx.x4 heartbeat_service_port: 9050 # Information of the storage volume in the new cluster. It is used for restoring a cloned snapshot. # NOTE: The name of the storage volume must be identical with that in the original cluster. storage_volumes: # Example for S3-compatible storage volume. - name: my_s3_volume type: S3 location: s3://defaultbucket/test/ comment: my s3 volume properties: - key: aws.s3.region value: us-west-2 - key: aws.s3.endpoint value: https://s3.us-west-2.amazonaws.com - key: aws.s3.access_key value: xxxxxxxxxx - key: aws.s3.secret_key value: yyyyyyyyyy # Example for HDFS storage volume. - name: my_hdfs_volume type: HDFS location: hdfs://127.0.0.1:9000/sr/test/ comment: my hdfs volume properties: - key: hadoop.security.authentication value: simple - key: username value: starrocks ``` note For more information on credentials for AWS, see [Authenticate to AWS S3](https://docs.starrocks.io/docs/integrations/authenticate_to_aws_resources.md). #### Limitations[​](#limitations "Direct link to Limitations") * Currently, standby mode is not supported. The primary and secondary clusters cannot be online simultaneously. Otherwise, the normal operation of the secondary cluster cannot be guaranteed. * Currently, only one automated cluster snapshot can be retained. --- ### Cross-cluster Data Migration Tool The StarRocks Cross-cluster Data Migration Tool is provided by StarRocks Community. You can use this tool to easily migrate data from the source cluster to the target cluster. | Migration Path | Support information | | ------------------------------------- | ------------------------------ | | From Shared-nothing to Shared-nothing | From v3.1.8 and v3.2.3 onwards | | From Shared-nothing to Shared-data | From v3.1.8 and v3.2.3 onwards | | From Shared-data to Shared-data | From v4.1 onwards | | From Shared-data to Shared-nothing | Not supported | #### Preparations[​](#preparations "Direct link to Preparations") * Migrate from Shared-nothing * Migrate between Shared-data ##### On source cluster[​](#on-source-cluster "Direct link to On source cluster") You do not need to perform any preparations on the source cluster. ##### On target cluster[​](#on-target-cluster "Direct link to On target cluster") The following preparations must be performed on the target cluster for data migration. ###### Open ports[​](#open-ports "Direct link to Open ports") If you have enabled the firewall, you must open these ports: | **Component** | **Port** | **Default** | | ------------- | -------------- | ----------- | | FE | query\_port | 9030 | | FE | http\_port | 8030 | | FE | rpc\_port | 9020 | | BE/CN | be\_http\_port | 8040 | | BE/CN | be\_port | 9060 | ##### On source cluster[​](#on-source-cluster-1 "Direct link to On source cluster") During migration, the source cluster's Auto-Vacuum mechanism may delete historical data versions that the target CNs still need to read. To prevent this situation, you must extend the Auto-Vacuum grace period by dynamically setting the FE configuration item `lake_autovacuum_grace_period_minutes` to a significantly large value: ```sql ADMIN SET FRONTEND CONFIG("lake_autovacuum_grace_period_minutes"="10000000"); ``` important This setting prevents the source cluster from reclaiming stale object storage files during migration, which will cause storage amplification. It is recommended to keep the migration window as short as possible, and to reset this item to its default value `30` after migration. ```sql ADMIN SET FRONTEND CONFIG("lake_autovacuum_grace_period_minutes"="30"); ``` ##### On target cluster[​](#on-target-cluster-1 "Direct link to On target cluster") The following preparations must be performed on the target cluster for data migration. ###### Open ports[​](#open-ports-1 "Direct link to Open ports") If you have enabled the firewall, you must open these ports: | **Component** | **Port** | **Default** | | ------------- | ----------- | ----------- | | FE | query\_port | 9030 | | FE | http\_port | 8030 | | FE | rpc\_port | 9020 | ###### Disable Compaction[​](#disable-compaction "Direct link to Disable Compaction") You must disable Compaction on the target cluster during migration to prevent conflicts with incoming replication data. 1. Dynamically disable Compaction: ```sql ADMIN SET FRONTEND CONFIG("lake_compaction_max_tasks"="0"); ``` 2. To prevent Compaction from being re-enabled after a cluster restart, also add the following configuration to the FE configuration file **fe.conf**: ```properties lake_compaction_max_tasks = 0 ``` important After migration is complete, re-enable Compaction by removing the above configuration from **fe.conf**, and enable Compaction dynamically by executing: ```sql ADMIN SET FRONTEND CONFIG("lake_compaction_max_tasks"="-1"); ``` ###### Create source storage volumes on the target cluster[​](#create-source-storage-volumes-on-the-target-cluster "Direct link to Create source storage volumes on the target cluster") The Migration Tool identifies which storage volume each source table uses, and looks up a corresponding storage volume on the target cluster using the naming convention `src_`. You must pre-create these storage volumes before starting migration. important These storage volumes are used **only during migration** to give target CNs read access to the source cluster's object storage. After migration is complete, they are no longer needed and can be dropped. 1. On the **source cluster**, list all storage volumes: ```sql SHOW STORAGE VOLUMES; ``` 2. For each storage volume used by the tables you plan to migrate, describe it to get its configuration: ```sql DESCRIBE STORAGE VOLUME ; ``` Example output: ```text +---------------------+------+-----------+-------------------------------+-----------------------------+ | Name | Type | IsDefault | Location | Params | +---------------------+------+-----------+-------------------------------+-----------------------------+ | builtin_storage_vol | S3 | true | s3://my-bucket | {"aws.s3.region":"...",...} | +---------------------+------+-----------+-------------------------------+-----------------------------+ ``` 3. On the **target cluster**, create a mirrored storage volume using the same object storage credentials, but with the name prefixed by `src_`: ```sql CREATE STORAGE VOLUME src_ TYPE = S3 LOCATIONS = ("") PROPERTIES ( "enabled" = "true", "aws.s3.region" = "", "aws.s3.endpoint" = "", "aws.s3.use_aws_sdk_default_behavior" = "false", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.enable_partitioned_prefix" = "false" ); ``` note * Set `aws.s3.enable_partitioned_prefix` to `false` regardless of the source cluster's setting. The migration tool reads files using the source partition's full path directly, so partitioned prefix must not be applied to the mirrored volume. * Repeat this step for **each** unique storage volume used by the tables to be migrated. For example, if the source uses `builtin_storage_volume`, create `src_builtin_storage_volume` on the target cluster. * It is recommended to use temporary credentials (access key / secret key) for the source storage volume. These can be revoked after migration is complete. ###### Enable Legacy Compatibility for Replication[​](#enable-legacy-compatibility-for-replication "Direct link to Enable Legacy Compatibility for Replication") StarRocks may behave differently between the old and new versions, causing problems during cross-cluster data migration. Therefore, you must enable Legacy Compatibility for the target cluster before data migration and disable it after data migration is completed. 1. You can check whether Legacy Compatibility for Replication is enabled by using the following statement: ```sql ADMIN SHOW FRONTEND CONFIG LIKE 'enable_legacy_compatibility_for_replication'; ``` If `true` is returned, it indicates that Legacy Compatibility for Replication is enabled. 2. Dynamically enable Legacy Compatibility for Replication: ```sql ADMIN SET FRONTEND CONFIG("enable_legacy_compatibility_for_replication"="true"); ``` 3. To prevent Legacy Compatibility for Replication from automatically disabling during the data migration process in case of cluster restart, you also need to add the following configuration item in the FE configuration file **fe.conf**: ```properties enable_legacy_compatibility_for_replication = true ``` important After the data migration is completed, you need to remove the configuration `enable_legacy_compatibility_for_replication = true` from the configuration file, and dynamically disable Legacy Compatibility for Replication using the following statement: ```sql ADMIN SET FRONTEND CONFIG("enable_legacy_compatibility_for_replication"="false"); ``` ###### Configure Data Migration (Optional)[​](#configure-data-migration-optional "Direct link to Configure Data Migration (Optional)") You can configure data migration operations using the following FE and BE parameters. In most cases, the default configuration can meet your needs. If you wish to use the default configuration, you can skip this step. note Please note that increasing the values of the following configuration items can accelerate migration but will also increase the load pressure on the source cluster. ###### FE Parameters[​](#fe-parameters "Direct link to FE Parameters") The following FE parameters are dynamic configuration items. Refer to [Configure FE Dynamic Parameters](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#configure-fe-dynamic-parameters) on how to modify them. | **Parameter** | **Default** | **Unit** | **Description** | | ------------------------------------------ | ----------- | -------- | ------------------------------------------------------------------------------------------------------------------------------- | | replication\_max\_parallel\_table\_count | 100 | - | The maximum number of concurrent data synchronization tasks allowed. StarRocks creates one synchronization task for each table. | | replication\_max\_parallel\_replica\_count | 10240 | - | The maximum number of tablet replica allowed for concurrent synchronization. | | replication\_max\_parallel\_data\_size\_mb | 1048576 | MB | The maximum size of data allowed for concurrent synchronization. | | replication\_transaction\_timeout\_sec | 86400 | Seconds | The timeout duration for synchronization tasks. | ###### BE Parameters[​](#be-parameters "Direct link to BE Parameters") The following BE parameter is a dynamic configuration item. Refer to [Configure BE Dynamic Parameters](https://docs.starrocks.io/docs/administration/management/BE_configuration.md) on how to modify it. | **Parameter** | **Default** | **Unit** | **Description** | | -------------------- | ----------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | replication\_threads | 0 | - | The number of threads for executing synchronization tasks. `0` indicates setting the number of threads to the 4 times of number of CPU cores on the machine where the BE resides. | #### Step 1: Install the Tool[​](#step-1-install-the-tool "Direct link to Step 1: Install the Tool") It is recommended to install the migration tool on the server where the target cluster resides. 1. Launch a terminal, and download the binary package of the tool. ```bash wget https://releases.starrocks.io/starrocks/starrocks-cluster-sync.tar.gz ``` 2. Decompress the package. ```bash tar -xvzf starrocks-cluster-sync.tar.gz ``` #### Step 2: Configure the Tool[​](#step-2-configure-the-tool "Direct link to Step 2: Configure the Tool") ##### Migration-related configuration[​](#migration-related-configuration "Direct link to Migration-related configuration") Navigate to the extracted folder and modify the configuration file **conf/sync.properties**. ```bash cd starrocks-cluster-sync vi conf/sync.properties ``` The file content is as follows: ```properties # If true, all tables will be synchronized only once, and the program will exit automatically after completion. one_time_run_mode=false source_fe_host= source_fe_query_port=9030 source_cluster_user=root source_cluster_password= source_cluster_password_secret_key= # You can leave this empty or omit it if you want to migrate data between shared-data source clusters. source_cluster_token= target_fe_host= target_fe_query_port=9030 target_cluster_user=root target_cluster_password= target_cluster_password_secret_key= jdbc_connect_timeout_ms=30000 jdbc_socket_timeout_ms=60000 # Comma-separated list of database names or table names like or # example: db1,db2.tbl2,db3 # Effective order: 1. include 2. exclude include_data_list= exclude_data_list= # If there are no special requirements, please maintain the default values for the following configurations. target_cluster_storage_volume= # This configuration item is for migration between shared-data clusters only. target_cluster_use_builtin_storage_volume_only=false target_cluster_replication_num=-1 target_cluster_max_disk_used_percent=80 # To maintain consistency with the source cluster, use null. target_cluster_enable_persistent_index= max_replication_data_size_per_job_in_gb=1024 meta_job_interval_seconds=180 meta_job_threads=4 ddl_job_interval_seconds=10 ddl_job_batch_size=10 # table config ddl_job_allow_drop_target_only=false ddl_job_allow_drop_schema_change_table=true ddl_job_allow_drop_inconsistent_partition=true ddl_job_allow_drop_inconsistent_time_partition = true ddl_job_allow_drop_partition_target_only=true # index config enable_bitmap_index_sync=false ddl_job_allow_drop_inconsistent_bitmap_index=true ddl_job_allow_drop_bitmap_index_target_only=true # MV config enable_materialized_view_sync=false ddl_job_allow_drop_inconsistent_materialized_view=true ddl_job_allow_drop_materialized_view_target_only=false # View config enable_view_sync=false ddl_job_allow_drop_inconsistent_view=true ddl_job_allow_drop_view_target_only=false replication_job_interval_seconds=10 replication_job_batch_size=10 report_interval_seconds=300 enable_table_property_sync=false ``` The description of the parameters is as follows: | **Parameter** | **Description** | | ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | one\_time\_run\_mode | Whether to enable one-time synchronization mode. When one-time synchronization mode is enabled, the migration tool only performs full synchronization instead of incremental synchronization. | | source\_fe\_host | The IP address or FQDN (Fully Qualified Domain Name) of the source cluster's FE. | | source\_fe\_query\_port | The query port (`query_port`) of the source cluster's FE. | | source\_cluster\_user | The username used to log in to the source cluster. This user must be granted the OPERATE privilege on the SYSTEM level. | | source\_cluster\_password | The user password used to log in to the source cluster. | | source\_cluster\_password\_secret\_key | The secret key used to encrypt the password of the login user for the source cluster. The default value is an empty string, which means that the login password is not encrypted. If you want to encrypt `source_cluster_password`, you can get the encrypted `source_cluster_password` string by using SQL statement `SELECT TO_BASE64(AES_ENCRYPT('',''))`. | | source\_cluster\_token | Token of the source cluster. For information on how to obtain the cluster token, refer to [Obtain Cluster Token](#obtain-cluster-token) below.
**NOTE**
The Cluster Token is not required for migration between shared-data clusters because files are read directly from object storage. You can leave this empty or omit it if you want to migrate data between shared-data source clusters. | | target\_fe\_host | The IP address or FQDN (Fully Qualified Domain Name) of the target cluster's FE. | | target\_fe\_query\_port | The query port (`query_port`) of the target cluster's FE. | | target\_cluster\_user | The username used to log in to the target cluster. This user must be granted the OPERATE privilege on the SYSTEM level. | | target\_cluster\_password | The user password used to log in to the target cluster. | | target\_cluster\_password\_secret\_key | The secret key used to encrypt the password of the login user for the target cluster. The default value is an empty string, which means that the login password is not encrypted. If you want to encrypt `target_cluster_password`, you can get the encrypted `target_cluster_password` string by using SQL statement `SELECT TO_BASE64(AES_ENCRYPT('',''))`. | | jdbc\_connect\_timeout\_ms | JDBC connection timeout in milliseconds for FE queries. Default: `30000`. | | jdbc\_socket\_timeout\_ms | JDBC socket timeout in milliseconds for FE queries. Default: `60000`. | | include\_data\_list | The databases and tables that need to be migrated, with multiple objects separated by commas (`,`). For example: `db1, db2.tbl2, db3`. This item takes effect prior to `exclude_data_list`. If you want to migrate all databases and tables in the cluster, you do not need to configure this item. | | exclude\_data\_list | The databases and tables that do not need to be migrated, with multiple objects separated by commas (`,`). For example: `db1, db2.tbl2, db3`. `include_data_list` takes effect prior to this item. If you want to migrate all databases and tables in the cluster, you do not need to configure this item. | | target\_cluster\_storage\_volume | The storage volume used to store tables in the target cluster when the target cluster is a shared-data cluster. If you want to use the default storage volume, you do not need to specify this item. | | target\_cluster\_use\_builtin\_storage\_volume\_only | Whether to use `builtin_storage_volume` for migration in the target cluster. It is required for migration between shared-data clusters only. When this item is set to `true`, tables created in the target cluster will use `builtin_storage_volume` uniformly, instead of using the source cluster's `storage_volume` configuration. This is useful when the source cluster has multiple custom storage volumes but you want to consolidate all tables under one storage volume in the target cluster. | | target\_cluster\_replication\_num | The number of replicas specified when creating tables in the target cluster. If you want to use the same replica number as the source cluster, you do not need to specify this item. | | target\_cluster\_max\_disk\_used\_percent | Disk usage percentage threshold for BE nodes of the target cluster when the target cluster is shared-nothing. Migration is terminated when the disk usage of any BE in the target cluster exceeds this threshold. The default value is `80`, which means 80%. | | meta\_job\_interval\_seconds | The interval, in seconds, at which the migration tool retrieves metadata from the source and target clusters. You can use the default value for this item. | | meta\_job\_threads | The number of threads used by the migration tool to obtain metadata from the source and target clusters. You can use the default value for this item. | | ddl\_job\_interval\_seconds | The interval, in seconds, at which the migration tool executes DDL statements on the target cluster. You can use the default value for this item. | | ddl\_job\_batch\_size | The batch size for executing DDL statements on the target cluster. You can use the default value for this item. | | ddl\_job\_allow\_drop\_target\_only | Whether to allow the migration tool to delete databases or tables that exist only in the target cluster but not in the source cluster. The default is `false`, which means they will not be deleted. You can use the default value for this item. | | ddl\_job\_allow\_drop\_schema\_change\_table | Whether to allow the migration tool to delete tables with inconsistent schemas between the source and target clusters. The default is `true`, meaning they will be deleted. You can use the default value for this item. The migration tool will automatically synchronize the deleted tables during the migration. | | ddl\_job\_allow\_drop\_inconsistent\_partition | Whether to allow the migration tool to delete partitions with inconsistent data distribution between the source and target clusters. The default is `true`, meaning they will be deleted. You can use the default value for this item. The migration tool will automatically synchronize the deleted partitions during the migration. | | ddl\_job\_allow\_drop\_partition\_target\_only | Whether to allow the migration tool to delete partitions that are deleted in the source cluster to keep the partitions consistent between the source and target clusters. The default is `true`, meaning they will be deleted. You can use the default value for this item. | | replication\_job\_interval\_seconds | The interval, in seconds, at which the migration tool triggers data synchronization tasks. You can use the default value for this item. | | replication\_job\_batch\_size | The batch size at which the migration tool triggers data synchronization tasks. You can use the default value for this item. | | max\_replication\_data\_size\_per\_job\_in\_gb | The data size threshold at which the migration tool triggers data synchronization tasks. Unit: GB. Multiple data synchronization tasks will be triggered if the size of the partition to be migrated exceed this value. The default value is `1024`. You can use the default value for this item. | | report\_interval\_seconds | The time interval at which the migration tool prints the progress information. Unit: Seconds. Default value: `300`. You can use the default value for this item. | | target\_cluster\_enable\_persistent\_index | Whether to enable persistent index the in the target cluster. If this item is not specified, the target cluster is consistent with the source cluster.
**NOTE**
When migrating data between two shared-data clusters, the tool automatically converts `persistent_index_type = LOCAL` to `CLOUD_NATIVE` in the CREATE TABLE statement for Primary Key tables. No manual action is needed. | | ddl\_job\_allow\_drop\_inconsistent\_time\_partition | Whether to allow the migration tool to delete partitions with inconsistent time between the source and target clusters. The default is `true`, meaning they will be deleted. You can use the default value for this item. The migration tool will automatically synchronize the deleted partitions during the migration. | | enable\_bitmap\_index\_sync | Whether to enable synchronization for Bitmap indexes. | | ddl\_job\_allow\_drop\_inconsistent\_bitmap\_index | Whether to allow the migration tool to delete inconsistent Bitmap indexes between the source and target clusters. The default is `true`, meaning they will be deleted. You can use the default value for this item. The migration tool will automatically synchronize the deleted indexes during the migration. | | ddl\_job\_allow\_drop\_bitmap\_index\_target\_only | Whether to allow the migration tool to delete Bitmap indexes that are deleted in the source cluster to keep the indexes consistent between the source and target clusters. The default is `true`, meaning they will be deleted. You can use the default value for this item. | | enable\_materialized\_view\_sync | Whether to enable synchronization for materialized views. | | ddl\_job\_allow\_drop\_inconsistent\_materialized\_view | Whether to allow the migration tool to delete inconsistent materialized views between the source and target clusters. The default is `true`, meaning they will be deleted. You can use the default value for this item. The migration tool will automatically synchronize the deleted materialized views during the migration. | | ddl\_job\_allow\_drop\_materialized\_view\_target\_only | Whether to allow the migration tool to delete materialized views that are deleted in the source cluster to keep the materialized views consistent between the source and target clusters. The default is `true`, meaning they will be deleted. You can use the default value for this item. | | enable\_view\_sync | Whether to enable synchronization for views. | | ddl\_job\_allow\_drop\_inconsistent\_view | Whether to allow the migration tool to delete inconsistent views between the source and target clusters. The default is `true`, meaning they will be deleted. You can use the default value for this item. The migration tool will automatically synchronize the deleted views during the migration. | | ddl\_job\_allow\_drop\_view\_target\_only | Whether to allow the migration tool to delete views that are deleted in the source cluster to keep the views consistent between the source and target clusters. The default is `true`, meaning they will be deleted. You can use the default value for this item. | | enable\_table\_property\_sync | Whether to enable synchronization for table properties. | * Migrate from Shared-nothing * Migrate between Shared-data ##### Obtain Cluster Token[​](#obtain-cluster-token "Direct link to Obtain Cluster Token") note The Cluster Token is not required for migration between shared-data clusters. You can skip this step if you want to migrate data between shared-data source clusters. The Cluster Token is available in the FE metadata. Log in to the server where the FE node is located and run the following command: ```bash cat fe/meta/image/VERSION | grep token ``` Output: ```properties token=wwwwwwww-xxxx-yyyy-zzzz-uuuuuuuuuu ``` ##### Map storage volumes[​](#map-storage-volumes "Direct link to Map storage volumes") When the migration tool creates a table on the target cluster, it determines the table's storage volume as follows (in order of precedence): 1. If `target_cluster_use_builtin_storage_volume_only` is set to `true`, `builtin_storage_volume` is used for all tables. 2. If `target_cluster_storage_volume` is set to a specific storage volume, the specified storage volume is used for all tables. 3. Otherwise, by default, the source table's storage volume property is reserved. Tables from different source storage volumes are created under the corresponding storage volumes on the target cluster, provided those storage volumes exist on the target. Therefore, if you want to keep the storage volume property for each table in the source cluster, you can pre-create the same storage volumes in the target cluster. After migration, each table in the target cluster will inherit the storage volume property it had in the source cluster. Note that the `src_`-prefixed storage volumes serve a different purpose: they are used **only during migration** to give target CNs read access to the source cluster's object storage. After migration is complete, the `src_` volumes are no longer needed and can be dropped. ##### Network-related configuration (Optional)[​](#network-related-configuration-optional "Direct link to Network-related configuration (Optional)") * Migrate from Shared-nothing * Migrate between Shared-data During data migration, the migration tool needs to access **all** FE nodes of both the source and target clusters, and the target cluster needs to access **all** BE and CN nodes of the source cluster. You can obtain the network addresses of these nodes by executing the following statements on the corresponding cluster: ```sql -- Obtain the network addresses of FE nodes in a cluster. SHOW FRONTENDS; -- Obtain the network addresses of BE nodes in a cluster. SHOW BACKENDS; -- Obtain the network addresses of CN nodes in a cluster. SHOW COMPUTE NODES; ``` If these nodes use private addresses that cannot be accessed outside the cluster, such as internal network addresses within a Kubernetes cluster, you need to map these private addresses to addresses that can be accessed from outside. Navigate to the extracted folder of the tool and modify the configuration file **conf/hosts.properties**. ```bash cd starrocks-cluster-sync vi conf/hosts.properties ``` The default content of the file is as follows, describing how network address mapping is configured: ```properties # _=[;:[,:...]] ``` note The `` must match the address shown in the `IP` column returned by `SHOW FRONTENDS`, `SHOW BACKENDS`, or `SHOW COMPUTE NODES`. The following example performs these operations: 1. Map the source cluster's private network addresses `192.1.1.1` and `192.1.1.2` to `10.1.1.1` and `10.1.1.2`. 2. Map the source cluster's FE ports `8030` and `9030` to `38030` and `39030` on `10.1.1.1`. 3. Map the target cluster's private network address `fe-0.starrocks.svc.cluster.local` to `10.1.2.1` and remap port `9030`. ```properties # _=[;:[,:...]] SOURCE_192.1.1.1=10.1.1.1;8030:38030,9030:39030 SOURCE_192.1.1.2=10.1.1.2 TARGET_fe-0.starrocks.svc.cluster.local=10.1.2.1;9030:19030 ``` During data migration, the migration tool needs to access **all** FE nodes of both the source and target clusters. note Unlike migration from shared-nothing clusters, you do **not** need to configure network access from the target cluster to the source cluster's CN nodes, because data is transferred directly between object storage systems. You can obtain the FE network addresses by executing the following statement on the corresponding cluster: ```sql -- FE nodes SHOW FRONTENDS; ``` If FE nodes use private addresses that cannot be accessed outside the cluster, such as internal network addresses within a Kubernetes cluster, you need to map these private addresses to addresses that can be accessed from outside. Navigate to the extracted folder of the tool and modify the configuration file **conf/hosts.properties**. ```bash cd starrocks-cluster-sync vi conf/hosts.properties ``` The default content of the file is as follows, describing how network address mapping is configured: ```properties # _=[;:[,:...]] ``` note The `` must match the address shown in the `IP` column returned by `SHOW FRONTENDS`. The following example maps the target cluster's internal Kubernetes FQDN to a reachable IP: ```properties TARGET_frontend-0.frontend.mynamespace.svc.cluster.local=10.1.2.1;9030:19030 ``` #### Step 3: Start the Migration Tool[​](#step-3-start-the-migration-tool "Direct link to Step 3: Start the Migration Tool") After configuring the tool, start the migration tool to initiate the data migration process. ```bash ./bin/start.sh ``` note * If you are migrating data from a shared-nothing cluster, make sure that the BE nodes of the source and target clusters can properly communicate via the network. * During runtime, the migration tool regularly checks whether the data in the target cluster is lagging behind the source cluster. If there is a lag, it initiates data migration tasks. * If new data is constantly loaded into the source cluster, data synchronization will continue until the data in the target cluster is consistent with that in the source cluster. * You can query tables in the target cluster during migration, but do not load new data into the tables, as it may result in inconsistencies between the data in the target cluster and the source cluster. Currently, the migration tool does not forbid data loading into the target cluster during migration. * Note that data migration does not automatically terminate. You need to manually check and confirm the completion of migration and then stop the migration tool. #### View Migration Progress[​](#view-migration-progress "Direct link to View Migration Progress") ##### View Migration Tool logs[​](#view-migration-tool-logs "Direct link to View Migration Tool logs") You can check the migration progress through the migration tool log **log/sync.INFO.log**. Example 1: View task progress. ![img](/assets/images/data_migration_tool-1-73d324f48ca1372ae6a4c8c7499d6a15.png) The important metrics are as follows: * `Sync job progress`: The progress of data migration. The migration tool regularly checks whether the data in the target cluster is lagging behind the source cluster. Therefore, a progress of 100% only means that the data synchronization is completed within the current check interval. If new data continues to be loaded into the source cluster, the progress may decrease in the next check interval. * `total`: The total number of all types of jobs in this migration operation. * `ddlPending`: The number of DDL jobs pending to be executed. * `jobPending`: The number of pending data synchronization jobs to be executed. * `sent`: The number of data synchronization jobs sent from the source cluster but not yet started. Theoretically, this value should not be too large. If the value keeps increasing, please contact our engineers. * `running`: The number of data synchronization jobs that are currently running. * `finished`: The number of data synchronization jobs that are finished. * `failed`: The number of failed data synchronization jobs. Failed data synchronization jobs will be resent. Therefore, in most cases, you can ignore this metric. If this value is significantly large, please contact our engineers. * `unknown`: The number of jobs with an unknown status. Theoretically, this value should always be `0`. If this value is not `0`, please contact our engineers. Example 2: View the table migration progress. ![img](/assets/images/data_migration_tool-2-fc47ac5133c6a1618e5738f88f8694d4.png) * `Sync table progress`: Table migration progress, that is, the ratio of tables that have been migrated in this migration task to all the tables that need to be migrated. * `finishedTableRatio`: Ratio of tables with at least one successful synchronization task execution. * `expiredTableRatio`: Ratio of tables with expired data. * `total table`: Total number of tables involved in this data migration progress. * `finished table`: Number of tables with at least one successful synchronization task execution. * `unfinished table`: Number of tables with no synchronization task execution. * `expired table`: Number of tables with expired data. ##### View Migration Transaction Status[​](#view-migration-transaction-status "Direct link to View Migration Transaction Status") The migration tool opens a transaction for each table. You can view the status of the migration for a table by checking the status of its corresponding transaction. ```sql SHOW PROC "/transactions//running"; ``` `` is the name of the database where the table is located. ##### View Partition Data Versions[​](#view-partition-data-versions "Direct link to View Partition Data Versions") You can compare the data versions of the corresponding partitions in the source and target clusters to view the migration status of that partition. ```sql SHOW PARTITIONS FROM ; ``` `` is the name of the table to which the partition belongs. ##### View Data Volume[​](#view-data-volume "Direct link to View Data Volume") You can compare the data volumes in the source and target clusters to view the migration status. ```sql SHOW DATA; ``` ##### View Table Row Count[​](#view-table-row-count "Direct link to View Table Row Count") You can compare the row counts of tables in the source and target clusters to view the migration status of each table. ```sql SELECT TABLE_NAME, TABLE_ROWS FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_TYPE = 'BASE TABLE' ORDER BY TABLE_NAME; ``` #### After Migration[​](#after-migration "Direct link to After Migration") When `Sync job progress` has been stable at 100% and your business is ready to switch, complete the cutover as follows: 1. Stop writes to the source cluster. 2. Verify `Sync job progress` reaches and remains at 100% after writes stop. 3. Stop the migration tool. 4. Point your applications to the target cluster address. 5. If you migrated data between shared-data clusters, restore the Auto-Vacuum setting on the source cluster: ```sql ADMIN SET FRONTEND CONFIG("lake_autovacuum_grace_period_minutes"="30"); ``` 6. If you migrated data between shared-data clusters, re-enable Compaction on the target cluster. Remove `lake_compaction_max_tasks = 0` from **fe.conf** and execute: ```sql ADMIN SET FRONTEND CONFIG("lake_compaction_max_tasks"="-1"); ``` 7. Disable Legacy Compatibility for Replication on the target cluster. Remove `enable_legacy_compatibility_for_replication = true` from **fe.conf** and execute: ```sql ADMIN SET FRONTEND CONFIG("enable_legacy_compatibility_for_replication"="false"); ``` #### Limits[​](#limits "Direct link to Limits") The list of objects that support synchronization currently is as follows (those not included indicate that synchronization is not supported): * Databases * Internal tables and their data * Materialized view schemas and their building statements (The data in the materialized view will not be synchronized. And if the base tables of the materialized view is not synchronized to the target cluster, the background refresh task of the materialized view reports an error.) * Logical views For migration between shared-data clusters: * The target cluster must be running on v4.1 or later. * Migration from a shared-data cluster to a shared-nothing target is not supported. * Each storage volume used by the source cluster's tables must have a corresponding `src_` storage volume pre-created on the target cluster. --- ### Data Recovery Recover mistakenly deleted databases/tables/partitions. After `drop table` or `drop database`, StarRocks will not physically delete the data immediately, but keep it in Trash for a period of time (1 day by default). Administrators can recover the mistakenly deleted data with the `RECOVER` command. #### Related commands[​](#related-commands "Direct link to Related commands") Syntax: ```sql -- 1) Recover database RECOVER DATABASE db_name; -- 2) Recover table RECOVER TABLE [db_name.]table_name; -- 3) Recover partition RECOVER PARTITION partition_name FROM [db_name.]table_name; ``` #### Notes[​](#notes "Direct link to Notes") 1. This operation can only restore the deleted meta information. The default time is 1 day, which can be configured by the `catalog_trash_expire_second` parameter in `fe.conf`. 2. If new meta information of the same name and type is created after the meta information is deleted, the previously deleted meta information cannot be restored. #### Examples[​](#examples "Direct link to Examples") 1. Recover the database named `example_db` ```sql RECOVER DATABASE example_db; ``` 2. Recover the table named `example_tbl` ```sql RECOVER TABLE example_db.example_tbl; ``` 3. Recover the partition named `p1` in the table `example_tbl` ```sql RECOVER PARTITION p1 FROM example_tbl; ``` --- ### HTTP Interface To facilitate the maintenance of StarRocks clusters, StarRocks provides various types of operation and query interfaces. This topic introduces these HTTP interfaces and their usage. #### FE[​](#fe "Direct link to FE") | Request Method | Request Path | Description | | -------------- | -------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | PUT | `/api/{db}/{table}/_stream_load` | Stream Load operation, see [Stream Load](https://docs.starrocks.io/docs/loading/StreamLoad.md) for details. | | POST/PUT | `/api/transaction/{txn_op}` | Stream Load transaction interface, see [Stream Load Transaction Interface](https://docs.starrocks.io/docs/loading/Stream_Load_transaction_interface.md) for details. | | GET | `/api/{db}/_load_info` | | | GET | `/api/_set_config?config_key1=config_value1` | Update FE configuration. | | GET | `/api/_get_ddl?db={}&tbl={}` | View table DDL statement. | | GET | `/api/_migration?db={}&tbl={}` | View table tablet information. | | GET | `/api/_check_storagetype` | | | POST | `/api/{db}/{table}/_cancel?label={}` | | | GET | `/api/{db}/get_load_state` | | | GET | `/api/health` | | | GET | `/metrics?type={core/json}` | View metrics of the current FE. | | GET | `/api/show_meta_info` | | | GET | `/api/show_proc` | | | GET | `/api/show_runtime_info` | | | HEAD/GET | `/api/get_log_file` | | | GET | `/api/get_small_file` | | | GET | `/api/rowcount` | | | GET | `/api/check_decommission` | | | GET | `/api/_meta_replay_state` | | | POST | `/api/colocate/bucketseq` | | | GET | `/api/colocate` | | | POST | `/api/colocate/group_stable` | | | POST | `/api/colocate/group_unstable` | | | POST | `/api/colocate/update_group` | | | POST | `/api/global_dict/table/enable` | | | GET | `/api/profile?query_id={}` | Get profile information for the specified query ID. | | GET | `/api/query_detail` | Get query details. See [Query detail API](https://docs.starrocks.io/docs/administration/http_interface/query_detail.md). | | GET | `/api/connection` | | | GET | `/api/show_data?db={}` | Query the size of the specified database. | | POST | `/api/query_dump` | Get query dump information, see [Query Dump](https://docs.starrocks.io/docs/faq/Dump_query.md) for details. | | GET | `/api/stop` | | | GET | `/image` | | | GET | `/info` | | | GET | `/version` | | | GET | `/put` | | | GET | `/journal_id` | | | GET | `/check` | | | GET | `/dump` | | | GET | `/role` | | | GET | `/api/{db}/{table}/_count` | | | GET | `/api/{db}/{table}/_schema` | View table schema. | | GET/POST | `/api/{db}/{table}/_query_plan` | | #### BE[​](#be "Direct link to BE") | HTTP Request Method | HTTP Request Path | Description | | ------------------- | ------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | PUT | `/api/{db}/{table}/_stream_load` | Stream Load operation, see [Stream Load](https://docs.starrocks.io/docs/loading/StreamLoad.md) for details. | | POST/PUT | `/api/transaction/{txn_op}` | Stream Load transaction interface, see [Stream Load Transaction Interface](https://docs.starrocks.io/docs/loading/Stream_Load_transaction_interface.md) for details. | | PUT | `/api/transaction/load` | | | HEAD/GET | `/api/_download_load` | | | HEAD/GET | `/api/_tablet/_download` | | | HEAD/GET | `/api/_load_error_log` | | | GET | `/api/health` | | | GET | `/api/_stop_be` | | | GET | `/pprof/heap` | | | GET | `/pprof/growth` | | | GET | `/pprof/profile` | | | GET | `/pprof/pmuprofile` | | | GET | `/pprof/contention` | | | GET | `/pprof/cmdline` | | | HEAD/GET/POST | `/pprof/symbol` | | | GET | `/metrics` | View metrics of the current BE. | | HEAD | `/api/meta/header/{tablet_id}` | | | GET | `/api/checksum` | | | GET | `/api/reload_tablet` | | | POST | `/api/restore_tablet` | | | GET | `/api/snapshot` | | | GET | `/api/compaction/show?tablet_id={}` | View compaction information for the specified tablet. | | POST | `/api/compact?tablet_id={}&compaction_type={base/cumulative}` | Manually perform compaction on the specified tablet. | | GET | `/api/compaction/show_repair` | | | PUT | `/api/compaction/submit_repair` | | | POST | `/api/update_config` | Update BE configuration, see [Update BE Configuration](https://docs.starrocks.io/docs/administration/management/BE_configuration.md) for details. | | GET/PUT | `/api/runtime_filter_cache/{action}` | | | POST | `/api/compact_rocksdb_meta` | | | GET/PUT | `/api/query_cache/{action}` | | | GET | `/api/pipeline_blocking_drivers/{action}` | | | GET | `/greplog` | | | GET | `/varz` | View current BE configuration. | #### CN[​](#cn "Direct link to CN") | Request Method | Request Path | Description | | -------------- | ------------------- | ------------------------------- | | GET | `/api/health` | | | GET | `/pprof/heap` | | | GET | `/pprof/growth` | | | GET | `/pprof/profile` | | | GET | `/pprof/pmuprofile` | | | GET | `/pprof/contention` | | | GET | `/pprof/cmdline` | | | HEAD/GET/POST | `/pprof/symbol` | | | GET | `/metrics` | View metrics of the current CN. | --- ### Query detail API The **query detail** API returns recent query execution details that are cached in FE memory. note Query detail records are collected only when the FE configuration `enable_collect_query_detail_info` is set to `true`. #### Endpoints[​](#endpoints "Direct link to Endpoints") * `GET /api/query_detail` (v1) * `GET /api/v2/query_detail` (v2, supports `is_request_all_frontend` and returns a wrapped result) #### Parameters[​](#parameters "Direct link to Parameters") | Name | Required | Default | Description | | ------------------------- | ------------ | ------- | ------------------------------------------------------------------------------------------------------------------------- | | `event_time` | Yes | - | Lower bound filter. Returns items whose `eventTime` is greater than this value. You can pass `0` to get all cached items. | | `user` | No | - | Filter by the `user` field. Case-insensitive match. | | `is_request_all_frontend` | No (v2 only) | `false` | When `true`, the current FE queries other alive FEs and merges their results. | #### Response[​](#response "Direct link to Response") * v1 returns a JSON array of `QueryDetail` objects. * v2 returns `{ "code": "0", "message": "OK", "result": [ ... ] }` where `result` is a `QueryDetail` list. #### Authentication and authorization[​](#authentication-and-authorization "Direct link to Authentication and authorization") This API requires HTTP Basic authentication. There is no additional privilege check beyond a successful login. Any authenticated user can access the endpoint and can view all cached query details unless a `user` filter is applied. #### QueryDetail fields[​](#querydetail-fields "Direct link to QueryDetail fields") | Field | Type | Description | | ------------------- | ------- | --------------------------------------------------------------------------------------------------- | | `queryId` | string | Query ID. | | `eventTime` | long | Internal timestamp used for filtering. Monotonic nanosecond timestamp derived from wall-clock time. | | `isQuery` | boolean | Whether the statement is a query. | | `remoteIP` | string | Client IP address or `System`. | | `connId` | int | Connection ID. | | `startTime` | long | Query start time in milliseconds since epoch. | | `endTime` | long | Query end time in milliseconds since epoch. `-1` if not finished. | | `latency` | long | Query latency in milliseconds. `-1` if not finished. | | `pendingTime` | long | Pending time in milliseconds. | | `netTime` | long | Net execution time in milliseconds. | | `netComputeTime` | long | Net compute time in milliseconds. | | `state` | string | One of `RUNNING`, `FINISHED`, `FAILED`, `CANCELLED`. | | `database` | string | Current database. | | `sql` | string | SQL text (may be desensitized if configured). | | `user` | string | Login user (qualified user). | | `impersonatedUser` | string | Target user of `EXECUTE AS`. `null` if not executing as another user. | | `errorMessage` | string | Error message when failed. | | `explain` | string | Explain plan (level controlled by `query_detail_explain_level`). | | `profile` | string | Profile text if collected. | | `resourceGroupName` | string | Resource group name. | | `scanRows` | long | Scanned rows. | | `scanBytes` | long | Scanned bytes. | | `returnRows` | long | Returned rows. | | `cpuCostNs` | long | CPU cost in nanoseconds. | | `memCostBytes` | long | Memory cost in bytes. | | `spillBytes` | long | Spill bytes. | | `cacheMissRatio` | float | Cache miss ratio in percent (0-100). | | `warehouse` | string | Warehouse name. | | `digest` | string | SQL digest. | | `catalog` | string | Catalog name. | | `command` | string | MySQL command name. | | `preparedStmtId` | string | Prepared statement ID. | | `queryFeMemory` | long | FE memory allocated by the query, in bytes. | | `querySource` | string | Query source: `EXTERNAL`, `INTERNAL`, `MV`, or `TASK`. | #### Examples[​](#examples "Direct link to Examples") ##### v1[​](#v1 "Direct link to v1") ```bash curl -u root: "http://:/api/query_detail?event_time=0" ``` ##### v2[​](#v2 "Direct link to v2") ```bash curl -u root: "http://:/api/v2/query_detail?event_time=0&is_request_all_frontend=true" ``` --- ### Manage audit logs within StarRocks via AuditLoader This topic describes how to manage StarRocks audit logs within a table via the plugin - AuditLoader. StarRocks stores its audit logs in the local file **fe/log/fe.audit.log** rather than an internal database. The plugin AuditLoader allows you to manage audit logs directly within your cluster. Once installed, AuditLoader reads logs from the file, and loads them into StarRocks via HTTP PUT. You can then query the audit logs in StarRocks using SQL statements. #### Create a table to store audit logs[​](#create-a-table-to-store-audit-logs "Direct link to Create a table to store audit logs") Create a database and a table in your StarRocks cluster to store its audit logs. See [CREATE DATABASE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/CREATE_DATABASE.md) and [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) for detailed instructions. Because the fields of audit logs vary among different StarRocks versions, it is important to follow recommendations mentioned below to avoid compatibility issues during upgrade: > **CAUTION** > > * All new fields should be marked as `NULL`. > * Fields should NOT be renamed, as users may rely on them. > * Only backward compatible changes should be applied to field types, e.g. `VARCHAR(32)` -> `VARCHAR(64)`, to avoid errors during insert. > * `AuditEvent` fields are resolved by name only. The order of columns within table doesn't matter, and can be changed by user in any time. > * `AuditEvent` fields which doesn't exist in the table are ignored, so users can remove columns they don't need. ```sql CREATE DATABASE starrocks_audit_db__; CREATE TABLE starrocks_audit_db__.starrocks_audit_tbl__ ( `queryId` VARCHAR(64) COMMENT "Unique ID of the query", `timestamp` DATETIME NOT NULL COMMENT "Query start time", `queryType` VARCHAR(12) COMMENT "Query type (query, slow_query, connection)", `clientIp` VARCHAR(32) COMMENT "Client IP", `user` VARCHAR(64) COMMENT "Query username", `authorizedUser` VARCHAR(64) COMMENT "Unique identifier of the user, i.e., user_identity", `resourceGroup` VARCHAR(64) COMMENT "Resource group name", `catalog` VARCHAR(32) COMMENT "Catalog name", `db` VARCHAR(96) COMMENT "Database where the query runs", `state` VARCHAR(8) COMMENT "Query state (EOF, ERR, OK)", `errorCode` VARCHAR(512) COMMENT "Error code", `queryTime` BIGINT COMMENT "Query execution time (milliseconds)", `scanBytes` BIGINT COMMENT "Number of bytes scanned by the query", `scanRows` BIGINT COMMENT "Number of rows scanned by the query", `returnRows` BIGINT COMMENT "Number of rows returned by the query", `cpuCostNs` BIGINT COMMENT "CPU time consumed by the query (nanoseconds)", `memCostBytes` BIGINT COMMENT "Memory consumed by the query (bytes)", `stmtId` INT COMMENT "Incremental ID of the SQL statement", `isQuery` TINYINT COMMENT "Whether the SQL is a query (1 or 0)", `feIp` VARCHAR(128) COMMENT "FE IP that executed the statement", `stmt` VARCHAR(1048576) COMMENT "Original SQL statement", `digest` VARCHAR(32) COMMENT "Fingerprint of slow SQL", `planCpuCosts` DOUBLE COMMENT "CPU usage during query planning (nanoseconds)", `planMemCosts` DOUBLE COMMENT "Memory usage during query planning (bytes)", `pendingTimeMs` BIGINT COMMENT "Time the query waited in the queue (milliseconds)", `candidateMVs` VARCHAR(65533) NULL COMMENT "List of candidate materialized views", `hitMvs` VARCHAR(65533) NULL COMMENT "List of matched materialized views", `QueriedRelations` ARRAY NULL COMMENT "List of directly referenced tables and views", `warehouse` VARCHAR(32) NULL COMMENT "Warehouse name" ) ENGINE = OLAP DUPLICATE KEY (`queryId`, `timestamp`, `queryType`) COMMENT "Audit log table" PARTITION BY date_trunc('day', `timestamp`) PROPERTIES ( "replication_num" = "1", "partition_live_number" = "30" ); ``` `starrocks_audit_tbl__` is created with dynamic partitions. By default, the first dynamic partition is created 10 minutes after the table is created. Audit logs can then be loaded into the table. You can check the partitions in the table using the following statement: ```sql SHOW PARTITIONS FROM starrocks_audit_db__.starrocks_audit_tbl__; ``` After a partition is created, you can move on to the next step. #### Download and configure AuditLoader[​](#download-and-configure-auditloader "Direct link to Download and configure AuditLoader") 1. [Download](https://releases.starrocks.io/resources/auditloader.zip) the AuditLoader installation package. The package is compatible with all available versions of StarRocks. 2. Unzip the installation package. ```shell unzip auditloader.zip ``` The following files are inflated: * **auditloader.jar**: the JAR file of AuditLoader. * **plugin.properties**: the properties file of AuditLoader. You do not need to modify this file. * **plugin.conf**: the configuration file of AuditLoader. In most cases, you only need to modify the `user` and `password` fields in the file. 3. Modify **plugin.conf** to configure AuditLoader. You must configure the following items to make sure AuditLoader can work properly: * `frontend_host_port`: FE IP address and HTTP port, in the format `:`. It is recommended to set it to its default value `127.0.0.1:8030`. Each FE in StarRocks manages its own Audit Log independently, and after installing the plugin, each FE will start its own background thread to fetch and save Audit Logs, and write them via Stream Load. The `frontend_host_port` configuration item is used to provide the IP and port of the HTTP protocol for the background Stream Load task of the plug-in, and this parameter does not support multiple values. The IP part of the parameter can use the IP of any FE in the cluster, but it is not recommended because if the corresponding FE crashes, the audit log writing task in the background of other FEs will also fail due to the failure of communication. It is recommended to set it to the default value `127.0.0.1:8030`, so that each FE uses its own HTTP port to communicate, thus avoiding the impact on the communication in case of an exception of the other FEs (all the write tasks will be forwarded to the FE Leader node to be executed eventually). * `database`: name of the database you created to host audit logs. * `table`: name of the table you created to host audit logs. * `user`: your cluster username. You MUST have the privilege to load data (LOAD\_PRIV) into the table. * `password`: your user password. * `secret_key`: the key (string, must not be longer than 16 bytes) used to encrypt the password. If this parameter is not set, it indicates that the password in **plugin.conf** will not be encrypted, and you only need to specify the plaintext password in `password`. If this parameter is specified, it indicates that the password is encrypted by this key, and you need to specify the encrypted string in `password`. The encrypted password can be generated in StarRocks using the `AES_ENCRYPT` function: `SELECT TO_BASE64(AES_ENCRYPT('password','secret_key'));`. * `filter`: the filter conditions for audit log loading. This parameter is based on the [WHERE parameter](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md#opt_properties) in Stream Load, i.e. `-H “where: ”`, defaults to an empty string. Example: `filter=isQuery=1 and clientIp like '127.0.0.1%' and user='root'`. 4. Zip the files back into a package. ```shell zip -q -m -r auditloader.zip auditloader.jar plugin.conf plugin.properties ``` 5. Dispatch the package to all machines that host FE nodes. Make sure all packages are stored in an identical path. Otherwise, the installation fails. Remember to copy the absolute path to the package after you dispatched the package. > **NOTE** > > You can also distribute **auditloader.zip** to an HTTP service accessible to all FEs (for example, `httpd` or `nginx`) and install it using the network. Note that in both cases the **auditloader.zip** needs to be persisted in the path after the installation is performed, and the source files should not be deleted after installation. #### Install AuditLoader[​](#install-auditloader "Direct link to Install AuditLoader") Execute the following statement along with the path you copied to install AuditLoader as a plugin in StarRocks: ```sql INSTALL PLUGIN FROM ""; ``` Example of installation from a local package: ```sql INSTALL PLUGIN FROM ""; ``` If you want install the plugin via a network path, you need to provide the md5 of the package in the properties of the INSTALL statement. Example: ```sql INSTALL PLUGIN FROM "http://xx.xx.xxx.xxx/extra/auditloader.zip" PROPERTIES("md5sum" = "3975F7B880C9490FE95F42E2B2A28E2D"); ``` See [INSTALL PLUGIN](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plugin/INSTALL_PLUGIN.md) for detailed instructions. #### Verify the installation and query audit logs[​](#verify-the-installation-and-query-audit-logs "Direct link to Verify the installation and query audit logs") 1. You can check if the installation is successful via [SHOW PLUGINS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plugin/SHOW_PLUGINS.md). In the following example, the `Status` of the plugin `AuditLoader` is `INSTALLED`, meaning installation is successful. ```plain mysql> SHOW PLUGINS\G *************************** 1. row *************************** Name: __builtin_AuditLogBuilder Type: AUDIT Description: builtin audit logger Version: 0.12.0 JavaVersion: 1.8.31 ClassName: com.starrocks.qe.AuditLogBuilder SoName: NULL Sources: Builtin Status: INSTALLED Properties: {} *************************** 2. row *************************** Name: AuditLoader Type: AUDIT Description: Available for versions 3.3.11+. Load audit log to starrocks, and user can view the statistic of queries Version: 5.0.0 JavaVersion: 11 ClassName: com.starrocks.plugin.audit.AuditLoaderPlugin SoName: NULL Sources: /x/xx/xxx/xxxxx/auditloader.zip Status: INSTALLED Properties: {} 2 rows in set (0.01 sec) ``` 2. Execute some random SQLs to generate audit logs, and wait for 60 seconds (or the time you have specified in the item `max_batch_interval_sec` when you configure AuditLoader) to allow AuditLoader to load audit logs into StarRocks. 3. Check the audit logs by querying the table. ```sql SELECT * FROM starrocks_audit_db__.starrocks_audit_tbl__; ``` The following example shows when audit logs are loaded into the table successfully: ```plain mysql> SELECT * FROM starrocks_audit_db__.starrocks_audit_tbl__\G *************************** 1. row *************************** queryId: 01975a33-4129-7520-97a2-05e641cec6c9 timestamp: 2025-06-10 14:16:37 queryType: query clientIp: xxx.xx.xxx.xx:65283 user: root authorizedUser: 'root'@'%' resourceGroup: default_wg catalog: default_catalog db: state: EOF errorCode: queryTime: 3 scanBytes: 0 scanRows: 0 returnRows: 1 cpuCostNs: 33711 memCostBytes: 4200 stmtId: 102 isQuery: 1 feIp: xxx.xx.xxx.xx stmt: SELECT * FROM starrocks_audit_db__.starrocks_audit_tbl__ digest: planCpuCosts: 908 planMemCosts: 0 pendingTimeMs: -1 candidateMvs: null hitMVs: null ``` QueriedRelations: \["default\_catalog.db1.tbl1","default\_catalog.db1.view1"] ………… ````text ## Troubleshooting If no audit logs are loaded to the table after the dynamic partition is created and the plugin is installed, you can check whether **plugin.conf** is configured properly or not. To modify it, you must first uninstall the plugin: ```SQL UNINSTALL PLUGIN AuditLoader; ```` Logs of AuditLoader are printed in **fe.log**, you can retrieve them by searching the keyword `audit` in **fe.log**. After all configurations are set correctly, you can follow the above steps to install AuditLoader again. --- ### Back up and restore data This topic describes how to back up and restore data in StarRocks, or migrate data to a new StarRocks cluster. StarRocks supports backing up data as snapshots into a remote storage system and restoring the data to any StarRocks clusters. From v3.4.0 onwards, StarRocks have enhanced the functionality of BACKUP and RESTORE by supporting more objects and refactoring the syntax for better flexibility. StarRocks supports the following remote storage systems: * Apache™ Hadoop® (HDFS) cluster * AWS S3 * Google GCS * MinIO StarRocks supports backing up the following objects: * Internal databases, tables (of all types and partitioning strategies), and partitions * Metadata of external catalogs (supported from v3.4.0 onwards) * Synchronous materialized views and asynchronous materialized views * Logical views (supported from v3.4.0 onwards) * User-defined functions (supported from v3.4.0 onwards) > **NOTE** > > Shared-data StarRocks clusters do not support data BACKUP and RESTORE. #### Create a repository[​](#create-a-repository "Direct link to Create a repository") Before backing up data, you need to create a repository, which is used to store data snapshots in a remote storage system. You can create multiple repositories in a StarRocks cluster. For detailed instructions, see [CREATE REPOSITORY](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/CREATE_REPOSITORY.md). * Create a repository in HDFS The following example creates a repository named `test_repo` in an HDFS cluster. ```sql CREATE REPOSITORY test_repo WITH BROKER ON LOCATION "hdfs://:/repo_dir/backup" PROPERTIES( "username" = "", "password" = "" ); ``` * Create a repository in AWS S3 You can choose IAM user-based credential (Access Key and Secret Key), Instance Profile, or Assumed Role as the credential method for accessing AWS S3. * The following example creates a repository named `test_repo` in the AWS S3 bucket `bucket_s3` using IAM user-based credentials as the credential method. ```sql CREATE REPOSITORY test_repo WITH BROKER ON LOCATION "s3a://bucket_s3/backup" PROPERTIES( "aws.s3.access_key" = "XXXXXXXXXXXXXXXXX", "aws.s3.secret_key" = "yyyyyyyyyyyyyyyyyyyyyyyy", "aws.s3.region" = "us-east-1" ); ``` * The following example creates a repository named `test_repo` in the AWS S3 bucket `bucket_s3` using Instance Profile as the credential method. ```sql CREATE REPOSITORY test_repo WITH BROKER ON LOCATION "s3a://bucket_s3/backup" PROPERTIES( "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "us-east-1" ); ``` * The following example creates a repository named `test_repo` in the AWS S3 bucket `bucket_s3` using Assumed Role as the credential method. ```sql CREATE REPOSITORY test_repo WITH BROKER ON LOCATION "s3a://bucket_s3/backup" PROPERTIES( "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::xxxxxxxxxx:role/yyyyyyyy", "aws.s3.region" = "us-east-1" ); ``` > **NOTE** > > StarRocks supports creating repositories in AWS S3 only according to the S3A protocol. Therefore, when you create repositories in AWS S3, you must replace `s3://` in the S3 URI you pass as a repository location in `ON LOCATION` with `s3a://`. * Create a repository in Google GCS The following example creates a repository named `test_repo` in the Google GCS bucket `bucket_gcs`. ```sql CREATE REPOSITORY test_repo WITH BROKER ON LOCATION "s3a://bucket_gcs/backup" PROPERTIES( "fs.s3a.access.key" = "xxxxxxxxxxxxxxxxxxxx", "fs.s3a.secret.key" = "yyyyyyyyyyyyyyyyyyyy", "fs.s3a.endpoint" = "storage.googleapis.com" ); ``` > **NOTE** > > * StarRocks supports creating repositories in Google GCS only according to the S3A protocol. Therefore, when you create repositories in Google GCS, you must replace the prefix in the GCS URI you pass as a repository location in `ON LOCATION` with `s3a://`. > * Do not specify `https` in the endpoint address. * Create a repository in MinIO The following example creates a repository named `test_repo` in the MinIO bucket `bucket_minio`. ```sql CREATE REPOSITORY test_repo WITH BROKER ON LOCATION "s3://bucket_minio/backup" PROPERTIES( "aws.s3.access_key" = "XXXXXXXXXXXXXXXXX", "aws.s3.secret_key" = "yyyyyyyyyyyyyyyyy", "aws.s3.endpoint" = "http://minio:9000" ); ``` After the repository is created, you can check the repository via [SHOW REPOSITORIES](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/SHOW_REPOSITORIES.md). After restoring data, you can delete the repository in StarRocks using [DROP REPOSITORY](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/DROP_REPOSITORY.md). However, data snapshots backed up in the remote storage system cannot be deleted through StarRocks. You need to delete them manually in the remote storage system. #### Back up data[​](#back-up-data "Direct link to Back up data") After the repository is created, you need to create a data snapshot and back up it in the remote repository. For detailed instructions, see [BACKUP](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/BACKUP.md). BACKUP is an asynchronous operation. You can check the status of a BACKUP job using [SHOW BACKUP](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/SHOW_BACKUP.md), or cancel a BACKUP job using [CANCEL BACKUP](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/CANCEL_BACKUP.md). StarRocks supports FULL backup on the granularity level of database, table, or partition. If you have stored a large amount of data in a table, we recommend that you back up and restore data by partition. This way, you can reduce the cost of retries in case of job failures. If you need to back up incremental data on a regular basis, you can configure a [partitioning plan](https://docs.starrocks.io/docs/table_design/data_distribution.md#partitioning) for your table, and back up only new partitions each time. ##### Back up database[​](#back-up-database "Direct link to Back up database") Performing a full BACKUP on a database will back up all tables, synchronous and asynchronous materialized views, logical views, and UDFs within the database. The following examples back up the database `sr_hub` in the snapshot `sr_hub_backup` and upload the snapshot to the repository `test_repo`. ```sql -- Supported from v3.4.0 onwards. BACKUP DATABASE sr_hub SNAPSHOT sr_hub_backup TO test_repo; -- Compatible with the syntax in earlier versions. BACKUP SNAPSHOT sr_hub.sr_hub_backup TO test_repo; ``` ##### Back up table[​](#back-up-table "Direct link to Back up table") StarRocks supports backing up and restoring tables of all types and partitioning strategies. Performing a full BACKUP on a table will back up the table and the synchronous materialized views built on it. The following examples back up the table `sr_member` from the database `sr_hub` in the snapshot `sr_member_backup` and upload the snapshot to the repository `test_repo`. ```sql -- Supported from v3.4.0 onwards. BACKUP DATABASE sr_hub SNAPSHOT sr_member_backup TO test_repo ON (TABLE sr_member); -- Compatible with the syntax in earlier versions. BACKUP SNAPSHOT sr_hub.sr_member_backup TO test_repo ON (sr_member); ``` The following example backs up two tables, `sr_member` and `sr_pmc`, from the database `sr_hub` in the snapshot `sr_core_backup` and upload the snapshot to the repository `test_repo`. ```sql BACKUP DATABASE sr_hub SNAPSHOT sr_core_backup TO test_repo ON (TABLE sr_member, TABLE sr_pmc); ``` The following example backs up all tables from the database `sr_hub` in the snapshot `sr_all_backup` and upload the snapshot to the repository `test_repo`. ```sql BACKUP DATABASE sr_hub SNAPSHOT sr_all_backup TO test_repo ON (ALL TABLES); ``` ##### Back up partition[​](#back-up-partition "Direct link to Back up partition") The following examples back up the partition `p1` of the table `sr_member` from the database `sr_hub` in the snapshot `sr_par_backup` and upload the snapshot to the repository `test_repo`. ```sql -- Supported from v3.4.0 onwards. BACKUP DATABASE sr_hub SNAPSHOT sr_par_backup TO test_repo ON (TABLE sr_member PARTITION (p1)); -- Compatible with the syntax in earlier versions. BACKUP SNAPSHOT sr_hub.sr_par_backup TO test_repo ON (sr_member PARTITION (p1)); ``` You can specify multiple partition names separated by commas (`,`) to back up partitions in batch. ##### Back up materialized view[​](#back-up-materialized-view "Direct link to Back up materialized view") You do not need to manually back up synchronous materialized views because they will be backed up along with the BACKUP operation of the base table. Asynchronous materialized views can be backed up along with the BACKUP operation of the database it belongs to. You can also manually back up them. The following example backs up the materialized view `sr_mv1` from the database `sr_hub` in the snapshot `sr_mv1_backup` and upload the snapshot to the repository `test_repo`. ```sql BACKUP DATABASE sr_hub SNAPSHOT sr_mv1_backup TO test_repo ON (MATERIALIZED VIEW sr_mv1); ``` The following example backs up two materialized views, `sr_mv1` and `sr_mv2`, from the database `sr_hub` in the snapshot `sr_mv2_backup` and upload the snapshot to the repository `test_repo`. ```sql BACKUP DATABASE sr_hub SNAPSHOT sr_mv2_backup TO test_repo ON (MATERIALIZED VIEW sr_mv1, MATERIALIZED VIEW sr_mv2); ``` The following example backs up all materialized views from the database `sr_hub` in the snapshot `sr_mv3_backup` and upload the snapshot to the repository `test_repo`. ```sql BACKUP DATABASE sr_hub SNAPSHOT sr_mv3_backup TO test_repo ON (ALL MATERIALIZED VIEWS); ``` ##### Back up logical view[​](#back-up-logical-view "Direct link to Back up logical view") The following example backs up the logical view `sr_view1` from the database `sr_hub` in the snapshot `sr_view1_backup` and upload the snapshot to the repository `test_repo`. ```sql BACKUP DATABASE sr_hub SNAPSHOT sr_view1_backup TO test_repo ON (VIEW sr_view1); ``` The following example backs up two logical views, `sr_view1` and `sr_view2`, from the database `sr_hub` in the snapshot `sr_view2_backup` and upload the snapshot to the repository `test_repo`. ```sql BACKUP DATABASE sr_hub SNAPSHOT sr_view2_backup TO test_repo ON (VIEW sr_view1, VIEW sr_view2); ``` The following example backs up all logical views from the database `sr_hub` in the snapshot `sr_view3_backup` and upload the snapshot to the repository `test_repo`. ```sql BACKUP DATABASE sr_hub SNAPSHOT sr_view3_backup TO test_repo ON (ALL VIEWS); ``` ##### Back up UDF[​](#back-up-udf "Direct link to Back up UDF") The following example backs up the UDF `sr_udf1` from the database `sr_hub` in the snapshot `sr_udf1_backup` and upload the snapshot to the repository `test_repo`. ```sql BACKUP DATABASE sr_hub SNAPSHOT sr_udf1_backup TO test_repo ON (FUNCTION sr_udf1); ``` The following example backs up two UDFs, `sr_udf1` and `sr_udf2`, from the database `sr_hub` in the snapshot `sr_udf2_backup` and upload the snapshot to the repository `test_repo`. ```sql BACKUP DATABASE sr_hub SNAPSHOT sr_udf2_backup TO test_repo ON (FUNCTION sr_udf1, FUNCTION sr_udf2); ``` The following example backs up all UDFs from the database `sr_hub` in the snapshot `sr_udf3_backup` and upload the snapshot to the repository `test_repo`. ```sql BACKUP DATABASE sr_hub SNAPSHOT sr_udf3_backup TO test_repo ON (ALL FUNCTIONS); ``` ##### Back up metadata of external catalog[​](#back-up-metadata-of-external-catalog "Direct link to Back up metadata of external catalog") The following example backs up the metadata of the external catalog `iceberg` in the snapshot `iceberg_backup` and upload the snapshot to the repository `test_repo`. ```sql BACKUP EXTERNAL CATALOG (iceberg) SNAPSHOT iceberg_backup TO test_repo; ``` The following example backs up the metadata of two external catalogs, `iceberg` and `hive`, in the snapshot `iceberg_hive_backup` and upload the snapshot to the repository `test_repo`. ```sql BACKUP EXTERNAL CATALOGS (iceberg, hive) SNAPSHOT iceberg_hive_backup TO test_repo; ``` The following example backs up the metadata of all external catalogs in the snapshot `all_catalog_backup` and upload the snapshot to the repository `test_repo`. ```sql BACKUP ALL EXTERNAL CATALOGS SNAPSHOT all_catalog_backup TO test_repo; ``` To cancel the BACKUP operation on external catalogs, execute the following statement: ```sql CANCEL BACKUP FOR EXTERNAL CATALOG; ``` #### Restore data[​](#restore-data "Direct link to Restore data") You can restore the data snapshot backed up in the remote storage system to the current or other StarRocks clusters to restore or migrate data. **When you restore an object from a snapshot, you must specify the timestamp of the snapshot.** Use the [RESTORE](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/RESTORE.md) statement to restore data snapshots in the remote storage system. RESTORE is an asynchronous operation. You can check the status of a RESTORE job using [SHOW RESTORE](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/SHOW_RESTORE.md), or cancel a RESTORE job using [CANCEL RESTORE](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/CANCEL_RESTORE.md). ##### (Optional) Create a repository in the new cluster[​](#optional-create-a-repository-in-the-new-cluster "Direct link to (Optional) Create a repository in the new cluster") To migrate data to another StarRocks cluster, you need to create a repository with the same **repository name** and **location** in the target cluster, otherwise, you will not be able to view the previously backed-up data snapshots. See [Create a repository](#create-a-repository) for details. ##### Obtain snapshot timestamp[​](#obtain-snapshot-timestamp "Direct link to Obtain snapshot timestamp") Before restoring data, you can check the snapshots in the repository to obtain the timestamps using [SHOW SNAPSHOT](https://docs.starrocks.io/docs/sql-reference/sql-statements/backup_restore/SHOW_SNAPSHOT.md). The following example checks the snapshot information in `test_repo`. ```plain mysql> SHOW SNAPSHOT ON test_repo; +------------------+-------------------------+--------+ | Snapshot | Timestamp | Status | +------------------+-------------------------+--------+ | sr_member_backup | 2023-02-07-14-45-53-143 | OK | +------------------+-------------------------+--------+ 1 row in set (1.16 sec) ``` ##### Restore database[​](#restore-database "Direct link to Restore database") The following examples restore the database `sr_hub` in the snapshot `sr_hub_backup` to the database `sr_hub` in the target cluster. If the database does not exist in the snapshot, the system will return an error. If the database does not exist in the target cluster, the system will create it automatically. ```sql -- Supported from v3.4.0 onwards. RESTORE SNAPSHOT sr_hub_backup FROM test_repo DATABASE sr_hub PROPERTIES("backup_timestamp" = "2024-12-09-10-25-58-842"); -- Compatible with the syntax in earlier versions. RESTORE SNAPSHOT sr_hub.sr_hub_backup FROM `test_repo` PROPERTIES("backup_timestamp" = "2024-12-09-10-25-58-842"); ``` The following example restore the database `sr_hub` in the snapshot `sr_hub_backup` to the database `sr_hub_new` in the target cluster. If the database `sr_hub` does not exist in the snapshot, the system will return an error. If the database `sr_hub_new` does not exist in the target cluster, the system will create it automatically. ```sql -- Supported from v3.4.0 onwards. RESTORE SNAPSHOT sr_hub_backup FROM test_repo DATABASE sr_hub AS sr_hub_new PROPERTIES("backup_timestamp" = "2024-12-09-10-25-58-842"); ``` ##### Restore table[​](#restore-table "Direct link to Restore table") The following examples restore the table `sr_member` of the database `sr_hub` in the snapshot `sr_member_backup` to the table `sr_member` of the database `sr_hub` in the target cluster. ```sql -- Supported from v3.4.0 onwards. RESTORE SNAPSHOT sr_member_backup FROM test_repo DATABASE sr_hub ON (TABLE sr_member) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); -- Compatible with the syntax in earlier versions. RESTORE SNAPSHOT sr_hub.sr_member_backup FROM test_repo ON (sr_member) PROPERTIES ("backup_timestamp"="2024-12-09-10-52-10-940"); ``` The following examples restore the table `sr_member` of the database `sr_hub` in the snapshot `sr_member_backup` to the table `sr_member_new` of the database `sr_hub_new` in the target cluster. ```sql RESTORE SNAPSHOT sr_member_backup FROM test_repo DATABASE sr_hub AS sr_hub_new ON (TABLE sr_member AS sr_member_new) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` The following example restores two tables, `sr_member` and `sr_pmc`, of the database `sr_hub` in the snapshot `sr_core_backup` to two tables, `sr_member` and `sr_pmc`, of the database `sr_hub` in the target cluster. ```sql RESTORE SNAPSHOT sr_core_backup FROM test_repo DATABASE sr_hub ON (TABLE sr_member, TABLE sr_pmc) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` The following example restores all tables from the database `sr_hub` in the snapshot `sr_all_backup`. ```sql RESTORE SNAPSHOT sr_all_backup FROM test_repo DATABASE sr_hub ON (ALL TABLES); ``` The following example restores one of all tables from the database `sr_hub` in the snapshot `sr_all_backup`. ```sql RESTORE SNAPSHOT sr_all_backup FROM test_repo DATABASE sr_hub ON (TABLE sr_member) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` ##### Restore partition[​](#restore-partition "Direct link to Restore partition") The following examples restore the partition `p1` of the table `sr_member` in the snapshot `sr_par_backup` to the partition `p1` of the table `sr_member` in the target cluster. ```sql -- Supported from v3.4.0 onwards. RESTORE SNAPSHOT sr_par_backup FROM test_repo DATABASE sr_hub ON (TABLE sr_member PARTITION (p1)) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); -- Compatible with the syntax in earlier versions. RESTORE SNAPSHOT sr_hub.sr_par_backup FROM test_repo ON (sr_member PARTITION (p1)) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` You can specify multiple partition names separated by commas (`,`) to restore partitions in batch. ##### Restore materialized view[​](#restore-materialized-view "Direct link to Restore materialized view") The following example restores the materialized view `sr_mv1` from the database `sr_hub` in the snapshot `sr_mv1_backup` to the target cluster. ```sql RESTORE SNAPSHOT sr_mv1_backup FROM test_repo DATABASE sr_hub ON (MATERIALIZED VIEW sr_mv1) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` The following example restores two materialized views, `sr_mv1` and `sr_mv2`, from the database `sr_hub` in the snapshot `sr_mv2_backup` to the target cluster. ```sql RESTORE SNAPSHOT sr_mv2_backup FROM test_repo DATABASE sr_hub ON (MATERIALIZED VIEW sr_mv1, MATERIALIZED VIEW sr_mv2) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` The following example restores all materialized views from the database `sr_hub` in the snapshot `sr_mv3_backup` to the target cluster. ```sql RESTORE SNAPSHOT sr_mv3_backup FROM test_repo DATABASE sr_hub ON (ALL MATERIALIZED VIEWS) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` The following example restores one of the materialized views from the database `sr_hub` in the snapshot `sr_mv3_backup` to the target cluster. ```sql RESTORE SNAPSHOT sr_mv3_backup FROM test_repo DATABASE sr_hub ON (MATERIALIZED VIEW sr_mv1) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` info After RESTORE, you can check the status of the materialized view using [SHOW MATERIALIZED VIEWS](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/SHOW_MATERIALIZED_VIEW.md). * If the materialized view is active, it can be used directly. * If the materialized view is inactive, it might be because its base tables are not restored. After all the base tables are restored, you can use [ALTER MATERIALIZED VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/ALTER_MATERIALIZED_VIEW.md) to re-activate the materialized view. ##### Restore logical view[​](#restore-logical-view "Direct link to Restore logical view") The following example restores the logical view `sr_view1` from the database `sr_hub` in the snapshot `sr_view1_backup` to the target cluster. ```sql RESTORE SNAPSHOT sr_view1_backup FROM test_repo DATABASE sr_hub ON (VIEW sr_view1) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` The following example restores two logical views, `sr_view1` and `sr_view2`, from the database `sr_hub` in the snapshot `sr_view2_backup` to the target cluster. ```sql RESTORE SNAPSHOT sr_view2_backup FROM test_repo DATABASE sr_hub ON (VIEW sr_view1, VIEW sr_view2) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` The following example restores all logical views from the database `sr_hub` in the snapshot `sr_view3_backup` to the target cluster. ```sql RESTORE SNAPSHOT sr_view3_backup FROM test_repo DATABASE sr_hub ON (ALL VIEWS) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` The following example restores one of all logical views from the database `sr_hub` in the snapshot `sr_view3_backup` to the target cluster. ```sql RESTORE SNAPSHOT sr_view3_backup FROM test_repo DATABASE sr_hub ON (VIEW sr_view1) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` ##### Restore UDF[​](#restore-udf "Direct link to Restore UDF") The following example restores the UDF `sr_udf1` from the database `sr_hub` in the snapshot `sr_udf1_backup` to the target cluster. ```sql RESTORE SNAPSHOT sr_udf1_backup FROM test_repo DATABASE sr_hub ON (FUNCTION sr_udf1) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` The following example restores two UDFs, `sr_udf1` and `sr_udf2`, from the database `sr_hub` in the snapshot `sr_udf2_backup` to the target cluster. ```sql RESTORE SNAPSHOT sr_udf2_backup FROM test_repo DATABASE sr_hub ON (FUNCTION sr_udf1, FUNCTION sr_udf2) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` The following example restores all UDFs from the database `sr_hub` in the snapshot `sr_udf3_backup` to the target cluster. ```sql RESTORE SNAPSHOT sr_udf3_backup FROM test_repo DATABASE sr_hub ON (ALL FUNCTIONS) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` The following example restores one of all UDFs from the database `sr_hub` in the snapshot `sr_udf3_backup` to the target cluster. ```sql RESTORE SNAPSHOT sr_udf3_backup FROM test_repo DATABASE sr_hub ON (FUNCTION sr_udf1) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` ##### Restore metadata of external catalog[​](#restore-metadata-of-external-catalog "Direct link to Restore metadata of external catalog") The following example restores the metadata of the external catalog `iceberg` in the snapshot `iceberg_backup` to the target cluster, and rename it as `iceberg_new`. ```sql RESTORE SNAPSHOT iceberg_backup FROM test_repo EXTERNAL CATALOG (iceberg AS iceberg_new) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` The following example restores the metadata of two external catalogs, `iceberg` and `hive`, in the snapshot `iceberg_hive_backup` to the target cluster. ```sql RESTORE SNAPSHOT iceberg_hive_backup FROM test_repo EXTERNAL CATALOGS (iceberg, hive) PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` The following example restores the metadata of all external catalogs in the snapshot `all_catalog_backup` to the target cluster. ```sql RESTORE SNAPSHOT all_catalog_backup FROM test_repo ALL EXTERNAL CATALOGS PROPERTIES ("backup_timestamp" = "2024-12-09-10-52-10-940"); ``` To cancel the RESTORE operation on external catalogs, execute the following statement: ```sql CANCEL RESTORE FOR EXTERNAL CATALOG; ``` #### Configure BACKUP or RESTORE jobs[​](#configure-backup-or-restore-jobs "Direct link to Configure BACKUP or RESTORE jobs") You can optimize the performance of BACKUP or RESTORE jobs by modifying the following configuration items in the BE configuration file **be.conf**: | Configuration item | Description | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | make\_snapshot\_worker\_count | The maximum number of threads for the make snapshot tasks of BACKUP jobs on a BE node. Default: `5`. Increase the value of this configuration item to increase the concurrency of the make snapshot task. | | release\_snapshot\_worker\_count | The maximum number of threads for the release snapshot tasks of failed BACKUP jobs on a BE node. Default: `5`. Increase the value of this configuration item to increase the concurrency of the release snapshot task. | | upload\_worker\_count | The maximum number of threads for the upload tasks of BACKUP jobs on a BE node. Default: `0`. `0` indicates setting the value to the number of CPU cores on the machine where the BE resides. Increase the value of this configuration item to increase the concurrency of the upload task. | | download\_worker\_count | The maximum number of threads for the download tasks of RESTORE jobs on a BE node. Default: `0`. `0` indicates setting the value to the number of CPU cores on the machine where the BE resides. Increase the value of this configuration item to increase the concurrency of the download task. | #### Usage notes[​](#usage-notes "Direct link to Usage notes") * Performing backup and restore operations on global, database, table, and partition levels requires different privileges. For detailed information, see [Customize roles based on scenarios](https://docs.starrocks.io/docs/administration/user_privs/authorization/User_privilege.md#customize-roles-based-on-scenarios). * In each database, only one running BACKUP or RESTORE job is allowed each time. Otherwise, StarRocks returns an error. * Because BACKUP and RESTORE jobs occupy many resources of your StarRocks cluster, you can back up and restore your data while your StarRocks cluster is not heavily loaded. * StarRocks does not support specifying data compression algorithms for data backup. * Because data is backed up as snapshots, the data loaded upon snapshot generation is not included in the snapshot. Therefore, if you load data into the old cluster after the snapshot is generated and before the RESTORE job is completed, you also need to load the data into the cluster that data is restored into. It is recommended that you load data into both clusters in parallel for a period of time after the data migration is complete, and then migrate your application to the new cluster after verifying the correctness of the data and services. * Before the RESTORE job is completed, you cannot operate the table to be restored. * Primary Key tables cannot be restored to a StarRocks cluster earlier than v2.5. * You do not need to create the table to be restored in the new cluster before restoring it. The RESTORE job automatically creates it. * If there is an existing table that has a duplicated name with the table to be restored, StarRocks first checks whether or not the schema of the existing table matches that of the table to be restored. If the schemas match, StarRocks overwrites the existing table with the data in the snapshot. If the schema does not match, the RESTORE job fails. You can either rename the table to be restored using the keyword `AS`, or delete the existing table before restoring data. * If the RESTORE job overwrites an existing database, table, or partition, the overwritten data cannot be restored after the job enters the COMMIT phase. If the RESTORE job fails or is canceled at this point, the data may be corrupted and inaccessible. In this case, you can only perform the RESTORE operation again and wait for the job to complete. Therefore, we recommend that you do not restore data by overwriting unless you are sure that the current data is no longer used. The overwrite operation first checks metadata consistency between the snapshot and the existing database, table, or partition. If an inconsistency is detected, the RESTORE operation cannot be performed. * Currently, StarRocks does not support backing up and restoring the configuration data related to user accounts, privileges, and resource groups. * Currently, StarRocks does not support backing up and restoring the Colocate Join relationship among tables. --- ### Manage BE and CN Blacklist From v3.3.0 onwards, StarRocks supports the BE Blacklist feature, which allows you to forbid the usage of certain BE nodes in query execution, thereby avoiding frequent query failures or other unexpected behaviors caused by the failed connections to the BE nodes. A network issue preventing connections to one or more BEs would be an example of when to use the blacklist. From v4.0 onwards, StarRocks supports adding Compute Nodes (CNs) to the Blacklist. By default, StarRocks can automatically manage the BE and CN Blacklist, adding the BE or CN nodes that have lost connection to the blacklist and removing them from the blacklist when the connection is reestablished. However, StarRocks will not remove the node from the Blacklist if it is manually blacklisted. note * Only users with the SYSTEM-level BLACKLIST privilege can use this feature. * Each FE node keeps its own BE and CN Blacklist, and will not share it with other FE nodes. #### Add a BE/CN to the blacklist[​](#add-a-becn-to-the-blacklist "Direct link to Add a BE/CN to the blacklist") You can manually add a BE/CN node to the Blacklist using [ADD BACKEND/COMPUTE NODE BLACKLIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/ADD_BACKEND_BLACKLIST.md). In this statement, you must specify the ID of the BE/CN node to be blacklisted. You can obtain the BE ID by executing [SHOW BACKENDS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md) and CN ID by executing [SHOW COMPUTE NODES](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md). Example: ```sql -- Obtain BE ID. SHOW BACKENDS\G *************************** 1. row *************************** BackendId: 10001 IP: xxx.xx.xx.xxx ... -- Add BE to the blacklist. ADD BACKEND BLACKLIST 10001; -- Obtain CN ID. SHOW COMPUTE NODES\G *************************** 1. row *************************** ComputeNodeId: 10005 IP: xxx.xx.xx.xxx ... -- Add CN to the blacklist. ADD COMPUTE NODE BLACKLIST 10005; ``` #### Remove a BE/CN from blacklist[​](#remove-a-becn-from-blacklist "Direct link to Remove a BE/CN from blacklist") You can manually remove a BE/CN node from the Blacklist using [DELETE BACKEND/COMPUTE NODE BLACKLIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/DELETE_BACKEND_BLACKLIST.md). In this statement, you must also specify the ID of the BE/CN node. Example: ```sql -- Remove a BE from the Blacklist. DELETE BACKEND BLACKLIST 10001; -- Remove a CN from the Blacklist. DELETE COMPUTE NODE BLACKLIST 10005; ``` #### View BE/CN Blacklist[​](#view-becn-blacklist "Direct link to View BE/CN Blacklist") You can view the BE/CN nodes in the Blacklist using [SHOW BACKEND/COMPUTE NODE BLACKLIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKEND_BLACKLIST.md). Example: ```sql -- View the BE Blacklist. SHOW BACKEND BLACKLIST; +-----------+------------------+---------------------+------------------------------+--------------------+ | BackendId | AddBlackListType | LostConnectionTime | LostConnectionNumberInPeriod | CheckTimePeriod(s) | +-----------+------------------+---------------------+------------------------------+--------------------+ | 10001 | MANUAL | 2024-04-28 11:52:09 | 0 | 5 | +-----------+------------------+---------------------+------------------------------+--------------------+ -- View the CN Blacklist. SHOW COMPUTE NODE BLACKLIST; +---------------+------------------+---------------------+------------------------------+--------------------+ | ComputeNodeId | AddBlackListType | LostConnectionTime | LostConnectionNumberInPeriod | CheckTimePeriod(s) | +---------------+------------------+---------------------+------------------------------+--------------------+ | 10005 | MANUAL | 2025-08-18 10:47:51 | 0 | 5 | +---------------+------------------+---------------------+------------------------------+--------------------+ ``` The following fields are returned: * `AddBlackListType`: How the BE/CN node was added to the blacklist. `MANUAL` indicates it is manually blacklisted by the user. `AUTO` indicates it is automatically blacklisted by StarRocks. * `LostConnectionTime`: * For the `MANUAL` type, it indicates the time when the BE/CN node was manually added to the blacklist. * For the `AUTO` type, it indicates the time when the last successful connection was established. * `LostConnectionNumberInPeriod`: The number of disconnections detected within `CheckTimePeriod(s)`, which is the interval at which StarRocks checks the connection status of the BE/CN nodes in the blacklist. * `CheckTimePeriod(s)`: The interval at which StarRocks checks the connection status of the blacklisted BE/CN nodes. Its value is evaluated to the value you specified for the FE configuration item `black_host_history_sec`. Unit: Seconds. #### Configure automatic management of BE/CN Blacklist[​](#configure-automatic-management-of-becn-blacklist "Direct link to Configure automatic management of BE/CN Blacklist") Each time a BE/CN node loses connection to the FE node, or a query fails due to timeout on a BE/CN node, the FE node adds the BE/CN node to its BE and CN Blacklist. The FE node will constantly assess the connectivity of the BE/CN node in the blacklist by counting its connection failures within a certain duration of time. StarRocks will remove a blacklisted BE/CN node only if the number of its connection failures is below a pre-specified threshold. You can configure the automatic management of the BE and CN Blacklist using the following [FE configurations](https://docs.starrocks.io/docs/administration/management/FE_configuration.md): * `black_host_history_sec`: The time duration for retaining historical connection failures of BE/CN nodes in the Blacklist. * `black_host_connect_failures_within_time`: The threshold of connection failures allowed for a blacklisted BE/CN node. If a BE/CN node is added to the Blacklist automatically, StarRocks will assess its connectivity and judge whether it can be removed from the Blacklist. Within `black_host_history_sec`, only if a blacklisted BE/CN node has fewer connection failures than the threshold set in `black_host_connect_failures_within_time`, it can be removed from the Blacklist. --- ### BE Configuration Some BE configuration items are dynamic parameters which you can set interactively when BE nodes are online. The rest of them are static parameters. You can only set the static parameters of a BE node by changing them in the corresponding configuration file **be.conf** and restarting the BE node to allow the change to take effect. #### View BE configuration items[​](#view-be-configuration-items "Direct link to View BE configuration items") You can view the BE configuration items using the following command: ```sql SELECT * FROM information_schema.be_configs WHERE NAME LIKE "%%" ``` #### Configure BE parameters[​](#configure-be-parameters "Direct link to Configure BE parameters") ##### Configure BE dynamic parameters[​](#configure-be-dynamic-parameters "Direct link to Configure BE dynamic parameters") You can configure a dynamic parameter of a BE node by updating the value in `information_schema.be_configs`. warning Setting an invalid value may cause unknown behaviors. Check twice before you run the command to update the configuration. ```sql -- Replace with the key of the configuration and with the value. UPDATE information_schema.be_configs SET VALUE = WHERE name = ""; ``` ##### Configure BE static parameters[​](#configure-be-static-parameters "Direct link to Configure BE static parameters") You can only set the static parameters of a BE by changing them in the corresponding configuration file **be.conf**, and restarting the BE to allow the changes to take effect. #### Parameter groups[​](#parameter-groups "Direct link to Parameter groups") The parameters are grouped in these categories: * [Logging](https://docs.starrocks.io/docs/administration/management/BE_parameters/log_server_meta.md) * [Server](https://docs.starrocks.io/docs/administration/management/BE_parameters/log_server_meta.md) * [Metadata and Cluster management](https://docs.starrocks.io/docs/administration/management/BE_parameters/log_server_meta.md) * [Query engine](https://docs.starrocks.io/docs/administration/management/BE_parameters/query_loading.md) * [Loading and unloading](https://docs.starrocks.io/docs/administration/management/BE_parameters/query_loading.md) * [Statistic report](https://docs.starrocks.io/docs/administration/management/BE_parameters/stats_storage.md) * [Storage](https://docs.starrocks.io/docs/administration/management/BE_parameters/stats_storage.md) * [Shared-data](https://docs.starrocks.io/docs/administration/management/BE_parameters/shared_lake_other.md) * [Data Lake](https://docs.starrocks.io/docs/administration/management/BE_parameters/shared_lake_other.md) * [Other](https://docs.starrocks.io/docs/administration/management/BE_parameters/shared_lake_other.md) --- ### BE Configuration - Logging, Server, and Metadata Some BE configuration items are dynamic parameters which you can set interactively when BE nodes are online. The rest of them are static parameters. You can only set the static parameters of a BE node by changing them in the corresponding configuration file **be.conf** and restarting the BE node to allow the change to take effect. #### View BE configuration items[​](#view-be-configuration-items "Direct link to View BE configuration items") You can view the BE configuration items using the following command: ```sql SELECT * FROM information_schema.be_configs [WHERE NAME LIKE "%%"] ``` #### Configure BE parameters[​](#configure-be-parameters "Direct link to Configure BE parameters") ##### Configure BE dynamic parameters[​](#configure-be-dynamic-parameters "Direct link to Configure BE dynamic parameters") You can configure a dynamic parameter of a BE node by updating the value in `information_schema.be_configs`. warning Setting an invalid value may cause unknown behaviors. Check twice before you run the command to update the configuration. ```sql -- Replace with the key of the configuration and with the value. UPDATE information_schema.be_configs SET VALUE = WHERE name = ""; ``` ##### Configure BE static parameters[​](#configure-be-static-parameters "Direct link to Configure BE static parameters") You can only set the static parameters of a BE by changing them in the corresponding configuration file **be.conf**, and restarting the BE to allow the changes to take effect. *** This topic introduces the following types of BE configurations: * [Logging](#logging) * [Server](#server) * [Metadata and Cluster Management](#metadata-and-cluster-management) #### Logging[​](#logging "Direct link to Logging") ##### diagnose\_stack\_trace\_interval\_ms[​](#diagnose_stack_trace_interval_ms "Direct link to diagnose_stack_trace_interval_ms") * Default: 1800000 (30 minutes) * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: Controls the minimum time gap between successive stack-trace diagnostics performed by DiagnoseDaemon for `STACK_TRACE` requests. When a diagnose request arrives, the daemon skips collecting and logging stack traces if the last collection happened less than `diagnose_stack_trace_interval_ms` milliseconds ago. Increase this value to reduce CPU overhead and log volume from frequent stack dumps; decrease it to capture more frequent traces to debug transient issues (for example, in load fail-point simulations of long `TabletsChannel::add_chunk` blocking). * Introduced in: v3.5.0 ##### lake\_replication\_slow\_log\_ms[​](#lake_replication_slow_log_ms "Direct link to lake_replication_slow_log_ms") * Default: 30000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: Threshold for emitting slow-log entries during lake replication. After each file copy the code measures elapsed time in microseconds and marks the operation as slow when elapsed time is greater than or equal to `lake_replication_slow_log_ms * 1000`. When triggered, StarRocks writes an INFO log with file size, cost and trace metrics for that replicated file. Increase the value to reduce noisy slow logs for large/slow transfers; decrease it to detect and surface smaller slow-copy events sooner. * Introduced in: - ##### load\_rpc\_slow\_log\_frequency\_threshold\_seconds[​](#load_rpc_slow_log_frequency_threshold_seconds "Direct link to load_rpc_slow_log_frequency_threshold_seconds") * Default: 60 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Controls how frequently the system prints slow-log entries for load RPCs that exceed their configured RPC timeout. The slow-log also includes the load channel runtime profile. Setting this value to 0 causes per-timeout logging in practice. * Introduced in: v3.4.3, v3.5.0 ##### log\_buffer\_level[​](#log_buffer_level "Direct link to log_buffer_level") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The strategy for flushing logs. The default value indicates that logs are buffered in memory. Valid values are `-1` and `0`. `-1` indicates that logs are not buffered in memory. * Introduced in: - ##### pprof\_profile\_dir[​](#pprof_profile_dir "Direct link to pprof_profile_dir") * Default: `${STARROCKS_HOME}/log` * Type: String * Unit: - * Is mutable: No * Description: Directory path where StarRocks writes pprof artifacts (Jemalloc heap snapshots and gperftools CPU profiles). * Introduced in: v3.2.0 ##### sys\_log\_dir[​](#sys_log_dir "Direct link to sys_log_dir") * Default: `${STARROCKS_HOME}/log` * Type: String * Unit: - * Is mutable: No * Description: The directory that stores system logs (including INFO, WARNING, ERROR, and FATAL). * Introduced in: - ##### sys\_log\_level[​](#sys_log_level "Direct link to sys_log_level") * Default: INFO * Type: String * Unit: - * Is mutable: Yes (from v3.3.0, v3.2.7, and v3.1.12) * Description: The severity levels into which system log entries are classified. Valid values: INFO, WARNING, ERROR, and FATAL. This item was changed to a dynamic configuration from v3.3.0, v3.2.7, and v3.1.12 onwards. * Introduced in: - ##### sys\_log\_roll\_mode[​](#sys_log_roll_mode "Direct link to sys_log_roll_mode") * Default: SIZE-MB-1024 * Type: String * Unit: - * Is mutable: No * Description: The mode in which system logs are segmented into log rolls. Valid values include `TIME-DAY`, `TIME-HOUR`, and `SIZE-MB-`size. The default value indicates that logs are segmented into rolls, each of which is 1 GB. * Introduced in: - ##### sys\_log\_roll\_num[​](#sys_log_roll_num "Direct link to sys_log_roll_num") * Default: 10 * Type: Int * Unit: - * Is mutable: No * Description: The number of log rolls to reserve. * Introduced in: - ##### sys\_log\_timezone[​](#sys_log_timezone "Direct link to sys_log_timezone") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to show timezone information in the log prefix. `true` indicates to show timezone information, `false` indicates not to show. * Introduced in: - ##### sys\_log\_verbose\_level[​](#sys_log_verbose_level "Direct link to sys_log_verbose_level") * Default: 10 * Type: Int * Unit: - * Is mutable: No * Description: The level of the logs to be printed. This configuration item is used to control the output of logs initiated with VLOG in codes. * Introduced in: - ##### sys\_log\_verbose\_modules[​](#sys_log_verbose_modules "Direct link to sys_log_verbose_modules") * Default: * Type: Strings * Unit: - * Is mutable: No * Description: Specifies the file names (without extensions) or file name wildcards for which VLOG logs should be printed. Multiple file names can be separated by commas. For example, if you set this configuration item to `storage_engine,tablet_manager`, StarRocks prints VLOG logs from the storage\_engine.cpp and tablet\_manager.cpp files. You can also use wildcards, e.g., set to `*` to print VLOG logs from all files. The VLOG log printing level is controlled by the `sys_log_verbose_level` parameter. * Introduced in: - #### Server[​](#server "Direct link to Server") ##### abort\_on\_large\_memory\_allocation[​](#abort_on_large_memory_allocation "Direct link to abort_on_large_memory_allocation") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: When a single allocation request exceeds the configured large-allocation threshold (g\_large\_memory\_alloc\_failure\_threshold `>` 0 and requested size `>` threshold), this flag controls how the process responds. If true, StarRocks calls std::abort() immediately (hard crash) when such a large allocation is detected. If false, the allocation is blocked and the allocator returns failure (nullptr or ENOMEM) so callers can handle the error. This check only takes effect for allocations that are not wrapped with the TRY\_CATCH\_BAD\_ALLOC path (the mem hook uses a different flow when bad-alloc is being caught). Enable for fail-fast debugging of unexpected huge allocations; keep disabled in production unless you want an immediate process abort on over-large allocation attempts. * Introduced in: v3.4.3, 3.5.0, 4.0.0 ##### arrow\_flight\_port[​](#arrow_flight_port "Direct link to arrow_flight_port") * Default: -1 * Type: Int * Unit: - * Is mutable: No * Description: TCP port for the BE Arrow Flight SQL server. `-1` indicaes to disable the Arrow Flight service. On non-macOS builds, BE invokes Arrow Flight SQL Server with this port during startup; if the port is unavailable, the server startup fails and the BE process exits. The configured port is reported to the FE in the heartbeat payload. * Introduced in: v3.4.0, v3.5.0 ##### be\_exit\_after\_disk\_write\_hang\_second[​](#be_exit_after_disk_write_hang_second "Direct link to be_exit_after_disk_write_hang_second") * Default: 60 * Type: Int * Unit: Seconds * Is mutable: No * Description: The length of time that the BE waits to exit after the disk hangs. * Introduced in: - ##### be\_http\_num\_workers[​](#be_http_num_workers "Direct link to be_http_num_workers") * Default: 48 * Type: Int * Unit: - * Is mutable: No * Description: The number of threads used by the HTTP server. * Introduced in: - ##### be\_http\_port[​](#be_http_port "Direct link to be_http_port") * Default: 8040 * Type: Int * Unit: - * Is mutable: No * Description: The BE HTTP server port. * Introduced in: - ##### enable\_http\_auth[​](#enable_http_auth "Direct link to enable_http_auth") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Introduced in: v4.2.0 * Description: When true, most external BE HTTP endpoints require HTTP Basic Auth. Credentials are verified by RPC to the FE leader using the `checkAuth` Thrift method, so the user/password store on the FE side (including LDAP / security-integration) is the source of truth. The following are exempt: * Token-gated internal transport (used by FE/BE for tablet clone and load-error file fetch): `/api/_tablet/_download`, `/api/_download_load`. These remain protected by their own token check; setting `enable_http_auth=true` does **not** compensate for `enable_token_check=false`. * Stream Load and transaction endpoints that authenticate inside the handler using the load label + table grants: `/api/{db}/{table}/_stream_load`, `/api/transaction/{txn_op}`, `/api/transaction/load`. Privileged endpoints additionally require a SYSTEM-level RBAC privilege (`OPERATE` or `NODE`) that is **active** in the session — use `SET DEFAULT ROLE TO ;` or set `activate_all_roles_on_login=true` if the role is granted but not default. LDAP / security-integration group → role mappings activate automatically. ##### be\_port[​](#be_port "Direct link to be_port") * Default: 9060 * Type: Int * Unit: - * Is mutable: No * Description: The BE thrift server port, which is used to receive requests from FEs. * Introduced in: - ##### be\_service\_threads[​](#be_service_threads "Direct link to be_service_threads") * Default: 64 * Type: Int * Unit: Threads * Is mutable: No * Description: Number of worker threads the BE Thrift server uses to serve backend RPC/execution requests. This value is passed to ThriftServer when creating the BackendService and controls how many concurrent request handlers are available; requests are queued when all worker threads are busy. Tune based on expected concurrent RPC load and available CPU/memory: increasing it raises concurrency but also per-thread memory and context-switch cost, decreasing it limits parallel handling and may increase request latency. * Introduced in: v3.2.0 ##### brpc\_connection\_type[​](#brpc_connection_type "Direct link to brpc_connection_type") * Default: `"single"` * Type: string * Unit: - * Is mutable: No * Description: The bRPC channel connection mode. Valid values: * `"single"` (Default): One persistent TCP connection for each channel. * `"pooled"`: A pool of persistent connections for higher concurrency at the cost of more sockets/file descriptors. * `"short"`: Short‑lived connections created per RPC to reduce persistent resource usage but with higher latency. The choice affects per-socket buffering behavior and can influence `Socket.Write` failures (EOVERCROWDED) when unwritten bytes exceed socket limits. * Introduced in: v3.2.5 ##### brpc\_max\_body\_size[​](#brpc_max_body_size "Direct link to brpc_max_body_size") * Default: 2147483648 * Type: Int * Unit: Bytes * Is mutable: No * Description: The maximum body size of a bRPC. * Introduced in: - ##### brpc\_max\_connections\_per\_server[​](#brpc_max_connections_per_server "Direct link to brpc_max_connections_per_server") * Default: 1 * Type: Int * Unit: - * Is mutable: No * Description: The maximum number of persistent bRPC connections the client keeps for each remote server endpoint. For each endpoint `BrpcStubCache` creates a `StubPool` whose `_stubs` vector is reserved to this size. On first accesses, new stubs are created until the limit is reached. After that, existing stubs are returned in a round‑robin fashion. Increasing this value raises per‑endpoint concurrency (reduces contention on a single channel) at the cost of more file descriptors, memory, and channels. * Introduced in: v3.2.0 ##### brpc\_num\_threads[​](#brpc_num_threads "Direct link to brpc_num_threads") * Default: -1 * Type: Int * Unit: - * Is mutable: No * Description: The number of bthreads of a bRPC. The value `-1` indicates the same number with the CPU threads. * Introduced in: - ##### brpc\_port[​](#brpc_port "Direct link to brpc_port") * Default: 8060 * Type: Int * Unit: - * Is mutable: No * Description: The BE bRPC port, which is used to view the network statistics of bRPCs. * Introduced in: - ##### brpc\_socket\_max\_unwritten\_bytes[​](#brpc_socket_max_unwritten_bytes "Direct link to brpc_socket_max_unwritten_bytes") * Default: 1073741824 * Type: Int * Unit: Bytes * Is mutable: No * Description: Sets the per-socket limit for unwritten outbound bytes in the bRPC server. When the amount of buffered, not-yet-written data on a socket reaches this limit, subsequent `Socket.Write` calls fail with EOVERCROWDED. This prevents unbounded per-connection memory growth but can cause RPC send failures for very large messages or slow peers. Align this value with `brpc_max_body_size` to ensure single-message bodies are not larger than the allowed unwritten buffer. Increasing the value raises memory usage per connection. * Introduced in: v3.2.0 ##### brpc\_stub\_expire\_s[​](#brpc_stub_expire_s "Direct link to brpc_stub_expire_s") * Default: 3600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The expire time of bRPC stub cache. The default value is 60 minutes. * Introduced in: - ##### compress\_rowbatches[​](#compress_rowbatches "Direct link to compress_rowbatches") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: A boolean value to control whether to compress the row batches in RPCs between BEs. `true` indicates compressing the row batches, and `false` indicates not compressing them. * Introduced in: - ##### consistency\_max\_memory\_limit\_percent[​](#consistency_max_memory_limit_percent "Direct link to consistency_max_memory_limit_percent") * Default: 20 * Type: Int * Unit: - * Is mutable: No * Description: Percentage cap used to compute the memory budget for consistency-related tasks. During BE startup, the final consistency limit is computed as the minimum of the value parsed from `consistency_max_memory_limit` (bytes) and (`process_mem_limit * consistency_max_memory_limit_percent / 100`). If `process_mem_limit` is unset (-1), consistency memory is considered unlimited. For `consistency_max_memory_limit_percent`, values less than 0 or greater than 100 are treated as 100. Adjusting this value increases or decreases memory reserved for consistency operations and therefore affects memory available for queries and other services. * Introduced in: v3.2.0 ##### delete\_worker\_count\_normal\_priority[​](#delete_worker_count_normal_priority "Direct link to delete_worker_count_normal_priority") * Default: 2 * Type: Int * Unit: Threads * Is mutable: No * Description: Number of normal-priority worker threads dedicated to handling delete (REALTIME\_PUSH with DELETE) tasks on the BE agent. At startup this value is added to delete\_worker\_count\_high\_priority to size the DeleteTaskWorkerPool (see agent\_server.cpp). The pool assigns the first delete\_worker\_count\_high\_priority threads as HIGH priority and the rest as NORMAL; normal-priority threads process standard delete tasks and contribute to overall delete throughput. Increase to raise concurrent delete capacity (higher CPU/IO usage); decrease to reduce resource contention. * Introduced in: v3.2.0 ##### disable\_mem\_pools[​](#disable_mem_pools "Direct link to disable_mem_pools") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to disable MemPool. When this item is set to `true`, the MemPool chunk pooling is disabled so each allocation gets its own sized chunk instead of reusing or increasing pooled chunks. Disabling pooling reduces long-lived retained buffer memory at the cost of more frequent allocations, increased number of chunks, and skipped integrity checks (which are avoided because of the large chunk count). Keep `disable_mem_pools` as `false` (default) to benefit from allocation reuse and fewer system calls. Set it to `true` only when you must avoid large pooled memory retention (for example, low-memory environments or diagnostic runs). * Introduced in: v3.2.0 ##### enable\_https[​](#enable_https "Direct link to enable_https") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: When this item is set to `true`, the BE's bRPC server is configured to use TLS: `ServerOptions.ssl_options` will be populated with the certificate and private key specified by `ssl_certificate_path` and `ssl_private_key_path` at BE startup. This enables HTTPS/TLS for incoming bRPC connections; clients must connect using TLS. Ensure the certificate and key files exist, are accessible to the BE process, and match bRPC/SSL expectations. * Introduced in: v4.0.0 ##### enable\_jemalloc\_memory\_tracker[​](#enable_jemalloc_memory_tracker "Direct link to enable_jemalloc_memory_tracker") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: When this item is set to `true`, the BE starts a background thread (jemalloc\_tracker\_daemon) that polls jemalloc statistics (once per second) and updates the GlobalEnv jemalloc metadata MemTracker with the jemalloc "stats.metadata" value. This ensures jemalloc metadata consumption is included in StarRocks process memory accounting and prevents under‑reporting of memory used by jemalloc internals. The tracker is only compiled/started on non‑macOS builds (#ifndef **APPLE**) and runs as a daemon thread named "jemalloc\_tracker\_daemon". Because this setting affects startup behaviour and threads that maintain MemTracker state, changing it requires a restart. Disable only if jemalloc is not used or when jemalloc tracking is intentionally managed differently; otherwise keep enabled to maintain accurate memory accounting and allocation safeguards. * Introduced in: v3.2.12 ##### enable\_jvm\_metrics[​](#enable_jvm_metrics "Direct link to enable_jvm_metrics") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: Controls whether the system initializes and registers JVM-specific metrics at startup. When enabled the metrics subsystem will create JVM-related collectors (for example, heap, GC and thread metrics) for export, and when disabled, those collectors are not initialized. This parameter is intended for forward compatibility and may be removed in a future release. Use `enable_system_metrics` to control system-level metric collection. * Introduced in: v4.0.0 ##### get\_pindex\_worker\_count[​](#get_pindex_worker_count "Direct link to get_pindex_worker_count") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: Sets the number of worker threads for the "get\_pindex" thread pool in UpdateManager, which is used to load / fetch persistent index data (used when applying rowsets for primary-key tables). At runtime, a config update will adjust the pool's maximum threads: if `>0` that value is applied; if 0 the runtime callback uses the number of CPU cores (CpuInfo::num\_cores()). On initialization the pool's max threads is computed as max(get\_pindex\_worker\_count, max\_apply\_thread\_cnt \* 2) where max\_apply\_thread\_cnt is the apply-thread pool maximum. Increase to raise parallelism for pindex loading; lowering reduces concurrency and memory/CPU usage. * Introduced in: v3.2.0 ##### heartbeat\_service\_port[​](#heartbeat_service_port "Direct link to heartbeat_service_port") * Default: 9050 * Type: Int * Unit: - * Is mutable: No * Description: The BE heartbeat service port, which is used to receive heartbeats from FEs. * Introduced in: - ##### heartbeat\_service\_thread\_count[​](#heartbeat_service_thread_count "Direct link to heartbeat_service_thread_count") * Default: 1 * Type: Int * Unit: - * Is mutable: No * Description: The thread count of the BE heartbeat service. * Introduced in: - ##### local\_library\_dir[​](#local_library_dir "Direct link to local_library_dir") * Default: `${UDF_RUNTIME_DIR}` * Type: string * Unit: - * Is mutable: No * Description: Local directory on the BE where UDF (user-defined function) libraries are staged and where Python UDF worker processes operate. StarRocks copies UDF libraries from HDFS into this path, creates per-worker Unix domain sockets at `/pyworker_`, and chdirs Python worker processes into this directory before exec. The directory must exist, be writable by the BE process, and reside on a filesystem that supports Unix domain sockets (i.e., a local filesystem). Because this config is immutable at runtime, set it before startup and ensure adequate permissions and disk space on each BE. * Introduced in: v3.2.0 ##### max\_transmit\_batched\_bytes[​](#max_transmit_batched_bytes "Direct link to max_transmit_batched_bytes") * Default: 262144 * Type: Int * Unit: Bytes * Is mutable: No * Description: Maximum number of serialized bytes to accumulate in a single transmit request before it is flushed to the network. Sender implementations add serialized ChunkPB payloads into a PTransmitChunkParams request and send the request once the accumulated bytes exceed `max_transmit_batched_bytes` or when EOS is reached. Increase this value to reduce RPC frequency and improve throughput at the cost of higher per-request latency and memory use; reduce it to lower latency and memory but increase RPC rate. * Introduced in: v3.2.0 ##### mem\_limit[​](#mem_limit "Direct link to mem_limit") * Default: 90% * Type: String * Unit: - * Is mutable: No * Description: BE process memory upper limit. You can set it as a percentage ("80%") or a physical limit ("100G"). The default hard limit is 90% of the server's memory size, and the soft limit is 80%. You need to configure this parameter if you want to deploy StarRocks with other memory-intensive services on a same server. * Introduced in: - ##### memory\_max\_alignment[​](#memory_max_alignment "Direct link to memory_max_alignment") * Default: 16 * Type: Int * Unit: Bytes * Is mutable: No * Description: Sets the maximum byte alignment that MemPool will accept for aligned allocations. Increase this value only when callers require larger alignment (for SIMD, device buffers, or ABI constraints). Larger values increase per-allocation padding and reserved memory waste and must remain within what the system allocator and platform support. * Introduced in: v3.2.0 ##### memory\_urgent\_level[​](#memory_urgent_level "Direct link to memory_urgent_level") * Default: 85 * Type: long * Unit: Percentage (0-100) * Is mutable: Yes * Description: The emergency memory water‑level expressed as a percentage of the process memory limit. When process memory consumption exceeds `(limit * memory_urgent_level / 100)`, BE triggers immediate memory reclamation, which forces data cache shrinkage, evicts update caches, and causes persistent/lake MemTables to be treated as "full" so they will be flushed/compacted soon. The code validates that this setting must be greater than `memory_high_level`, and `memory_high_level` must be greater or equal to `1`, and less thant or equal to `100`). A lower value causes more aggressive, earlier reclamation, that is, more frequent cache evictions and flushes. A higher value delays reclamation and risks OOM if too close to 100. Tune this item together with `memory_high_level` and Data Cache-related auto‑adjust settings. * Introduced in: v3.2.0 ##### net\_use\_ipv6\_when\_priority\_networks\_empty[​](#net_use_ipv6_when_priority_networks_empty "Direct link to net_use_ipv6_when_priority_networks_empty") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: A boolean value to control whether to use IPv6 addresses preferentially when `priority_networks` is not specified. `true` indicates to allow the system to use an IPv6 address preferentially when the server that hosts the node has both IPv4 and IPv6 addresses and `priority_networks` is not specified. * Introduced in: v3.3.0 ##### num\_cores[​](#num_cores "Direct link to num_cores") * Default: 0 * Type: Int * Unit: Cores * Is mutable: No * Description: Controls the number of CPU cores the system will use for CPU-aware decisions (for example, thread-pool sizing and runtime scheduling). A value of 0 enables auto-detection: the system reads `/proc/cpuinfo` and uses all available cores. If set to a positive integer, that value overrides the detected core count and becomes the effective core count. When running inside containers, cgroup cpuset or cpu quota settings can further restrict usable cores; `CpuInfo` also respects those cgroup limits. * Introduced in: v3.2.0 ##### plugin\_path[​](#plugin_path "Direct link to plugin_path") * Default: `${STARROCKS_HOME}/plugin` * Type: String * Unit: - * Is mutable: No * Description: Filesystem directory where StarRocks loads external plugins (dynamic libraries, connector artifacts, UDF binaries, etc.). `plugin_path` should point to a directory accessible by the BE process (read and execute permissions) and must exist before plugins are loaded. Ensure correct ownership and that plugin files use the platform's native binary extension (for example, .so on Linux). * Introduced in: v3.2.0 ##### priority\_networks[​](#priority_networks "Direct link to priority_networks") * Default: An empty string * Type: String * Unit: - * Is mutable: No * Description: Declares a selection strategy for servers that have multiple IP addresses. Note that at most one IP address must match the list specified by this parameter. The value of this parameter is a list that consists of entries, which are separated with semicolons (;) in CIDR notation, such as `10.10.10.0/24`. If no IP address matches the entries in this list, an available IP address of the server will be randomly selected. From v3.3.0, StarRocks supports deployment based on IPv6. If the server has both IPv4 and IPv6 addresses, and this parameter is not specified, the system uses an IPv4 address by default. You can change this behavior by setting `net_use_ipv6_when_priority_networks_empty` to `true`. * Introduced in: - ##### process\_force\_exit\_after\_crash\_handler\_hang\_second[​](#process_force_exit_after_crash_handler_hang_second "Direct link to process_force_exit_after_crash_handler_hang_second") * Default: 0 * Type: Int * Unit: Seconds * Is mutable: No * Description: If the fatal-signal (crash) handler hangs, for example a jemalloc deadlock while releasing resources before the core dump, the BE/CN forces the process to exit after this many seconds so that an orchestrator can restart it. The crash flag is still set first, so the FE keeps seeing `SHUTDOWN` heartbeats during the grace window; this only bounds how long a crashing process can linger while alive. Disabled by default (`0`) so that an upgrade keeps the existing crash and core dump behavior unchanged; set a positive value (and restart the node) to opt in, choosing a timeout suited to your environment. The value is read once at startup, and the watchdog thread is only launched when it is positive; a value of `0` or less keeps the watchdog disabled. * Introduced in: v4.0.15, v4.1.5 ##### rpc\_compress\_ratio\_threshold[​](#rpc_compress_ratio_threshold "Direct link to rpc_compress_ratio_threshold") * Default: 1.1 * Type: Double * Unit: - * Is mutable: Yes * Description: Threshold (uncompressed\_size / compressed\_size) used when deciding whether to send serialized row-batches over the network in compressed form. When compression is attempted (e.g., in DataStreamSender, exchange sink, tablet sink index channel, dictionary cache writer), StarRocks computes compress\_ratio = uncompressed\_size / compressed\_size; it uses the compressed payload only if compress\_ratio `>` rpc\_compress\_ratio\_threshold. With the default 1.1, compressed data must be at least ~9.1% smaller than uncompressed to be used. Lower the value to prefer compression (more CPU for smaller bandwidth savings); raise it to avoid compression overhead unless it yields larger size reductions. Note: this applies to RPC/shuffle serialization and is effective only when row-batch compression is enabled (compress\_rowbatches). * Introduced in: v3.2.0 ##### enable\_threadpool\_catch\_task\_exception[​](#enable_threadpool_catch_task_exception "Direct link to enable_threadpool_catch_task_exception") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether a ThreadPool worker swallows an exception thrown by a task and continues with the next task. When set to `false` (default), there is no catch clause enclosing the task body, so an exception that escapes the task finds no handler and terminates the BE process at the throw point. When set to `true`, the exception is logged at the ERROR level, the process-wide metric [`threadpool_task_exception_total`](https://docs.starrocks.io/docs/administration/management/monitoring/metric_details/t-z.md#threadpool_task_exception_total) is incremented, and the worker proceeds to the next task, which keeps the process alive. Note that keeping the worker alive does not make the task exception-safe: if a task signals its completion from a `DeferOp` or from its destructor while the statements that record its result are skipped, the waiter reads the task as successful, because an unset `Status` reads as OK. The failure then produces wrong results instead of an error. Set this item to `true` only to mitigate a crash loop, and expect the affected failures to become silent. * Introduced in: - ##### ssl\_private\_key\_path[​](#ssl_private_key_path "Direct link to ssl_private_key_path") * Default: An empty string * Type: String * Unit: - * Is mutable: No * Description: File system path to the TLS/SSL private key (PEM) that the BE's brpc server uses as the private key for the default certificate. When `enable_https` is set to `true`, the system sets `brpc::ServerOptions::ssl_options().default_cert.private_key` to this path at process start. The file must be accessible by the BE process and must match the certificate provided by `ssl_certificate_path`. If this value is not set or the file is missing or unaccessible, HTTPS will not be configured and the bRPC server may fail to start. Protect this file with restrictive filesystem permissions (for example, 600). * Introduced in: v4.0.0 ##### thrift\_client\_retry\_interval\_ms[​](#thrift_client_retry_interval_ms "Direct link to thrift_client_retry_interval_ms") * Default: 100 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: The time interval at which a thrift client retries. * Introduced in: - ##### thrift\_connect\_timeout\_seconds[​](#thrift_connect_timeout_seconds "Direct link to thrift_connect_timeout_seconds") * Default: 3 * Type: Int * Unit: Seconds * Is mutable: No * Description: Connection timeout (in seconds) used when creating Thrift clients. ClientCacheHelper::\_create\_client multiplies this value by 1000 and passes it to ThriftClientImpl::set\_conn\_timeout(), so it controls the TCP/connect handshake timeout for new Thrift connections opened by the BE client cache. This setting affects only connection establishment; send/receive timeouts are configured separately. Very small values can cause spurious connection failures on high-latency networks, while large values delay detection of unreachable peers. * Introduced in: v3.2.0 ##### thrift\_port[​](#thrift_port "Direct link to thrift_port") * Default: 0 * Type: Int * Unit: - * Is mutable: No * Description: Port used to export the internal Thrift-based BackendService. When the process runs as a Compute Node and this item is set to a non-zero value, it overrides `be_port` and the Thrift server binds to this value; otherwise `be_port` is used. This configuration is deprecated — setting a non-zero `thrift_port` logs a warning advising to use `be_port` instead. * Introduced in: v3.2.0 ##### thrift\_rpc\_connection\_max\_valid\_time\_ms[​](#thrift_rpc_connection_max_valid_time_ms "Direct link to thrift_rpc_connection_max_valid_time_ms") * Default: 5000 * Type: Int * Unit: Milliseconds * Is mutable: No * Description: Maximum valid time for a thrift RPC connection. A connection will be closed if it has existed in the connection pool for longer than this value. It must be set consistent with FE configuration `thrift_client_timeout_ms`. * Introduced in: - ##### thrift\_rpc\_max\_body\_size[​](#thrift_rpc_max_body_size "Direct link to thrift_rpc_max_body_size") * Default: 0 * Type: Int * Unit: * Is mutable: No * Description: The maximum string body size of RPC. `0` indicates the size is unlimited. * Introduced in: - ##### thrift\_rpc\_strict\_mode[​](#thrift_rpc_strict_mode "Direct link to thrift_rpc_strict_mode") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: Whether thrift's strict execution mode is enabled. For more information on thrift strict mode, see [Thrift Binary protocol encoding](https://github.com/apache/thrift/blob/master/doc/specs/thrift-binary-protocol.md). * Introduced in: - ##### thrift\_rpc\_timeout\_ms[​](#thrift_rpc_timeout_ms "Direct link to thrift_rpc_timeout_ms") * Default: 5000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: The timeout for a thrift RPC. * Introduced in: - ##### transaction\_apply\_thread\_pool\_num\_min[​](#transaction_apply_thread_pool_num_min "Direct link to transaction_apply_thread_pool_num_min") * Default: 0 * Type: Int * Unit: Threads * Is mutable: Yes * Description: Sets the minimum number of threads for the "update\_apply" thread pool in BE's UpdateManager — the pool that applies rowsets for primary-key tables. A value of 0 disables a fixed minimum (no enforced lower bound); when transaction\_apply\_worker\_count is also 0 the pool's max threads defaults to the number of CPU cores, so effective worker capacity equals CPU cores. You can raise this to guarantee a baseline concurrency for applying transactions; setting it too high may increase CPU contention. Changes are applied at runtime via the update\_config HTTP handler (it calls update\_min\_threads on the apply thread pool). * Introduced in: v3.2.11 ##### transaction\_publish\_version\_thread\_pool\_num\_min[​](#transaction_publish_version_thread_pool_num_min "Direct link to transaction_publish_version_thread_pool_num_min") * Default: 0 * Type: Int * Unit: Threads * Is mutable: Yes * Description: Sets the minimum number of threads reserved in the AgentServer "publish\_version" dynamic thread pool (used to publish transaction versions / handle TTaskType::PUBLISH\_VERSION tasks). At startup the pool is created with min = max(config value, MIN\_TRANSACTION\_PUBLISH\_WORKER\_COUNT) (MIN\_TRANSACTION\_PUBLISH\_WORKER\_COUNT = 1), so the default 0 results in a minimum of 1 thread. Changing this value at runtime invokes the update callback to call ThreadPool::update\_min\_threads, raising or lowering the pool's guaranteed minimum (but not below the enforced minimum of 1). Coordinate with transaction\_publish\_version\_worker\_count (max threads) and transaction\_publish\_version\_thread\_pool\_idle\_time\_ms (idle timeout). * Introduced in: v3.2.11 ##### use\_mmap\_allocate\_chunk[​](#use_mmap_allocate_chunk "Direct link to use_mmap_allocate_chunk") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: When this item is set to `true`, the system allocates chunks using anonymous private mmap mappings (MAP\_ANONYMOUS | MAP\_PRIVATE) and frees them with munmap. Enabling this may create many virtual memory mappings, thus you must raise the kernel limit (as root user, running `sysctl -w vm.max_map_count=262144` or `echo 262144 > /proc/sys/vm/max_map_count`), and set `chunk_reserved_bytes_limit` to a relatively large value. Otherwise, enabling mmap can cause very poor performance due to frequent mapping/unmapping. * Introduced in: v3.2.0 ##### ssl\_certificate\_path[​](#ssl_certificate_path "Direct link to ssl_certificate_path") * Default: * Type: String * Unit: - * Is mutable: No * Description: Absolute path to the TLS/SSL certificate file (PEM) that the BE's brpc server will use when enable\_https is true. At BE startup this value is copied into `brpc::ServerOptions::ssl_options().default_cert.certificate`; you must also set `ssl_private_key_path` to the matching private key. Provide the server certificate and any intermediate certificates in PEM format (certificate chain) if required by your CA. The file must be readable by the StarRocks BE process and is applied only at startup. If unset or invalid while enable\_https is enabled, brpc TLS setup may fail and prevent the server from starting correctly. * Introduced in: v4.0.0 #### Metadata and cluster management[​](#metadata-and-cluster-management "Direct link to Metadata and cluster management") ##### cluster\_id[​](#cluster_id "Direct link to cluster_id") * Default: -1 * Type: Int * Unit: - * Is mutable: No * Description: Global cluster identifier for this StarRocks backend. At startup StorageEngine reads config::cluster\_id into its effective cluster id and verifies that all data root paths contain the same cluster id (see StorageEngine::\_check\_all\_root\_path\_cluster\_id). A value of -1 means "unset" — the engine may derive the effective id from existing data directories or from master heartbeats. If a non‑negative id is configured, any mismatch between configured id and ids stored in data directories will cause startup verification to fail (Status::Corruption). When some roots lack an id and the engine is allowed to write ids (options.need\_write\_cluster\_id), it will persist the effective id into those roots. * Introduced in: v3.2.0 ##### consistency\_max\_memory\_limit[​](#consistency_max_memory_limit "Direct link to consistency_max_memory_limit") * Default: 10G * Type: String * Unit: - * Is mutable: No * Description: Memory size specification for the CONSISTENCY memory tracker. * Introduced in: v3.2.0 ##### make\_snapshot\_rpc\_timeout\_ms[​](#make_snapshot_rpc_timeout_ms "Direct link to make_snapshot_rpc_timeout_ms") * Default: 20000 * Type: Int * Unit: Milliseconds * Is mutable: No * Description: Sets the Thrift RPC timeout in milliseconds used when making a snapshot on a remote BE. Increase this value when remote snapshot creation regularly exceeds the default timeout; reduce it to fail faster on unresponsive BEs. Note other timeouts may affect end-to-end operations (for example the effective tablet-writer open timeout can relate to `tablet_writer_open_rpc_timeout_sec` and `load_timeout_sec`). * Introduced in: v3.2.0 ##### metadata\_cache\_memory\_limit\_percent[​](#metadata_cache_memory_limit_percent "Direct link to metadata_cache_memory_limit_percent") * Default: 30 * Type: Int * Unit: Percent * Is mutable: Yes * Description: Sets the metadata LRU cache size as a percentage of the process memory limit. At startup StarRocks computes cache bytes as (process\_mem\_limit \* metadata\_cache\_memory\_limit\_percent / 100) and passes that to the metadata cache allocator. The cache is only used for non-PRIMARY\_KEYS rowsets (PK tables are not supported) and is enabled only when `metadata_cache_memory_limit_percent > 0`; set it to `<= 0` to disable the metadata cache. Increasing this value raises metadata cache capacity but reduces memory available to other components; tune based on workload and system memory. Not active in BE\_TEST builds. * Introduced in: v3.2.10 ##### retry\_apply\_interval\_second[​](#retry_apply_interval_second "Direct link to retry_apply_interval_second") * Default: 30 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Base interval (in seconds) used when scheduling retries of failed tablet apply operations. It is used directly to schedule a retry after a submission failure and as the base multiplier for backoff: the next retry delay is calculated as min(600, `retry_apply_interval_second` \* failed\_attempts). The code also uses `retry_apply_interval_second` to compute the cumulative retry duration (arithmetic-series sum) which is compared against `retry_apply_timeout_second` to decide whether to keep retrying. Effective only when `enable_retry_apply` is true. Increasing this value lengthens both individual retry delays and the cumulative time spent retrying; decreasing it makes retries more frequent and may increase the number of attempts before reaching `retry_apply_timeout_second`. * Introduced in: v3.2.9 ##### retry\_apply\_timeout\_second[​](#retry_apply_timeout_second "Direct link to retry_apply_timeout_second") * Default: 7200 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Maximum cumulative retry time (in seconds) allowed for applying a pending version before the apply process gives up and the tablet enters an error state. The apply logic accumulates exponential/backoff intervals based on `retry_apply_interval_second` and compares the total duration against `retry_apply_timeout_second`. If `enable_retry_apply` is true and the error is considered retryable, apply attempts will be rescheduled until the accumulated backoff exceeds `retry_apply_timeout_second`; then apply stops and the tablet transitions to error. Explicitly non-retryable errors (e.g., Corruption) are not retried regardless of this setting. Tune this value to control how long StarRocks will keep retrying apply operations (default 7200s = 2 hours). * Introduced in: v3.3.13, v3.4.3, v3.5.0 ##### stream\_load\_thrift\_rpc\_timeout\_ms[​](#stream_load_thrift_rpc_timeout_ms "Direct link to stream_load_thrift_rpc_timeout_ms") * Default: 60000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: Maximum allowed lifetime (in milliseconds) for Thrift RPC connections used by BE stream-load and transaction commit calls. StarRocks sets this value as the `thrift_rpc_timeout_ms` on requests sent to FE (used in stream\_load planning, loadTxnBegin/loadTxnPrepare/loadTxnCommit, and getLoadTxnStatus). If a connection has been pooled longer than this value it will be closed. When a per-request timeout (`ctx->timeout_second`) is provided, the BE computes the RPC timeout as rpc\_timeout\_ms = max(ctx*1000/4, min(ctx*1000/2, stream\_load\_thrift\_rpc\_timeout\_ms)), so the effective RPC timeout is bounded by the context and this configuration. Keep this consistent with FE's `thrift_client_timeout_ms` to avoid mismatched timeouts. The legacy name `txn_commit_rpc_timeout_ms` is still accepted as a backward-compatible alias. * Introduced in: v3.2.0 ##### txn\_map\_shard\_size[​](#txn_map_shard_size "Direct link to txn_map_shard_size") * Default: 128 * Type: Int * Unit: - * Is mutable: No * Description: Number of lock-map shards used by the transaction manager to partition transaction locks and reduce contention. Its value should be a power of two (2^n); increasing it augments concurrency and reduces lock contention at the cost of additional memory and marginal bookkeeping overhead. Choose a shard count sized for expected concurrent transactions and available memory. * Introduced in: v3.2.0 ##### txn\_shard\_size[​](#txn_shard_size "Direct link to txn_shard_size") * Default: 1024 * Type: Int * Unit: - * Is mutable: No * Description: Controls the number of lock shards used by the transaction manager. This value determines the shard size for txn locks. It must be a power of two; Setting it to a larger value reduces lock contention and improves concurrent COMMIT/PUBLISH throughput at the expense of additional memory and finer-grained internal bookkeeping. * Introduced in: v3.2.0 ##### update\_schema\_worker\_count[​](#update_schema_worker_count "Direct link to update_schema_worker_count") * Default: 3 * Type: Int * Unit: Threads * Is mutable: No * Description: Sets the maximum number of worker threads in the backend's "update\_schema" dynamic ThreadPool that processes TTaskType::UPDATE\_SCHEMA tasks. The ThreadPool is created in agent\_server during startup with a minimum of 0 threads (it can scale down to zero when idle) and a max equal to this setting; the pool uses the default idle timeout and an effectively unlimited queue. Increase this value to allow more concurrent schema-update tasks (higher CPU and memory usage), or lower it to limit parallel schema operations. * Introduced in: v3.2.3 ##### update\_tablet\_meta\_info\_worker\_count[​](#update_tablet_meta_info_worker_count "Direct link to update_tablet_meta_info_worker_count") * Default: 1 * Type: Int * Unit: - * Is mutable: Yes * Description: Sets the maximum number of worker threads in the backend thread pool that handles tablet metadata update tasks. The thread pool is created during backend startup with a minimum of 0 threads (it can scale down to zero when idle) and a max equal to this setting (clamped to at least 1). Updating this value at runtime adjusts the pool's max threads. Increase it to allow more concurrent metadata-update tasks, or lower it to limit concurrency. * Introduced in: v4.1.0, v4.0.6, v3.5.13 --- ### BE Configuration - Query and Loading Some BE configuration items are dynamic parameters which you can set interactively when BE nodes are online. The rest of them are static parameters. You can only set the static parameters of a BE node by changing them in the corresponding configuration file **be.conf** and restarting the BE node to allow the change to take effect. #### View BE configuration items[​](#view-be-configuration-items "Direct link to View BE configuration items") You can view the BE configuration items using the following command: ```sql SELECT * FROM information_schema.be_configs [WHERE NAME LIKE "%%"] ``` #### Configure BE parameters[​](#configure-be-parameters "Direct link to Configure BE parameters") ##### Configure BE dynamic parameters[​](#configure-be-dynamic-parameters "Direct link to Configure BE dynamic parameters") You can configure a dynamic parameter of a BE node by updating the value in `information_schema.be_configs`. warning Setting an invalid value may cause unknown behaviors. Check twice before you run the command to update the configuration. ```sql -- Replace with the key of the configuration and with the value. UPDATE information_schema.be_configs SET VALUE = WHERE name = ""; ``` ##### Configure BE static parameters[​](#configure-be-static-parameters "Direct link to Configure BE static parameters") You can only set the static parameters of a BE by changing them in the corresponding configuration file **be.conf**, and restarting the BE to allow the changes to take effect. *** This topic introduces the following types of BE configurations: * [Query](#query) * [Loading and unloading](#loading-and-unloading) #### Query[​](#query "Direct link to Query") ##### agg\_hash\_map\_prefetch\_dist[​](#agg_hash_map_prefetch_dist "Direct link to agg_hash_map_prefetch_dist") * Default: 16 * Type: Int * Unit: Rows * Is mutable: Yes * Description: Software prefetch distance (in rows) for the aggregation hash-map / hash-set probe loop. While building the aggregation hash table, the loop prefetches the bucket for the row this many positions ahead of the one it is currently processing, hiding memory latency on large tables. Setting it to `0` disables software prefetch. The value is read once per chunk, so changes take effect on the next chunk. The default 16 is empirical for L3-resident tables; raise it for DRAM-resident workloads and lower it for cache-resident ones. Prefetch is additionally gated by `agg_prefetch_l2_ratio`: regardless of this distance, no prefetch is issued while the hash table still fits in L2. * Introduced in: - ##### agg\_prefetch\_l2\_ratio[​](#agg_prefetch_l2_ratio "Direct link to agg_prefetch_l2_ratio") * Default: 1.0 * Type: Double * Unit: - * Is mutable: Yes * Description: Gates aggregation hash-table software prefetch on L2 residency. Prefetch is enabled only once the bucket array spills L2, that is, when `bucket_count * slot_bytes >= L2_size * agg_prefetch_l2_ratio`, where the L2 size is detected at runtime (falling back to 1 MiB if detection fails). Below this point the table is L2-resident and prefetching is a net loss. Lower the ratio on contended deployments that run many drivers per core, where the effective per-table share of L2 is smaller than the nominal per-core size; raising it above 1.0 delays prefetch until the table is well past L2. See also `agg_hash_map_prefetch_dist`. * Introduced in: - ##### clear\_udf\_cache\_when\_start[​](#clear_udf_cache_when_start "Direct link to clear_udf_cache_when_start") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: When enabled, the BE's UserFunctionCache will clear all locally cached user function libraries on startup. During UserFunctionCache::init, the code calls \_reset\_cache\_dir(), which removes UDF files from the configured UDF library directory (organized into kLibShardNum subdirectories) and deletes files with Java/Python UDF suffixes (.jar/.py). When disabled (default), the BE loads existing cached UDF files instead of deleting them. Enabling this forces UDF binaries to be re-downloaded on first use after restart (increasing network traffic and first-use latency). * Introduced in: v4.0.0 ##### dictionary\_speculate\_min\_chunk\_size[​](#dictionary_speculate_min_chunk_size "Direct link to dictionary_speculate_min_chunk_size") * Default: 10000 * Type: Int * Unit: Rows * Is mutable: No * Description: Minimum number of rows (chunk size) used by StringColumnWriter and DictColumnWriter to trigger dictionary-encoding speculation. If an incoming column (or the accumulated buffer plus incoming rows) has size larger than or equal `dictionary_speculate_min_chunk_size` the writer will run speculation immediately and set an encoding (DICT, PLAIN or BIT\_SHUFFLE) rather than buffering more rows. Speculation uses `dictionary_encoding_ratio` for string columns and `dictionary_encoding_ratio_for_non_string_column` for numeric/non-string columns to decide whether dictionary encoding is beneficial. Also, a large column byte\_size (larger than or equal to UINT32\_MAX) forces immediate speculation to avoid `BinaryColumn` overflow. * Introduced in: v3.2.0 ##### disable\_storage\_page\_cache[​](#disable_storage_page_cache "Direct link to disable_storage_page_cache") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: A boolean value to control whether to disable PageCache. * When PageCache is enabled, StarRocks caches the recently scanned data. * PageCache can significantly improve the query performance when similar queries are repeated frequently. * `true` indicates disabling PageCache. * The default value of this item has been changed from `true` to `false` since StarRocks v2.4. * Introduced in: - ##### enable\_bitmap\_index\_memory\_page\_cache[​](#enable_bitmap_index_memory_page_cache "Direct link to enable_bitmap_index_memory_page_cache") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable memory cache for Bitmap index. Memory cache is recommended if you want to use Bitmap indexes to accelerate point queries. * Introduced in: v3.1 ##### enable\_compaction\_flat\_json[​](#enable_compaction_flat_json "Direct link to enable_compaction_flat_json") * Default: True * Type: Boolean * Unit: * Is mutable: Yes * Description: Whether to enable compaction for Flat JSON data. * Introduced in: v3.3.3 ##### enable\_json\_flat[​](#enable_json_flat "Direct link to enable_json_flat") * Default: false * Type: Boolean * Unit: * Is mutable: Yes * Description: Whether to enable the Flat JSON feature. After this feature is enabled, newly loaded JSON data will be automatically flattened, improving JSON query performance. * Introduced in: v3.3.0 ##### enable\_lazy\_dynamic\_flat\_json[​](#enable_lazy_dynamic_flat_json "Direct link to enable_lazy_dynamic_flat_json") * Default: True * Type: Boolean * Unit: * Is mutable: Yes * Description: Whether to enable Lazy Dyamic Flat JSON when a query misses Flat JSON schema in read process. When this item is set to `true`, StarRocks will postpone the Flat JSON operation to calculation process instead of read process. * Introduced in: v3.3.3 ##### enable\_ordinal\_index\_memory\_page\_cache[​](#enable_ordinal_index_memory_page_cache "Direct link to enable_ordinal_index_memory_page_cache") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable memory cache for ordinal index. Ordinal index is a mapping from row IDs to data page positions, and it can be used to accelerate scans. * Introduced in: - ##### enable\_string\_prefix\_zonemap[​](#enable_string_prefix_zonemap "Direct link to enable_string_prefix_zonemap") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable ZoneMap for string (CHAR/VARCHAR) columns using prefix-based min/max. For non-key string columns, the min/max values are truncated to a fixed prefix length configured by `string_prefix_zonemap_prefix_len`. * Introduced in: - ##### enable\_zonemap\_index\_memory\_page\_cache[​](#enable_zonemap_index_memory_page_cache "Direct link to enable_zonemap_index_memory_page_cache") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable memory cache for zonemap index. Memory cache is recommended if you want to use zonemap indexes to accelerate scan. * Introduced in: - ##### exchg\_node\_buffer\_size\_bytes[​](#exchg_node_buffer_size_bytes "Direct link to exchg_node_buffer_size_bytes") * Default: 10485760 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: The maximum buffer size on the receiver end of an exchange node for each query. This configuration item is a soft limit. A backpressure is triggered when data is sent to the receiver end with an excessive speed. * Introduced in: - ##### exec\_state\_report\_max\_threads[​](#exec_state_report_max_threads "Direct link to exec_state_report_max_threads") * Default: 2 * Type: Int * Unit: Threads * Is mutable: Yes * Description: Maximum number of threads for the exec-state-report thread pool. This pool is used by `ExecStateReporter` to asynchronously send non-priority execution status reports (such as fragment completion and error status) from BE to FE via RPC. The actual pool size at startup is `max(1, exec_state_report_max_threads)`. Changing this config at runtime triggers `update_max_threads` on the pool in every executor set (shared and exclusive). The pool has a fixed task queue size of 1000; report submissions are silently dropped when all threads are busy and the queue is full. Paired with `priority_exec_state_report_max_threads` for the high-priority pool. Increase this value when delayed or dropped exec-state reports are observed under high query concurrency. * Introduced in: v4.1.0, v4.0.8, v3.5.15 ##### file\_descriptor\_cache\_capacity[​](#file_descriptor_cache_capacity "Direct link to file_descriptor_cache_capacity") * Default: 16384 * Type: Int * Unit: - * Is mutable: No * Description: The number of file descriptors that can be cached. * Introduced in: - ##### flamegraph\_tool\_dir[​](#flamegraph_tool_dir "Direct link to flamegraph_tool_dir") * Default: `${STARROCKS_HOME}/bin/flamegraph` * Type: String * Unit: - * Is mutable: No * Description: Directory of the flamegraph tool, which should contain pprof, stackcollapse-go.pl, and flamegraph.pl scripts for generating flame graphs from profile data. * Introduced in: - ##### fragment\_pool\_queue\_size[​](#fragment_pool_queue_size "Direct link to fragment_pool_queue_size") * Default: 2048 * Type: Int * Unit: - * Is mutable: No * Description: The upper limit of the query number that can be processed on each BE node. * Introduced in: - ##### fragment\_pool\_thread\_num\_max[​](#fragment_pool_thread_num_max "Direct link to fragment_pool_thread_num_max") * Default: 4096 * Type: Int * Unit: - * Is mutable: No * Description: The maximum number of threads used for query. * Introduced in: - ##### fragment\_pool\_thread\_num\_min[​](#fragment_pool_thread_num_min "Direct link to fragment_pool_thread_num_min") * Default: 64 * Type: Int * Unit: Minutes - * Is mutable: No * Description: The minimum number of threads used for query. * Introduced in: - ##### hdfs\_client\_enable\_hedged\_read[​](#hdfs_client_enable_hedged_read "Direct link to hdfs_client_enable_hedged_read") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: Specifies whether to enable the hedged read feature. * Introduced in: v3.0 ##### hdfs\_client\_hedged\_read\_threadpool\_size[​](#hdfs_client_hedged_read_threadpool_size "Direct link to hdfs_client_hedged_read_threadpool_size") * Default: 128 * Type: Int * Unit: - * Is mutable: No * Description: Specifies the size of the Hedged Read thread pool on your HDFS client. The thread pool size limits the number of threads to dedicate to the running of hedged reads in your HDFS client. It is equivalent to the `dfs.client.hedged.read.threadpool.size` parameter in the **hdfs-site.xml** file of your HDFS cluster. * Introduced in: v3.0 ##### hdfs\_client\_hedged\_read\_threshold\_millis[​](#hdfs_client_hedged_read_threshold_millis "Direct link to hdfs_client_hedged_read_threshold_millis") * Default: 2500 * Type: Int * Unit: Milliseconds * Is mutable: No * Description: Specifies the number of milliseconds to wait before starting up a hedged read. For example, you have set this parameter to `30`. In this situation, if a read from a block has not returned within 30 milliseconds, your HDFS client immediately starts up a new read against a different block replica. It is equivalent to the `dfs.client.hedged.read.threshold.millis` parameter in the **hdfs-site.xml** file of your HDFS cluster. * Introduced in: v3.0 ##### io\_coalesce\_adaptive\_lazy\_active[​](#io_coalesce_adaptive_lazy_active "Direct link to io_coalesce_adaptive_lazy_active") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Based on the selectivity of predicates, adaptively determines whether to combine the I/O of predicate columns and non-predicate columns. * Introduced in: v3.2 ##### jit\_lru\_cache\_size[​](#jit_lru_cache_size "Direct link to jit_lru_cache_size") * Default: 0 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: The LRU cache size for JIT compilation. It represents the actual size of the cache if it is set to greater than 0. If it is set to less than or equal to 0, the system will adaptively set the cache using the formula `jit_lru_cache_size = min(mem_limit*0.01, 1GB)` (while `mem_limit` of the node must be greater or equal to 16 GB). * Introduced in: - ##### json\_flat\_column\_max[​](#json_flat_column_max "Direct link to json_flat_column_max") * Default: 100 * Type: Int * Unit: * Is mutable: Yes * Description: The maximum number of sub-fields that can be extracted by Flat JSON. This parameter takes effect only when `enable_json_flat` is set to `true`. * Introduced in: v3.3.0 ##### json\_flat\_create\_zonemap[​](#json_flat_create_zonemap "Direct link to json_flat_create_zonemap") * Default: true * Type: Boolean * Unit: * Is mutable: Yes * Description: Whether to create ZoneMaps for flattened JSON sub-columns during write. This parameter takes effect only when `enable_json_flat` is set to `true`. * Introduced in: - ##### json\_flat\_null\_factor[​](#json_flat_null_factor "Direct link to json_flat_null_factor") * Default: 0.3 * Type: Double * Unit: * Is mutable: Yes * Description: The proportion of NULL values in the column to extract for Flat JSON. A column will not be extracted if its proportion of NULL value is higher than this threshold. This parameter takes effect only when `enable_json_flat` is set to `true`. * Introduced in: v3.3.0 ##### json\_flat\_sparsity\_factor[​](#json_flat_sparsity_factor "Direct link to json_flat_sparsity_factor") * Default: 0.3 * Type: Double * Unit: * Is mutable: Yes * Description: The proportion of columns with the same name for Flat JSON. Extraction is not performed if the proportion of columns with the same name is lower than this value. This parameter takes effect only when `enable_json_flat` is set to `true`. * Introduced in: v3.3.0 ##### lake\_tablet\_ignore\_invalid\_delete\_predicate[​](#lake_tablet_ignore_invalid_delete_predicate "Direct link to lake_tablet_ignore_invalid_delete_predicate") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: A boolean value to control whether ignore invalid delete predicates in tablet rowset metadata which may be introduced by logic deletion to a duplicate key table after the column name renamed. * Introduced in: v4.0 ##### late\_materialization\_ratio[​](#late_materialization_ratio "Direct link to late_materialization_ratio") * Default: 10 * Type: Int * Unit: - * Is mutable: No * Description: Integer ratio in range \[0-1000] that controls the use of late materialization in the SegmentIterator (vector query engine). A value of `0` (or ≤ 0) disables late materialization; `1000` (or ≥ 1000) forces late materialization for all reads. Values > 0 and < 1000 enable a conditional strategy where both late and early materialization contexts are prepared and the iterator selects behavior based on predicate filter ratios (higher values favor late materialization). When a segment contains complex metric types, StarRocks uses `metric_late_materialization_ratio` instead. If `lake_io_opts.cache_file_only` is set, late materialization is disabled. * Introduced in: v3.2.0 ##### max\_hdfs\_file\_handle[​](#max_hdfs_file_handle "Direct link to max_hdfs_file_handle") * Default: 1000 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of HDFS file descriptors that can be opened. * Introduced in: - ##### max\_hdfs\_scanner\_num[​](#max_hdfs_scanner_num "Direct link to max_hdfs_scanner_num") * Default: 50 * Type: Int * Unit: - * Is mutable: No * Description: Maximum number of concurrent remote scanners (HDFS, object storage, etc.) that ConnectorScanNode can run simultaneously. This value caps estimated concurrency at startup and also limits pending-scanner scheduling at runtime, controlling thread, memory, and file-handle pressure. * Introduced in: v3.2.0 ##### max\_memory\_sink\_batch\_count[​](#max_memory_sink_batch_count "Direct link to max_memory_sink_batch_count") * Default: 20 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of Scan Cache batches. * Introduced in: - ##### max\_pushdown\_conditions\_per\_column[​](#max_pushdown_conditions_per_column "Direct link to max_pushdown_conditions_per_column") * Default: 1024 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of conditions that allow pushdown in each column. If the number of conditions exceeds this limit, the predicates are not pushed down to the storage layer. * Introduced in: - ##### max\_scan\_key\_num[​](#max_scan_key_num "Direct link to max_scan_key_num") * Default: 1024 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of scan keys segmented by each query. * Introduced in: - ##### metric\_late\_materialization\_ratio[​](#metric_late_materialization_ratio "Direct link to metric_late_materialization_ratio") * Default: 1000 * Type: Int * Unit: - * Is mutable: No * Description: Controls when the late-materialization row access strategy is used for reads that include complex metric columns. Valid range: \[0-1000]. `0` disables late materialization; `1000` forces late materialization for all applicable reads. Values 1–999 enable a conditional strategy where both late and early materialization contexts are prepared and chosen at runtime based on predicate/selectivity. When complex metric types exist, `metric_late_materialization_ratio` overrides the general `late_materialization_ratio`. Note: `cache_file_only` I/O mode will cause late materialization to be disabled regardless of this setting. * Introduced in: v3.2.0 ##### min\_file\_descriptor\_number[​](#min_file_descriptor_number "Direct link to min_file_descriptor_number") * Default: 60000 * Type: Int * Unit: - * Is mutable: No * Description: The minimum number of file descriptors in the BE process. * Introduced in: - ##### object\_storage\_client\_cache\_size[​](#object_storage_client_cache_size "Direct link to object_storage_client_cache_size") * Default: 8 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of object storage clients (S3-compatible and Azure Blob) cached per client factory. The value is read on each client creation, so lowering it takes effect gradually as cached clients are evicted during subsequent creations. Values below `1` are treated as `1`. * Introduced in: v4.1.4, v4.0.14 ##### object\_storage\_connect\_timeout\_ms[​](#object_storage_connect_timeout_ms "Direct link to object_storage_connect_timeout_ms") * Default: -1 * Type: Int * Unit: Milliseconds * Is mutable: No * Description: Timeout duration to establish socket connections with object storage. `-1` indicates to use the default timeout duration of the SDK configurations. * Introduced in: v3.0.9 ##### object\_storage\_request\_timeout\_ms[​](#object_storage_request_timeout_ms "Direct link to object_storage_request_timeout_ms") * Default: -1 * Type: Int * Unit: Milliseconds * Is mutable: No * Description: Timeout duration to establish HTTP connections with object storage. `-1` indicates to use the default timeout duration of the SDK configurations. * Introduced in: v3.0.9 ##### parquet\_late\_materialization\_enable[​](#parquet_late_materialization_enable "Direct link to parquet_late_materialization_enable") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: A boolean value to control whether to enable the late materialization of Parquet reader to improve performance. `true` indicates enabling late materialization, and `false` indicates disabling it. * Introduced in: - ##### parquet\_page\_index\_enable[​](#parquet_page_index_enable "Direct link to parquet_page_index_enable") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: A boolean value to control whether to enable the pageindex of Parquet file to improve performance. `true` indicates enabling pageindex, and `false` indicates disabling it. * Introduced in: v3.3 ##### parquet\_reader\_bloom\_filter\_enable[​](#parquet_reader_bloom_filter_enable "Direct link to parquet_reader_bloom_filter_enable") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: A boolean value to control whether to enable the bloom filter of Parquet file to improve performance. `true` indicates enabling the bloom filter, and `false` indicates disabling it. You can also control this behavior on session level using the system variable `enable_parquet_reader_bloom_filter`. Bloom filters in Parquet are maintained **at the column level within each row group**. If a Parquet file contains bloom filters for certain columns, queries can use predicates on those columns to efficiently skip row groups. * Introduced in: v3.5 ##### path\_gc\_check\_step[​](#path_gc_check_step "Direct link to path_gc_check_step") * Default: 1000 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of files that can be scanned continuously each time. * Introduced in: - ##### path\_gc\_check\_step\_interval\_ms[​](#path_gc_check_step_interval_ms "Direct link to path_gc_check_step_interval_ms") * Default: 10 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: The time interval between file scans. * Introduced in: - ##### path\_scan\_interval\_second[​](#path_scan_interval_second "Direct link to path_scan_interval_second") * Default: 86400 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval at which GC cleans expired data. * Introduced in: - ##### pipeline\_connector\_scan\_thread\_num\_per\_cpu[​](#pipeline_connector_scan_thread_num_per_cpu "Direct link to pipeline_connector_scan_thread_num_per_cpu") * Default: 8 * Type: Double * Unit: - * Is mutable: Yes * Description: The number of scan threads assigned to Pipeline Connector per CPU core in the BE node. This configuration is changed to dynamic from v3.1.7 onwards. * Introduced in: - ##### pipeline\_enable\_large\_column\_checker[​](#pipeline_enable_large_column_checker "Direct link to pipeline_enable_large_column_checker") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable large column detection in the pipeline execution framework. When enabled, queries fail with a capacity limit error if an intermediate column reaches the chunk capacity limit in pipeline execution or spill serialization. * Introduced in: v4.0.0 ##### pipeline\_poller\_timeout\_guard\_ms[​](#pipeline_poller_timeout_guard_ms "Direct link to pipeline_poller_timeout_guard_ms") * Default: -1 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: When this item is set to greater than `0`, if a driver takes longer than `pipeline_poller_timeout_guard_ms` for a single dispatch in the poller, then the information of the driver and operator is printed. * Introduced in: - ##### pipeline\_prepare\_thread\_pool\_queue\_size[​](#pipeline_prepare_thread_pool_queue_size "Direct link to pipeline_prepare_thread_pool_queue_size") * Default: 102400 * Type: Int * Unit: - * Is mutable: No * Description: The maximum queue lenggth of PREPARE fragment thread pool for Pipeline execution engine. * Introduced in: - ##### pipeline\_prepare\_thread\_pool\_thread\_num[​](#pipeline_prepare_thread_pool_thread_num "Direct link to pipeline_prepare_thread_pool_thread_num") * Default: 0 * Type: Int * Unit: - * Is mutable: No * Description: Number of threads in the pipeline execution engine PREPARE fragment thread pool. `0` indicates the value is equal to the number of system VCPU core number. * Introduced in: - ##### pipeline\_prepare\_timeout\_guard\_ms[​](#pipeline_prepare_timeout_guard_ms "Direct link to pipeline_prepare_timeout_guard_ms") * Default: -1 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: When this item is set to greater than `0`, if a plan fragment exceeds `pipeline_prepare_timeout_guard_ms` during the PREPARE process, a stack trace of the plan fragment is printed. * Introduced in: - ##### pipeline\_scan\_thread\_pool\_queue\_size[​](#pipeline_scan_thread_pool_queue_size "Direct link to pipeline_scan_thread_pool_queue_size") * Default: 102400 * Type: Int * Unit: - * Is mutable: No * Description: The maximum task queue length of SCAN thread pool for Pipeline execution engine. * Introduced in: - ##### pk\_index\_parallel\_get\_threadpool\_size[​](#pk_index_parallel_get_threadpool_size "Direct link to pk_index_parallel_get_threadpool_size") * Default: 1048576 * Type: Int * Unit: - * Is mutable: Yes * Description: Sets the maximum queue size (number of pending tasks) for the "cloud\_native\_pk\_index\_get" thread pool used by PK index parallel get operations in shared-data (cloud-native/lake) mode. The actual thread count for that pool is controlled by `pk_index_parallel_get_threadpool_max_threads`; this setting only limits how many tasks may be queued awaiting execution. The very large default (2^20) effectively makes the queue unbounded; lowering it prevents excessive memory growth from queued tasks but may cause task submissions to block or fail when the queue is full. Tune together with `pk_index_parallel_get_threadpool_max_threads` based on workload concurrency and memory constraints. * Introduced in: - ##### priority\_exec\_state\_report\_max\_threads[​](#priority_exec_state_report_max_threads "Direct link to priority_exec_state_report_max_threads") * Default: 2 * Type: Int * Unit: Threads * Is mutable: Yes * Description: Maximum number of threads for the high-priority exec-state-report thread pool. This pool is used by `ExecStateReporter` to asynchronously send high-priority execution status reports (such as urgent fragment failures) from BE to FE via RPC. Unlike the normal exec-state-report pool, this pool has an unbounded task queue. The actual pool size at startup is `max(1, priority_exec_state_report_max_threads)`. Changing this config at runtime triggers `update_max_threads` on the priority pool in every executor set (shared and exclusive). Paired with `exec_state_report_max_threads` for the normal pool. Increase this value when high-priority reports are delayed under heavy concurrent query loads. * Introduced in: v4.1.0, v4.0.8, v3.5.15 ##### priority\_queue\_remaining\_tasks\_increased\_frequency[​](#priority_queue_remaining_tasks_increased_frequency "Direct link to priority_queue_remaining_tasks_increased_frequency") * Default: 512 * Type: Int * Unit: - * Is mutable: Yes * Description: Controls how often the BlockingPriorityQueue increases ("ages") the priority of all remaining tasks to avoid starvation. Each successful get/pop increments an internal `_upgrade_counter`; when `_upgrade_counter` exceeds `priority_queue_remaining_tasks_increased_frequency`, the queue increments every element's priority, rebuilds the heap, and resets the counter. Lower values cause more frequent priority aging (reducing starvation but increasing CPU cost due to iterating and re-heapifying); higher values reduce that overhead but delay priority adjustments. The value is a simple operation count threshold, not a time duration. * Introduced in: v3.2.0 ##### query\_cache\_capacity[​](#query_cache_capacity "Direct link to query_cache_capacity") * Default: 536870912 * Type: Int * Unit: Bytes * Is mutable: No * Description: The size of the query cache in the BE. The default size is 512 MB. The size cannot be less than 4 MB. If the memory capacity of the BE is insufficient to provision your expected query cache size, you can increase the memory capacity of the BE. * Introduced in: - ##### query\_pool\_spill\_mem\_limit\_threshold[​](#query_pool_spill_mem_limit_threshold "Direct link to query_pool_spill_mem_limit_threshold") * Default: 1.0 * Type: Double * Unit: - * Is mutable: No * Description: If automatic spilling is enabled, when the memory usage of all queries exceeds `query_pool memory limit * query_pool_spill_mem_limit_threshold`, intermediate result spilling will be triggered. * Introduced in: v3.2.7 ##### query\_scratch\_dirs[​](#query_scratch_dirs "Direct link to query_scratch_dirs") * Default: `${STARROCKS_HOME}` * Type: string * Unit: - * Is mutable: No * Description: Comma-separated list of writable scratch directories used by query execution to spill intermediate data (for example, external sorts, hash joins, and other operators). Specify one or more paths separated by `;` (e.g. `/mnt/ssd1/tmp;/mnt/ssd2/tmp`). Directories should be accessible and writable by the BE process and have sufficient free space; StarRocks will pick among them to distribute spill I/O. Changes require a restart to take effect. If a directory is missing, not writable, or full, spilling may fail or degrade query performance. * Introduced in: v3.2.0 ##### result\_buffer\_cancelled\_interval\_time[​](#result_buffer_cancelled_interval_time "Direct link to result_buffer_cancelled_interval_time") * Default: 300 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The wait time before BufferControlBlock releases data. * Introduced in: - ##### scan\_context\_gc\_interval\_min[​](#scan_context_gc_interval_min "Direct link to scan_context_gc_interval_min") * Default: 5 * Type: Int * Unit: Minutes * Is mutable: Yes * Description: The time interval at which to clean the Scan Context. * Introduced in: - ##### scanner\_row\_num[​](#scanner_row_num "Direct link to scanner_row_num") * Default: 16384 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum row count returned by each scan thread in a scan. * Introduced in: - ##### scanner\_thread\_pool\_queue\_size[​](#scanner_thread_pool_queue_size "Direct link to scanner_thread_pool_queue_size") * Default: 102400 * Type: Int * Unit: - * Is mutable: No * Description: The number of scan tasks supported by the storage engine. * Introduced in: - ##### scanner\_thread\_pool\_thread\_num[​](#scanner_thread_pool_thread_num "Direct link to scanner_thread_pool_thread_num") * Default: 48 * Type: Int * Unit: - * Is mutable: Yes * Description: The number of threads which the storage engine used for concurrent storage volume scanning. All threads are managed in the thread pool. * Introduced in: - ##### string\_prefix\_zonemap\_prefix\_len[​](#string_prefix_zonemap_prefix_len "Direct link to string_prefix_zonemap_prefix_len") * Default: 16 * Type: Int * Unit: - * Is mutable: Yes * Description: Prefix length used for string ZoneMap min/max when `enable_string_prefix_zonemap` is enabled. * Introduced in: - ##### udf\_thread\_pool\_size[​](#udf_thread_pool_size "Direct link to udf_thread_pool_size") * Default: 1 * Type: Int * Unit: Threads * Is mutable: No * Description: Sets the size of the UDF call PriorityThreadPool created in ExecEnv (used for executing user-defined functions / UDF-related tasks). The value is used as the pool thread count and also as the pool queue capacity when constructing the thread pool (PriorityThreadPool("udf", thread\_num, queue\_size)). Increase to allow more concurrent UDF executions; keep small to avoid excessive CPU and memory contention. * Introduced in: v3.2.0 ##### update\_memory\_limit\_percent[​](#update_memory_limit_percent "Direct link to update_memory_limit_percent") * Default: 60 * Type: Int * Unit: Percent * Is mutable: No * Description: Fraction of the BE process memory reserved for update-related memory and caches. During startup `GlobalEnv` computes the `MemTracker` for updates as process\_mem\_limit \* clamp(update\_memory\_limit\_percent, 0, 100) / 100. `UpdateManager` also uses this percentage to size its primary-index/index-cache capacity (index cache capacity = GlobalEnv::process\_mem\_limit \* update\_memory\_limit\_percent / 100). The HTTP config update logic registers a callback that calls `update_primary_index_memory_limit` on the update managers, so changes would be applied to the update subsystem if the config were changed. Increasing this value gives more memory to update/primary-index paths (reducing memory available for other pools); decreasing it reduces update memory and cache capacity. Values are clamped to the range 0–100. * Introduced in: v3.2.0 ##### vector\_chunk\_size[​](#vector_chunk_size "Direct link to vector_chunk_size") * Default: 4096 * Type: Int * Unit: Rows * Is mutable: No * Description: The number of rows per vectorized chunk (batch) used throughout the execution and storage code paths. This value controls Chunk and RuntimeState batch\_size creation, affects operator throughput, memory footprint per operator, spill and sort buffer sizing, and I/O heuristics (for example, ORC writer natural write size). Increasing it can improve CPU and I/O efficiency for wide/CPU-bound workloads but raises peak memory usage and can increase latency for small-result queries. Tune only when profiling shows batch-size is a bottleneck; otherwise keep the default for balanced memory and performance. * Introduced in: v3.2.0 #### Loading and unloading[​](#loading-and-unloading "Direct link to Loading and unloading") ##### clear\_transaction\_task\_worker\_count[​](#clear_transaction_task_worker_count "Direct link to clear_transaction_task_worker_count") * Default: 1 * Type: Int * Unit: - * Is mutable: No * Description: The number of threads used for clearing transaction. * Introduced in: - ##### column\_mode\_partial\_update\_insert\_batch\_size[​](#column_mode_partial_update_insert_batch_size "Direct link to column_mode_partial_update_insert_batch_size") * Default: 4096 * Type: Int * Unit: - * Is mutable: Yes * Description: Batch size for column mode partial update when processing inserted rows. If this item is set to `0` or negative, it will be clamped to `1` to avoid infinite loop. This item controls the number of newly inserted rows processed in each batch. Larger values can improve write performance but will consume more memory. * Introduced in: v3.5.10, v4.0.2 ##### partial\_update\_memory\_limit\_per\_worker[​](#partial_update_memory_limit_per_worker "Direct link to partial_update_memory_limit_per_worker") * Default: 2147483648 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: Maximum memory per worker thread for partial update operations. Controls the memory footprint of individual worker threads when processing partial updates. * Introduced in: - ##### enable\_load\_spill\_parallel\_merge[​](#enable_load_spill_parallel_merge "Direct link to enable_load_spill_parallel_merge") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Specifies whether to enable parallel spill merge within a single tablet. Enabling this can improve the performance of spill merge during data loading. * Introduced in: - ##### enable\_parallel\_memtable\_finalize[​](#enable_parallel_memtable_finalize "Direct link to enable_parallel_memtable_finalize") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Specifies whether to enable parallel memtable finalize when loading data to lake tables (shared-data mode). When enabled, the memtable finalize operation (sort/aggregate) is moved from the write thread to the flush thread, allowing the write thread to continue inserting data into a new memtable while the previous one is being finalized and flushed in parallel. This can significantly improve load throughput by overlapping CPU-intensive finalize operations with I/O-bound flush operations. Note that this optimization is automatically disabled when auto-increment columns need to be filled, as auto-increment ID assignment must happen before the memtable is submitted for flush. * Introduced in: - ##### allow\_list\_object\_for\_random\_bucketing\_on\_cache\_miss[​](#allow_list_object_for_random_bucketing_on_cache_miss "Direct link to allow_list_object_for_random_bucketing_on_cache_miss") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Controls whether to allow object-storage LIST fallback when lake metadata cache misses during random bucketing size checks. `true` means fallback to LIST metadata files to compute base size (historical behavior, more accurate size estimation). `false` means skip LIST and use `base_size = 0`, which reduces LIST object requests but may delay immutable marking due to less accurate size estimation. * Introduced in: 4.1.0, 4.0.7, 3.5.15 ##### enable\_stream\_load\_verbose\_log[​](#enable_stream_load_verbose_log "Direct link to enable_stream_load_verbose_log") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Specifies whether to log the HTTP requests and responses for Stream Load jobs. * Introduced in: v2.5.17, v3.0.9, v3.1.6, v3.2.1 ##### flush\_thread\_num\_per\_store[​](#flush_thread_num_per_store "Direct link to flush_thread_num_per_store") * Default: 2 * Type: Int * Unit: - * Is mutable: Yes * Description: Number of threads that are used for flushing MemTable in each store. * Introduced in: - ##### lake\_flush\_thread\_num\_per\_store[​](#lake_flush_thread_num_per_store "Direct link to lake_flush_thread_num_per_store") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: Number of threads that are used for flushing MemTable in each store in a shared-data cluster. When this value is set to `0`, the system uses twice of the CPU core count as the value. When this value is set to less than `0`, the system uses the product of its absolute value and the CPU core count as the value. * Introduced in: v3.1.12, 3.2.7 ##### load\_data\_reserve\_hours[​](#load_data_reserve_hours "Direct link to load_data_reserve_hours") * Default: 4 * Type: Int * Unit: Hours * Is mutable: No * Description: The reservation time for the files produced by small-scale loadings. * Introduced in: - ##### load\_error\_log\_reserve\_hours[​](#load_error_log_reserve_hours "Direct link to load_error_log_reserve_hours") * Default: 48 * Type: Int * Unit: Hours * Is mutable: Yes * Description: The time for which data loading logs are reserved. * Introduced in: - ##### load\_process\_max\_memory\_limit\_bytes[​](#load_process_max_memory_limit_bytes "Direct link to load_process_max_memory_limit_bytes") * Default: 107374182400 * Type: Int * Unit: Bytes * Is mutable: No * Description: The maximum size limit of memory resources that can be taken up by all load processes on a BE node. * Introduced in: - ##### load\_spill\_memory\_usage\_per\_merge[​](#load_spill_memory_usage_per_merge "Direct link to load_spill_memory_usage_per_merge") * Default: 1073741824 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: The maximum memory usage per merge operation during spill merge. Default is 1 GB (1073741824 bytes). This parameter controls the memory consumption of individual merge tasks during data loading spill merge to prevent excessive memory usage. * Introduced in: - ##### max\_consumer\_num\_per\_group[​](#max_consumer_num_per_group "Direct link to max_consumer_num_per_group") * Default: 3 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of consumers in a consumer group of Routine Load. * Introduced in: - ##### max\_runnings\_transactions\_per\_txn\_map[​](#max_runnings_transactions_per_txn_map "Direct link to max_runnings_transactions_per_txn_map") * Default: 100 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of transactions that can run concurrently in each partition. * Introduced in: - ##### number\_tablet\_writer\_threads[​](#number_tablet_writer_threads "Direct link to number_tablet_writer_threads") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: The number of tablet writer threads used in ingestion, such as Stream Load, Broker Load and Insert. When the parameter is set to less than or equal to 0, the system uses half of the number of CPU cores, with a minimum of 16. When the parameter is set to greater than 0, the system uses that value. This configuration is changed to dynamic from v3.1.7 onwards. * Introduced in: - ##### push\_worker\_count\_high\_priority[​](#push_worker_count_high_priority "Direct link to push_worker_count_high_priority") * Default: 3 * Type: Int * Unit: - * Is mutable: No * Description: The number of threads used to handle a load task with HIGH priority. * Introduced in: - ##### push\_worker\_count\_normal\_priority[​](#push_worker_count_normal_priority "Direct link to push_worker_count_normal_priority") * Default: 3 * Type: Int * Unit: - * Is mutable: No * Description: The number of threads used to handle a load task with NORMAL priority. * Introduced in: - ##### streaming\_load\_max\_batch\_size\_mb[​](#streaming_load_max_batch_size_mb "Direct link to streaming_load_max_batch_size_mb") * Default: 100 * Type: Int * Unit: MB * Is mutable: Yes * Description: The maximum size of a JSON file that can be streamed into StarRocks. * Introduced in: - ##### streaming\_load\_max\_mb[​](#streaming_load_max_mb "Direct link to streaming_load_max_mb") * Default: 102400 * Type: Int * Unit: MB * Is mutable: Yes * Description: The maximum size of a file that can be streamed into StarRocks. From v3.0, the default value has been changed from `10240` to `102400`. * Introduced in: - ##### streaming\_load\_rpc\_max\_alive\_time\_sec[​](#streaming_load_rpc_max_alive_time_sec "Direct link to streaming_load_rpc_max_alive_time_sec") * Default: 1200 * Type: Int * Unit: Seconds * Is mutable: No * Description: The RPC timeout for Stream Load. * Introduced in: - ##### transaction\_publish\_version\_thread\_pool\_num\_min[​](#transaction_publish_version_thread_pool_num_min "Direct link to transaction_publish_version_thread_pool_num_min") * Default: 0 * Type: Int * Unit: Threads * Is mutable: Yes * Description: Minimum number of threads in the Publish Version thread pool. The pool can shrink to this value when idle. `0` means no fixed lower bound. * Introduced in: - ##### transaction\_publish\_version\_thread\_pool\_idle\_time\_ms[​](#transaction_publish_version_thread_pool_idle_time_ms "Direct link to transaction_publish_version_thread_pool_idle_time_ms") * Default: 60000 * Type: Int * Unit: Milliseconds * Is mutable: No * Description: The idle time before a thread is reclaimed by the Publish Version thread pool. * Introduced in: - ##### transaction\_publish\_version\_worker\_count[​](#transaction_publish_version_worker_count "Direct link to transaction_publish_version_worker_count") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of threads used to publish a version. When this value is set to less than or equal to `0`, the system uses the CPU core count as the value, so as to avoid insufficient thread resources when import concurrency is high but only a fixed number of threads are used. From v2.5, the default value has been changed from `8` to `0`. * Introduced in: - ##### use\_mmap\_allocate\_chunk[​](#use_mmap_allocate_chunk "Direct link to use_mmap_allocate_chunk") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to use anonymous mmap (`MAP_ANONYMOUS | MAP_PRIVATE`) for chunk allocation. When enabled, many VM mappings are created; you must raise `vm.max_map_count` (e.g., `echo 262144 > /proc/sys/vm/max_map_count`) and set a large `chunk_reserved_bytes_limit`, otherwise frequent map/unmap operations will cause severe performance degradation. * Introduced in: v3.2.0 ##### write\_buffer\_size[​](#write_buffer_size "Direct link to write_buffer_size") * Default: 104857600 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: The buffer size of MemTable in the memory. This configuration item is the threshold to trigger a flush. * Introduced in: - ##### broker\_write\_timeout\_seconds[​](#broker_write_timeout_seconds "Direct link to broker_write_timeout_seconds") * Default: 30 * Type: int * Unit: Seconds * Is mutable: No * Description: Timeout (in seconds) used by backend broker operations for write/IO RPCs. The value is multiplied by 1000 to produce millisecond timeouts and is passed as the default timeout\_ms to BrokerFileSystem and BrokerServiceConnection instances (e.g., file export and snapshot upload/download). Increase this when brokers or network are slow or when transferring large files to avoid premature timeouts; decreasing it may cause broker RPCs to fail earlier. This value is defined in common/config and is applied at process start (not dynamically reloadable). * Introduced in: v3.2.0 ##### enable\_load\_channel\_rpc\_async[​](#enable_load_channel_rpc_async "Direct link to enable_load_channel_rpc_async") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: When enabled, handling of load-channel open RPCs (for example, `PTabletWriterOpen`) is offloaded from the BRPC worker to a dedicated thread pool: the request handler creates a `ChannelOpenTask` and submits it to the internal `_async_rpc_pool` instead of running `LoadChannelMgr::_open` inline. This reduces work and blocking inside BRPC threads and allows tuning concurrency via `load_channel_rpc_thread_pool_num` and `load_channel_rpc_thread_pool_queue_size`. If the thread pool submission fails (when pool is full or shut down), the request is canceled and an error status is returned. The pool is shut down on `LoadChannelMgr::close()`, so consider capacity and lifecycle when you want to enable this feature so as to avoid request rejections or delayed processing. * Introduced in: v3.5.0 ##### enable\_load\_diagnose[​](#enable_load_diagnose "Direct link to enable_load_diagnose") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: When enabled, StarRocks will attempt an automated load diagnosis from BE OlapTableSink/NodeChannel after a brpc timeout matching "\[E1008]Reached timeout". The code creates a `PLoadDiagnoseRequest` and sends an RPC to the remote LoadChannel to collect a profile and/or stack trace (controlled by `load_diagnose_rpc_timeout_profile_threshold_ms` and `load_diagnose_rpc_timeout_stack_trace_threshold_ms`). The diagnose RPC uses `load_diagnose_send_rpc_timeout_ms` as its timeout. Diagnosis is skipped if a diagnose request is already in progress. Enabling this produces additional RPCs and profiling work on target nodes; disable on sensitive production workloads to avoid extra overhead. * Introduced in: v3.5.0 ##### enable\_load\_segment\_parallel[​](#enable_load_segment_parallel "Direct link to enable_load_segment_parallel") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: When enabled, rowset segment loading and rowset-level reads are performed concurrently using StarRocks background thread pools (ExecEnv::load\_segment\_thread\_pool and ExecEnv::load\_rowset\_thread\_pool). Rowset::load\_segments and TabletReader::get\_segment\_iterators submit per-segment or per-rowset tasks to these pools, falling back to serial loading and logging a warning if submission fails. Enable this to reduce read/load latency for large rowsets at the cost of increased CPU/IO concurrency and memory pressure. Note: parallel loading can change the load completion order of segments and therefore prevents partial compaction (code checks `_parallel_load` and disables partial compaction when enabled); consider implications for operations that rely on segment order. * Introduced in: v3.3.0, v3.4.0, v3.5.0 ##### enable\_streaming\_load\_thread\_pool[​](#enable_streaming_load_thread_pool "Direct link to enable_streaming_load_thread_pool") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Controls whether streaming load scanners are submitted to the dedicated streaming load thread pool. When enabled and a query is a LOAD with `TLoadJobType::STREAM_LOAD`, ConnectorScanNode submits scanner tasks to the `streaming_load_thread_pool` (which is configured with INT32\_MAX threads and queue sizes, i.e. effectively unbounded). When disabled, scanners use the general `thread_pool` and its `PriorityThreadPool` submission logic (priority computation, try\_offer/offer behavior). Enabling isolates streaming-load work from regular query execution to reduce interference; however, because the dedicated pool is effectively unbounded, enabling may increase concurrent threads and resource usage under heavy streaming-load traffic. This option is on by default and typically does not require modification. * Introduced in: v3.2.0 ##### es\_http\_timeout\_ms[​](#es_http_timeout_ms "Direct link to es_http_timeout_ms") * Default: 5000 * Type: Int * Unit: Milliseconds * Is mutable: No * Description: HTTP connection timeout (in milliseconds) used by the ES network client in ESScanReader for Elasticsearch scroll requests. This value is applied via network\_client.set\_timeout\_ms() before sending subsequent scroll POSTs and controls how long the client waits for an ES response during scrolling. Increase this value for slow networks or large queries to avoid premature timeouts; decrease to fail faster on unresponsive ES nodes. This setting complements `es_scroll_keepalive`, which controls the scroll context keep-alive duration. * Introduced in: v3.2.0 ##### es\_index\_max\_result\_window[​](#es_index_max_result_window "Direct link to es_index_max_result_window") * Default: 10000 * Type: Int * Unit: - * Is mutable: No * Description: Limits the maximum number of documents StarRocks will request from Elasticsearch in a single batch. StarRocks sets the ES request batch size to min(`es_index_max_result_window`, `chunk_size`) when building `KEY_BATCH_SIZE` for the ES reader. If an ES request exceeds the Elasticsearch index setting `index.max_result_window`, Elasticsearch returns HTTP 400 (Bad Request). Adjust this value when scanning large indexes or increase the ES `index.max_result_window` on the Elasticsearch side to permit larger single requests. * Introduced in: v3.2.0 ##### ignore\_load\_tablet\_failure[​](#ignore_load_tablet_failure "Direct link to ignore_load_tablet_failure") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: When this item is set to `false`, the system will treat any tablet header load failures (non-NotFound and non-AlreadyExist errors) as fatal: the code logs the error and calls LOG(FATAL) to stop the BE process. When it is set to `true`, the BE continues startup despite such per-tablet load errors — failed tablet IDs are recorded and skipped while successful tablets are still loaded. Note that this parameter does NOT suppress fatal errors from the RocksDB meta scan itself, which always cause the process to quit. * Introduced in: v3.2.0 ##### load\_channel\_abort\_clean\_up\_delay\_seconds[​](#load_channel_abort_clean_up_delay_seconds "Direct link to load_channel_abort_clean_up_delay_seconds") * Default: 600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Controls how long (in seconds) the system keeps the load IDs of aborted load channels before removing them from `_aborted_load_channels`. When a load job is cancelled or fails, the load ID stays recorded so any late-arriving load RPCs can be rejected immediately; once the delay expires, the entry is cleaned during the periodic background sweep (minimum sweep interval is 60 seconds). Setting the delay too low risks accepting stray RPCs after an abort, while setting it too high may retain state and consume resources longer than necessary. Tune this to balance correctness of late-request rejection and resource retention for aborted loads. * Introduced in: v3.5.11, v4.0.4 ##### load\_channel\_rpc\_thread\_pool\_num[​](#load_channel_rpc_thread_pool_num "Direct link to load_channel_rpc_thread_pool_num") * Default: -1 * Type: Int * Unit: Threads * Is mutable: Yes * Description: Maximum number of threads for the load-channel async RPC thread pool. When set to less than or equal to 0 (default `-1`) the pool size is auto-set to the number of CPU cores (`CpuInfo::num_cores()`). The configured value is used as ThreadPoolBuilder's max threads and the pool's min threads is set to min(5, max\_threads). The pool queue size is controlled separately by `load_channel_rpc_thread_pool_queue_size`. This setting was introduced to align the async RPC pool size with brpc workers' default (`brpc_num_threads`) so behavior remains compatible after switching load RPC handling from synchronous to asynchronous. Changing this config at runtime triggers `ExecEnv::GetInstance()->load_channel_mgr()->async_rpc_pool()->update_max_threads(...)`. * Introduced in: v3.5.0 ##### load\_channel\_rpc\_thread\_pool\_queue\_size[​](#load_channel_rpc_thread_pool_queue_size "Direct link to load_channel_rpc_thread_pool_queue_size") * Default: 1024000 * Type: int * Unit: Count * Is mutable: No * Description: Sets the maximum pending-task queue size for the Load channel RPC thread pool created by LoadChannelMgr. This thread pool executes asynchronous `open` requests when `enable_load_channel_rpc_async` is enabled; the pool size is paired with `load_channel_rpc_thread_pool_num`. The large default (1024000) aligns with brpc workers' defaults to preserve behavior after switching from synchronous to asynchronous handling. If the queue is full, ThreadPool::submit() will fail and the incoming open RPC is cancelled with an error, causing the caller to receive a rejection. Increase this value to buffer larger bursts of concurrent `open` requests; reducing it tightens backpressure but may cause more rejections under load. * Introduced in: v3.5.0 ##### load\_diagnose\_rpc\_timeout\_profile\_threshold\_ms[​](#load_diagnose_rpc_timeout_profile_threshold_ms "Direct link to load_diagnose_rpc_timeout_profile_threshold_ms") * Default: 60000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: When a load RPC times out (error contains "\[E1008]Reached timeout") and `enable_load_diagnose` is true, this threshold controls whether a full profiling diagnose is requested. If the request-level RPC timeout `_rpc_timeout_ms` is greater than `load_diagnose_rpc_timeout_profile_threshold_ms`, profiling is enabled for that diagnose. For smaller `_rpc_timeout_ms` values, profiling is sampled once every 20 timeouts to avoid frequent heavy diagnostics for real-time/short-timeout loads. This value affects the `profile` flag in the `PLoadDiagnoseRequest` sent; stack-trace behavior is controlled separately by `load_diagnose_rpc_timeout_stack_trace_threshold_ms` and send timeout by `load_diagnose_send_rpc_timeout_ms`. * Introduced in: v3.5.0 ##### load\_diagnose\_rpc\_timeout\_stack\_trace\_threshold\_ms[​](#load_diagnose_rpc_timeout_stack_trace_threshold_ms "Direct link to load_diagnose_rpc_timeout_stack_trace_threshold_ms") * Default: 600000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: Threshold (in ms) used to decide when to request remote stack traces for long-running load RPCs. When a load RPC times out with a timeout error and the effective RPC timeout (\_rpc\_timeout\_ms) exceeds this value, `OlapTableSink`/`NodeChannel` will include `stack_trace=true` in a `load_diagnose` RPC to the target BE so the BE can return stack traces for debugging. `LocalTabletsChannel::SecondaryReplicasWaiter` also triggers a best-effort stack-trace diagnose from the primary if waiting for secondary replicas exceeds this interval. This behavior requires `enable_load_diagnose` and uses `load_diagnose_send_rpc_timeout_ms` for the diagnose RPC timeout; profiling is gated separately by `load_diagnose_rpc_timeout_profile_threshold_ms`. Lowering this value increases how aggressively stack traces are requested. * Introduced in: v3.5.0 ##### load\_diagnose\_send\_rpc\_timeout\_ms[​](#load_diagnose_send_rpc_timeout_ms "Direct link to load_diagnose_send_rpc_timeout_ms") * Default: 2000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: Timeout (in milliseconds) applied to diagnosis-related brpc calls initiated by BE load paths. It is used to set the controller timeout for `load_diagnose` RPCs (sent by NodeChannel/OlapTableSink when a LoadChannel brpc call times out) and for replica-status queries (used by SecondaryReplicasWaiter / LocalTabletsChannel when checking primary replica state). Choose a value high enough to allow the remote side to respond with profile or stack-trace data, but not so high that failure handling is delayed. This parameter works together with `enable_load_diagnose`, `load_diagnose_rpc_timeout_profile_threshold_ms`, and `load_diagnose_rpc_timeout_stack_trace_threshold_ms` which control when and what diagnostic information is requested. * Introduced in: v3.5.0 ##### load\_fp\_brpc\_timeout\_ms[​](#load_fp_brpc_timeout_ms "Direct link to load_fp_brpc_timeout_ms") * Default: -1 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: Overrides the per-channel brpc RPC timeout used by OlapTableSink when the `node_channel_set_brpc_timeout` fail point is triggered. If set to a positive value, NodeChannel will set its internal `_rpc_timeout_ms` to this value (in milliseconds) causing open/add-chunk/cancel RPCs to use the shorter timeout and enabling simulation of brpc timeouts that produce the "\[E1008]Reached timeout" error. Default (`-1`) disables the override. Changing this value is intended for testing and fault injection; small values may produce false timeouts and trigger load diagnostics (see `enable_load_diagnose`, `load_diagnose_rpc_timeout_profile_threshold_ms`, `load_diagnose_rpc_timeout_stack_trace_threshold_ms`, and `load_diagnose_send_rpc_timeout_ms`). * Introduced in: v3.5.0 ##### load\_fp\_tablets\_channel\_add\_chunk\_block\_ms[​](#load_fp_tablets_channel_add_chunk_block_ms "Direct link to load_fp_tablets_channel_add_chunk_block_ms") * Default: -1 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: When enabled (set to a positive milliseconds value) this fail-point configuration makes TabletsChannel::add\_chunk sleep for the specified time during load processing. It is used to simulate BRPC timeout errors (e.g., "\[E1008]Reached timeout") and to emulate an expensive add\_chunk operation that increases load latency. A value less than or equal to 0 (default `-1`) disables the injection. Intended for testing fault handling, timeouts, and replica synchronization behavior — do not enable in normal production workloads as it delays write completion and can trigger upstream timeouts or replica aborts. * Introduced in: v3.5.0 ##### load\_segment\_thread\_pool\_num\_max[​](#load_segment_thread_pool_num_max "Direct link to load_segment_thread_pool_num_max") * Default: 128 * Type: Int * Unit: - * Is mutable: No * Description: Sets the maximum number of worker threads for BE load-related thread pools. This value is used by ThreadPoolBuilder to limit threads for both `load_rowset_pool` and `load_segment_pool` in exec\_env.cpp, controlling concurrency for processing loaded rowsets and segments (e.g., decoding, indexing, writing) during streaming and batch loads. Increasing this value raises parallelism and can improve load throughput but also increases CPU, memory usage, and potential contention; decreasing it limits concurrent load processing and may reduce throughput. Tune together with `load_segment_thread_pool_queue_size` and `streaming_load_thread_pool_idle_time_ms`. Change requires BE restart. * Introduced in: v3.3.0, v3.4.0, v3.5.0 ##### load\_segment\_thread\_pool\_queue\_size[​](#load_segment_thread_pool_queue_size "Direct link to load_segment_thread_pool_queue_size") * Default: 10240 * Type: Int * Unit: Tasks * Is mutable: No * Description: Sets the maximum queue length (number of pending tasks) for the load-related thread pools created as "load\_rowset\_pool" and "load\_segment\_pool". These pools use `load_segment_thread_pool_num_max` for their max thread count and this configuration controls how many load segment/rowset tasks can be buffered before the ThreadPool's overflow policy takes effect (further submissions may be rejected or blocked depending on the ThreadPool implementation). Increase to allow more pending load work (uses more memory and can raise latency); decrease to limit buffered load concurrency and reduce memory usage. * Introduced in: v3.3.0, v3.4.0, v3.5.0 ##### max\_pulsar\_consumer\_num\_per\_group[​](#max_pulsar_consumer_num_per_group "Direct link to max_pulsar_consumer_num_per_group") * Default: 10 * Type: Int * Unit: - * Is mutable: Yes * Description: Controls the maximum number of Pulsar consumers that may be created in a single data consumer group for routine load on a BE. Because cumulative acknowledge is not supported for multi-topic subscriptions, each consumer subscribes exactly one topic/partition; if the number of partitions in `pulsar_info->partitions` exceeds this value, group creation fails with an error advising to increase `max_pulsar_consumer_num_per_group` on the BE or add more BEs. This limit is enforced when constructing a PulsarDataConsumerGroup and prevents a BE from hosting more than this many consumers for one routine load group. For Kafka routine load, `max_consumer_num_per_group` is used instead. * Introduced in: v3.2.0 ##### pull\_load\_task\_dir[​](#pull_load_task_dir "Direct link to pull_load_task_dir") * Default: `${STARROCKS_HOME}/var/pull_load` * Type: string * Unit: - * Is mutable: No * Description: Filesystem path where the BE stores data and working files for "pull load" tasks (downloaded source files, task state, temporary output, etc.). The directory must be writable by the BE process and have sufficient disk space for incoming loads. The default is relative to STARROCKS\_HOME; tests create and expect this directory to exist (see test configuration). * Introduced in: v3.2.0 ##### routine\_load\_kafka\_timeout\_second[​](#routine_load_kafka_timeout_second "Direct link to routine_load_kafka_timeout_second") * Default: 10 * Type: Int * Unit: Seconds * Is mutable: No * Description: Timeout in seconds used for Kafka-related routine load operations. When a client request does not specify a timeout, `routine_load_kafka_timeout_second` is used as the default RPC timeout (converted to milliseconds) for `get_info`. It is also used as the per-call consume poll timeout for the librdkafka consumer (converted to milliseconds and capped by remaining runtime). Note: the internal `get_info` path reduces this value to 80% before passing it to librdkafka to avoid FE-side timeout races. Set this to a value that balances timely failure reporting and sufficient time for network/broker responses; changes require a restart because the setting is not mutable. * Introduced in: v3.2.0 ##### routine\_load\_pulsar\_timeout\_second[​](#routine_load_pulsar_timeout_second "Direct link to routine_load_pulsar_timeout_second") * Default: 10 * Type: Int * Unit: Seconds * Is mutable: No * Description: Default timeout (in seconds) that the BE uses for Pulsar-related routine load operations when the request does not supply an explicit timeout. Specifically, `PInternalServiceImplBase::get_pulsar_info` multiplies this value by 1000 to form the millisecond timeout passed to the routine load task executor methods that fetch Pulsar partition metadata and backlog. Increase to allow slower Pulsar responses at the cost of longer failure detection; decrease to fail faster on slow brokers. Analogous to `routine_load_kafka_timeout_second` used for Kafka. * Introduced in: v3.2.0 ##### streaming\_load\_thread\_pool\_idle\_time\_ms[​](#streaming_load_thread_pool_idle_time_ms "Direct link to streaming_load_thread_pool_idle_time_ms") * Default: 2000 * Type: Int * Unit: Milliseconds * Is mutable: No * Description: Sets the thread idle timeout (in milliseconds) for streaming-load related thread pools. The value is used as the idle timeout passed to ThreadPoolBuilder for the `stream_load_io` pool and also for `load_rowset_pool` and `load_segment_pool`. Threads in these pools are reclaimed when idle for this duration; lower values reduce idle resource usage but increase thread creation overhead, while higher values keep threads alive longer. The `stream_load_io` pool is used when `enable_streaming_load_thread_pool` is enabled. * Introduced in: v3.2.0 ##### streaming\_load\_thread\_pool\_num\_min[​](#streaming_load_thread_pool_num_min "Direct link to streaming_load_thread_pool_num_min") * Default: 0 * Type: Int * Unit: - * Is mutable: No * Description: Minimum number of threads for the streaming load IO thread pool ("stream\_load\_io") created during ExecEnv initialization. The pool is built with `set_max_threads(INT32_MAX)` and `set_max_queue_size(INT32_MAX)` so it is effectively unbounded to avoid deadlocks for concurrent streaming loads. A value of 0 lets the pool start with no threads and grow on demand; setting a positive value reserves that many threads at startup. This pool is used when `enable_streaming_load_thread_pool` is true and its idle timeout is controlled by `streaming_load_thread_pool_idle_time_ms`. Overall concurrency is still constrained by `fragment_pool_thread_num_max` and `webserver_num_workers`; changing this value is rarely necessary and may increase resource usage if set too high. * Introduced in: v3.2.0 --- ### BE Configuration - Shared-data, Data Lake, and Others Some BE configuration items are dynamic parameters which you can set interactively when BE nodes are online. The rest of them are static parameters. You can only set the static parameters of a BE node by changing them in the corresponding configuration file **be.conf** and restarting the BE node to allow the change to take effect. #### View BE configuration items[​](#view-be-configuration-items "Direct link to View BE configuration items") You can view the BE configuration items using the following command: ```sql SELECT * FROM information_schema.be_configs [WHERE NAME LIKE "%%"] ``` #### Configure BE parameters[​](#configure-be-parameters "Direct link to Configure BE parameters") ##### Configure BE dynamic parameters[​](#configure-be-dynamic-parameters "Direct link to Configure BE dynamic parameters") You can configure a dynamic parameter of a BE node by updating the value in `information_schema.be_configs`. warning Setting an invalid value may cause unknown behaviors. Check twice before you run the command to update the configuration. ```sql -- Replace with the key of the configuration and with the value. UPDATE information_schema.be_configs SET VALUE = WHERE name = ""; ``` ##### Configure BE static parameters[​](#configure-be-static-parameters "Direct link to Configure BE static parameters") You can only set the static parameters of a BE by changing them in the corresponding configuration file **be.conf**, and restarting the BE to allow the changes to take effect. *** This topic introduces the following types of BE configurations: * [Shared-data](#shared-data) * [Data Lake](#data-lake) * [Other](#other) #### Shared-data[​](#shared-data "Direct link to Shared-data") ##### cloud\_native\_pk\_index\_rebuild\_files\_threshold[​](#cloud_native_pk_index_rebuild_files_threshold "Direct link to cloud_native_pk_index_rebuild_files_threshold") * Default: 50 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of segment files that need to be rebuilt in cloud-native Primary Key index. If the number of files that need to be rebuilt during index recovery exceeds this threshold, StarRocks will flush the in-memory MemTable immediately to reduce the number of segments that must be replayed. Set to `0` to disable this early-flush strategy. * Introduced in: - ##### cloud\_native\_pk\_index\_rebuild\_rows\_threshold[​](#cloud_native_pk_index_rebuild_rows_threshold "Direct link to cloud_native_pk_index_rebuild_rows_threshold") * Default: 10000000 * Type: Long * Unit: Rows * Is mutable: Yes * Description: The maximum number of rows that need to be rebuilt in cloud-native Primary Key index. If the number of rows that need to be rebuilt during index recovery exceeds this threshold, StarRocks will flush the in-memory MemTable immediately to reduce the rebuild overhead. Set to `0` to disable this early-flush strategy. Works in conjunction with `cloud_native_pk_index_rebuild_files_threshold`; a flush is triggered if either threshold is exceeded. The row count includes segment rows plus the tombstone (delete) rows recorded in del files, so a delete-heavy workload that produces a few large del files also counts toward this threshold; del files written by older versions that did not record a row count contribute 0. * Introduced in: - ##### download\_buffer\_size[​](#download_buffer_size "Direct link to download_buffer_size") * Default: 4194304 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: Size (in bytes) of the in-memory copy buffer used when downloading snapshot files. SnapshotLoader::download passes this value to fs::copy as the per-transfer chunk size when reading from the remote sequential file into the local writable file. Larger values can improve throughput on high-bandwidth links by reducing syscall/IO overhead; smaller values reduce peak memory use per active transfer. Note: this parameter controls buffer size per stream, not the number of download threads—total memory consumption = download\_buffer\_size \* number\_of\_concurrent\_downloads. * Introduced in: v3.2.13 ##### graceful\_exit\_wait\_for\_frontend\_heartbeat[​](#graceful_exit_wait_for_frontend_heartbeat "Direct link to graceful_exit_wait_for_frontend_heartbeat") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Determines whether to await at least one frontend heartbeat response indicating SHUTDOWN status before completing graceful exit. When enabled, the graceful shutdown process remains active until a SHUTDOWN confirmation is responded via heartbeat RPC, ensuring the frontend has sufficient time to detect the termination state between two regular heartbeat intervals. * Introduced in: v3.4.5 ##### lake\_compaction\_stream\_buffer\_size\_bytes[​](#lake_compaction_stream_buffer_size_bytes "Direct link to lake_compaction_stream_buffer_size_bytes") * Default: 1048576 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: The reader's remote I/O buffer size for cloud-native table compaction in a shared-data cluster. The default value is 1MB. You can increase this value to accelerate compaction process. * Introduced in: v3.2.3 ##### lake\_enable\_pk\_preserve\_txn\_delete\_order[​](#lake_enable_pk_preserve_txn_delete_order "Direct link to lake_enable_pk_preserve_txn_delete_order") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to preserve the in-transaction upsert/delete order for Primary Key tables in a shared-data cluster. When a single load transaction contains both a `DELETE` and a later re-`UPSERT` of the same key, enabling this makes the re-upsert win (consistent with shared-nothing clusters). It is disabled by default for downgrade safety: when enabled, a load can persist on-disk metadata that a BE rolled back to a version without this fix would misinterpret, potentially producing duplicate primary keys. Only enable it after the entire cluster has been upgraded to a version that supports this feature and you no longer intend to roll back. When disabled, deletes fall back to the legacy behavior (applied after all upserts in the transaction). * Introduced in: - ##### lake\_enable\_protobuf\_file\_checksum[​](#lake_enable_protobuf_file_checksum "Direct link to lake_enable_protobuf_file_checksum") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to write tablet metadata and transaction log files of a shared-data cluster with an Adler-32 checksum, so that corruption of these files can be detected when they are read. Regardless of this item, readers always detect and verify the checksum automatically when it is present; this item only controls the write format. Set it to `false` only while the cluster may still be downgraded to a version that predates the checksummed format. During a rolling upgrade or a downgrade, an earlier BE or CN uses the legacy reader and cannot parse files written in the new format. * Introduced in: v4.2 ##### lake\_pk\_compaction\_max\_input\_rowsets[​](#lake_pk_compaction_max_input_rowsets "Direct link to lake_pk_compaction_max_input_rowsets") * Default: 500 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of input rowsets allowed in a Primary Key table compaction task in a shared-data cluster. The default value of this parameter is changed from `5` to `1000` since v3.2.4 and v3.1.10, and to `500` since v3.3.1 and v3.2.9. After the Sized-tiered Compaction policy is enabled for Primary Key tables (by setting `enable_pk_size_tiered_compaction_strategy` to `true`), StarRocks does not need to limit the number of rowsets for each compaction to reduce write amplification. Therefore, the default value of this parameter is increased. * Introduced in: v3.1.8, v3.2.3 ##### lake\_pk\_compaction\_base\_delete\_ratio\_threshold[​](#lake_pk_compaction_base_delete_ratio_threshold "Direct link to lake_pk_compaction_base_delete_ratio_threshold") * Default: 0.5 * Type: Double * Unit: - * Is mutable: Yes * Description: One of two triggers that switch a Primary Key tablet in a shared-data cluster from cumulative compaction (size-tiered small-file merges) to base compaction, which rewrites the delete-bearing rowsets (the ones with the most deleted rows first) to drop deleted rows and shrink their delete vectors. Base compaction runs when the tablet's aggregate delete ratio (`sum(num_dels) / sum(num_rows)` across rowsets) reaches this value, when its absolute delete-row count reaches `lake_pk_compaction_base_delete_rows_threshold`, or when a manual `ALTER TABLE ... COMPACT` forces a base compaction. Set both thresholds high enough to disable the automatic triggers. * Introduced in: v4.1.4 ##### lake\_pk\_compaction\_base\_delete\_rows\_threshold[​](#lake_pk_compaction_base_delete_rows_threshold "Direct link to lake_pk_compaction_base_delete_rows_threshold") * Default: 10000000 * Type: Int * Unit: - * Is mutable: Yes * Description: One of two triggers for Primary Key base compaction in a shared-data cluster (see `lake_pk_compaction_base_delete_ratio_threshold`). Base compaction runs when a tablet's absolute delete-row count (`sum(num_dels)` across rowsets) reaches this value. This absolute-count trigger complements the ratio trigger: on hot update/delete tables the delete vectors bloat and space is wasted while the aggregate delete ratio stays low (diluted by many mostly-live rowsets), so the ratio trigger alone would not fire. Raise it to make base compaction less frequent, or lower it to reclaim delete vectors sooner. * Introduced in: v4.1.4 ##### lake\_put\_txn\_log\_timeout\_guard\_ms[​](#lake_put_txn_log_timeout_guard_ms "Direct link to lake_put_txn_log_timeout_guard_ms") * Default: -1 * Type: Int64 * Unit: Milliseconds * Is mutable: Yes * Description: Timeout guard for writing a transaction log to object storage in a shared-data cluster (the `put_txn_log` and `put_combined_txn_log` paths). If writing a transaction log takes longer than this value, StarRocks dumps the stack trace of the slow thread to the BE log to help diagnose slow object-storage writes. Disabled by default (a value less than or equal to `0` disables the guard); set it to a positive value such as `4000` (4 seconds) to enable. * Introduced in: - ##### lake\_rows\_mapper\_read\_parallelism[​](#lake_rows_mapper_read_parallelism "Direct link to lake_rows_mapper_read_parallelism") * Default: 32 * Type: Int * Unit: sub-chunks * Is mutable: Yes * Description: Maximum number of in-flight `.lcrm` (lake compaction rows-mapper) sub-chunk reads kept by `RowsMapperIterator` during light Primary Key compaction publish in a shared-data cluster. Each sub-chunk is `lake_rows_mapper_sub_chunk_bytes` in size and never crosses a segment boundary; the iterator submits up to this many reads to the PK index execution thread pool and pipelines them against the caller's per-segment processing. Memory bound is `lake_rows_mapper_read_parallelism * lake_rows_mapper_sub_chunk_bytes`. Set to `1` to disable pipelining and fall back to sequential reads. ##### lake\_rows\_mapper\_sub\_chunk\_bytes[​](#lake_rows_mapper_sub_chunk_bytes "Direct link to lake_rows_mapper_sub_chunk_bytes") * Default: 4194304 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: Sub-chunk granularity for `RowsMapperIterator` pipelined reads of `.lcrm` files during light Primary Key compaction publish in a shared-data cluster. Each output segment is split into `ceil(segment_bytes / lake_rows_mapper_sub_chunk_bytes)` sub-chunks pipelined independently. Smaller values raise the achievable parallelism for few-but-large output segments at the cost of more range reads and an extra memcpy on consume. Defaults to 4 MiB to align with the starcache disk-tier block size. ##### lake\_vacuum\_min\_batch\_delete\_size[​](#lake_vacuum_min_batch_delete_size "Direct link to lake_vacuum_min_batch_delete_size") * Default: 200 * Type: Int64 * Unit: Number of files * Is mutable: Yes * Description: The number of stale files Vacuum batches into a single `DeleteObjects` request on a shared-data cluster. A larger batch amortizes per-call HTTP / auth / signing overhead and reduces the prefix-level request rate against the object store, at the cost of higher single-call latency and a larger replay cost when a transient error retries. Users running on AWS S3 are encouraged to raise this further (up to the protocol cap of 1000) where per-request server time is nearly batch-size insensitive. ##### loop\_count\_wait\_fragments\_finish[​](#loop_count_wait_fragments_finish "Direct link to loop_count_wait_fragments_finish") * Default: 2 * Type: Int * Unit: - * Is mutable: Yes * Description: The number of loops to be waited when the BE/CN process exits. Each loop is a fixed interval of 10 seconds. You can set it to `0` to disable the loop wait. From v3.4 onwards, this item is changed to mutable and its default value is changed from `0` to `2`. * Introduced in: v2.5 ##### max\_client\_cache\_size\_per\_host[​](#max_client_cache_size_per_host "Direct link to max_client_cache_size_per_host") * Default: 10 * Type: Int * Unit: entries (cached client instances) per host * Is mutable: No * Description: The maximum number of cached client instances retained for each remote host by BE-wide client caches. This single setting is used when creating BackendServiceClientCache, FrontendServiceClientCache, and BrokerServiceClientCache during ExecEnv initialization, so it limits the number of client stubs/connections kept per host across those caches. Raising this value reduces reconnects and stub creation overhead at the cost of increased memory and file-descriptor usage; lowering it saves resources but may increase connection churn. The value is read at startup and cannot be changed at runtime. Currently one shared setting controls all client cache types; separate per-cache configuration may be introduced later. * Introduced in: v3.2.0 ##### starlet\_filesystem\_instance\_cache\_capacity[​](#starlet_filesystem_instance_cache_capacity "Direct link to starlet_filesystem_instance_cache_capacity") * Default: 10000 * Type: Int * Unit: - * Is mutable: Yes * Description: The cache capacity of starlet filesystem instances. * Introduced in: v3.2.16, v3.3.11, v3.4.1 ##### starlet\_filesystem\_instance\_cache\_ttl\_sec[​](#starlet_filesystem_instance_cache_ttl_sec "Direct link to starlet_filesystem_instance_cache_ttl_sec") * Default: 86400 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The cache expiration time of starlet filesystem instances. * Introduced in: v3.3.15, 3.4.5 ##### starlet\_port[​](#starlet_port "Direct link to starlet_port") * Default: 9070 * Type: Int * Unit: - * Is mutable: No * Description: An extra agent service port for BE and CN. * Introduced in: - ##### starlet\_star\_cache\_disk\_size\_percent[​](#starlet_star_cache_disk_size_percent "Direct link to starlet_star_cache_disk_size_percent") * Default: 80 * Type: Int * Unit: - * Is mutable: No * Description: The percentage of disk capacity that Data Cache can use at most in a shared-data cluster. Only takes effect when `datacache_unified_instance_enable` is `false`. * Introduced in: v3.1 ##### starlet\_use\_star\_cache[​](#starlet_use_star_cache "Direct link to starlet_use_star_cache") * Default: false in v3.1 and true from v3.2.3 * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable Data Cache in a shared-data cluster. `true` indicates enabling this feature and `false` indicates disabling it. The default value is set from `false` to `true` from v3.2.3 onwards. * Introduced in: v3.1 ##### starlet\_write\_file\_with\_tag[​](#starlet_write_file_with_tag "Direct link to starlet_write_file_with_tag") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: In a shared-data cluster, whether to tag files written to object storage with object storage tags for convenient custom file management. * Introduced in: v3.5.3 ##### table\_schema\_service\_max\_retries[​](#table_schema_service_max_retries "Direct link to table_schema_service_max_retries") * Default: 3 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of retries for Table Schema Service requests. * Introduced in: v4.1 #### Data Lake[​](#data-lake "Direct link to Data Lake") ##### datacache\_block\_buffer\_enable[​](#datacache_block_buffer_enable "Direct link to datacache_block_buffer_enable") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to enable Block Buffer to optimize Data Cache efficiency. When Block Buffer is enabled, the system reads the Block data from the Data Cache and caches it in a temporary buffer, thus reducing the extra overhead caused by frequent cache reads. * Introduced in: v3.2.0 ##### datacache\_disk\_adjust\_interval\_seconds[​](#datacache_disk_adjust_interval_seconds "Direct link to datacache_disk_adjust_interval_seconds") * Default: 10 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The interval of Data Cache automatic capacity scaling. At regular intervals, the system checks the cache disk usage, and triggers Automatic Scaling when necessary. * Introduced in: v3.3.0 ##### datacache\_disk\_idle\_seconds\_for\_expansion[​](#datacache_disk_idle_seconds_for_expansion "Direct link to datacache_disk_idle_seconds_for_expansion") * Default: 7200 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The minimum wait time for Data Cache automatic expansion. Automatic scaling up is triggered only if the disk usage remains below `datacache_disk_low_level` for longer than this duration. * Introduced in: v3.3.0 ##### datacache\_disk\_size[​](#datacache_disk_size "Direct link to datacache_disk_size") * Default: 0 * Type: String * Unit: - * Is mutable: Yes * Description: The maximum amount of data that can be cached on a single disk. You can set it as a percentage (for example, `80%`) or a physical limit (for example, `2T`, `500G`). For example, if you use two disks and set the value of the `datacache_disk_size` parameter as `21474836480` (20 GB), a maximum of 40 GB data can be cached on these two disks. The default value is `0`, which indicates that only memory is used to cache data. * Introduced in: - ##### datacache\_enable[​](#datacache_enable "Direct link to datacache_enable") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to enable Data Cache. `true` indicates Data Cache is enabled, and `false` indicates Data Cache is disabled. The default value is changed to `true` from v3.3. * Introduced in: - ##### datacache\_eviction\_policy[​](#datacache_eviction_policy "Direct link to datacache_eviction_policy") * Default: slru * Type: String * Unit: - * Is mutable: No * Description: The eviction policy of Data Cache. Valid values: `lru` (least recently used) and `slru` (Segmented LRU). * Introduced in: v3.4.0 ##### datacache\_inline\_item\_count\_limit[​](#datacache_inline_item_count_limit "Direct link to datacache_inline_item_count_limit") * Default: 130172 * Type: Int * Unit: - * Is mutable: No * Description: The maximum number of inline cache items in Data Cache. For some particularly small cache blocks, Data Cache stores them in `inline` mode, which caches the block data and metadata together in memory. * Introduced in: v3.4.0 ##### datacache\_mem\_size[​](#datacache_mem_size "Direct link to datacache_mem_size") * Default: 0 * Type: String * Unit: - * Is mutable: Yes * Description: The maximum amount of data that can be cached in memory. You can set it as a percentage (for example, `10%`) or a physical limit (for example, `10G`, `21474836480`). * Introduced in: - ##### datacache\_min\_disk\_quota\_for\_adjustment[​](#datacache_min_disk_quota_for_adjustment "Direct link to datacache_min_disk_quota_for_adjustment") * Default: 10737418240 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: The minimum effective capacity for Data Cache Automatic Scaling. If the system tries to adjust the cache capacity to less than this value, the cache capacity will be directly set to `0` to prevent suboptimal performance caused by frequent cache fills and evictions due to insufficient cache capacity. * Introduced in: v3.3.0 ##### disk\_high\_level[​](#disk_high_level "Direct link to disk_high_level") * Default: 90 * Type: Int * Unit: - * Is mutable: Yes * Description: The upper limit of disk usage (in percentage) that triggers the automatic scaling up of the cache capacity. When the disk usage exceeds this value, the system automatically evicts cache data from the Data Cache. From v3.4.0 onwards, the default value is changed from `80` to `90`. This item is renamed from `datacache_disk_high_level` to `disk_high_level` from v4.0 onwards. * Introduced in: v3.3.0 ##### disk\_low\_level[​](#disk_low_level "Direct link to disk_low_level") * Default: 60 * Type: Int * Unit: - * Is mutable: Yes * Description: The lower limit of disk usage (in percentage) that triggers the automatic scaling down of the cache capacity. When the disk usage remains below this value for the period specified in `datacache_disk_idle_seconds_for_expansion`, and the space allocated for Data Cache is fully utilized, the system will automatically expand the cache capacity by increasing the upper limit. This item is renamed from `datacache_disk_low_level` to `disk_low_level` from v4.0 onwards. * Introduced in: v3.3.0 ##### disk\_safe\_level[​](#disk_safe_level "Direct link to disk_safe_level") * Default: 80 * Type: Int * Unit: - * Is mutable: Yes * Description: The safe level of disk usage (in percentage) for Data Cache. When Data Cache performs automatic scaling, the system adjusts the cache capacity with the goal of maintaining disk usage as close to this value as possible. From v3.4.0 onwards, the default value is changed from `70` to `80`. This item is renamed from `datacache_disk_safe_level` to `disk_safe_level` from v4.0 onwards. * Introduced in: v3.3.0 ##### enable\_connector\_sink\_spill[​](#enable_connector_sink_spill "Direct link to enable_connector_sink_spill") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable Spilling for writes to external tables. Enabling this feature prevents the generation of a large number of small files as a result of writing to an external table when memory is insufficient. Currently, this feature only supports writing to Iceberg tables. * Introduced in: v4.0.0 ##### enable\_datacache\_disk\_auto\_adjust[​](#enable_datacache_disk_auto_adjust "Direct link to enable_datacache_disk_auto_adjust") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable Automatic Scaling for Data Cache disk capacity. When it is enabled, the system dynamically adjusts the cache capacity based on the current disk usage rate. This item is renamed from `datacache_auto_adjust_enable` to `enable_datacache_disk_auto_adjust` from v4.0 onwards. * Introduced in: v3.3.0 ##### datacache\_unified\_instance\_enable[​](#datacache_unified_instance_enable "Direct link to datacache_unified_instance_enable") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to use a unified Data Cache instance to manage data caching for both internal catalog and external catalog in a shared-data cluster. * Introduced in: v3.4.0 ##### jdbc\_connection\_idle\_timeout\_ms[​](#jdbc_connection_idle_timeout_ms "Direct link to jdbc_connection_idle_timeout_ms") * Default: 600000 * Type: Int * Unit: Milliseconds * Is mutable: No * Description: The length of time after which an idle connection in the JDBC connection pool expires. If the connection idle time in the JDBC connection pool exceeds this value, the connection pool closes idle connections beyond the number specified in the configuration item `jdbc_minimum_idle_connections`. * Introduced in: - ##### jdbc\_connection\_pool\_size[​](#jdbc_connection_pool_size "Direct link to jdbc_connection_pool_size") * Default: 8 * Type: Int * Unit: - * Is mutable: No * Description: The JDBC connection pool size. On each BE node, queries that access the external table with the same `jdbc_url` share the same connection pool. * Introduced in: - ##### jdbc\_minimum\_idle\_connections[​](#jdbc_minimum_idle_connections "Direct link to jdbc_minimum_idle_connections") * Default: 1 * Type: Int * Unit: - * Is mutable: No * Description: The minimum number of idle connections in the JDBC connection pool. * Introduced in: - ##### jdbc\_connection\_max\_lifetime\_ms[​](#jdbc_connection_max_lifetime_ms "Direct link to jdbc_connection_max_lifetime_ms") * Default: 300000 * Type: Long * Unit: Milliseconds * Is mutable: No * Description: Maximum lifetime of a connection in the JDBC connection pool. Connections are recycled before this timeout to prevent stale connections. Minimum allowed value is 30000 (30 seconds). * Introduced in: - ##### jdbc\_connection\_keepalive\_time\_ms[​](#jdbc_connection_keepalive_time_ms "Direct link to jdbc_connection_keepalive_time_ms") * Default: 30000 * Type: Long * Unit: Milliseconds * Is mutable: No * Description: Keepalive interval for idle JDBC connections. Idle connections are tested at this interval to detect stale connections proactively. Set to 0 to disable keepalive probing. When enabled, must be >= 30000 and less than `jdbc_connection_max_lifetime_ms`. Invalid enabled values are silently disabled (reset to 0). * Introduced in: - ##### lake\_clear\_corrupted\_cache\_data[​](#lake_clear_corrupted_cache_data "Direct link to lake_clear_corrupted_cache_data") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to allow the system to clear the corrupted data cache in a shared-data cluster. * Introduced in: v3.4 ##### lake\_clear\_corrupted\_cache\_meta[​](#lake_clear_corrupted_cache_meta "Direct link to lake_clear_corrupted_cache_meta") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to allow the system to clear the corrupted metadata cache in a shared-data cluster. * Introduced in: v3.3 ##### lake\_enable\_vertical\_compaction\_fill\_data\_cache[​](#lake_enable_vertical_compaction_fill_data_cache "Direct link to lake_enable_vertical_compaction_fill_data_cache") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to allow vertical compaction tasks to cache data on local disks in a shared-data cluster. * Introduced in: v3.1.7, v3.2.3 ##### lake\_replication\_read\_buffer\_size[​](#lake_replication_read_buffer_size "Direct link to lake_replication_read_buffer_size") * Default: 16777216 * Type: Long * Unit: Bytes * Is mutable: Yes * Description: The read buffer size used when downloading lake segment files during lake replication. This value determines the per-read allocation for reading remote files; the implementation uses the larger of this setting and a 1 MB minimum. A larger value reduces the number of read calls and can improve throughput but increases memory used per concurrent download; a smaller value lowers memory usage at the cost of more I/O calls. Tune according to network bandwidth, storage I/O characteristics, and the number of parallel replication threads. * Introduced in: v4.1.2 ##### lake\_replication\_max\_file\_copy\_retry[​](#lake_replication_max_file_copy_retry "Direct link to lake_replication_max_file_copy_retry") * Default: 3 * Type: Int * Unit: - * Is mutable: Yes * Description: Maximum number of retry attempts for non-segment file copy (`.sst`, `.delvec`, `.del`, `.cols`) during lake-to-lake (shared-data) cross-cluster replication. Each attempt verifies the copied file size matches the source to detect truncated copies caused by transient object storage issues. Increase this value if experiencing intermittent file corruption during replication over unreliable storage. * Introduced in: v4.1.2 ##### lake\_replication\_file\_copy\_threads[​](#lake_replication_file_copy_threads "Direct link to lake_replication_file_copy_threads") * Default: 0 * Type: Int * Unit: - * Is mutable: No * Description: Number of threads in the dedicated thread pool used by lake-to-lake (shared-data) cross-cluster replication for per-file copy. `0` means `cpu_cores * 4` (the same default semantics as `replication_threads`); negative values mean `-value * cpu_cores`. This pool is intentionally separate from the agent-task `replicate_snapshot` pool so that per-file copy sub-tasks can be awaited from the outer task without tripping the thread-pool self-deadlock guard. The pool is built once at startup and has no runtime resize hook, so CN restart is required to change its size. * Introduced in: v4.1.2 ##### lake\_service\_max\_concurrency[​](#lake_service_max_concurrency "Direct link to lake_service_max_concurrency") * Default: 0 * Type: Int * Unit: - * Is mutable: No * Description: The maximum concurrency of RPC requests in a shared-data cluster. Incoming requests will be rejected when this threshold is reached. When this item is set to `0`, no limit is imposed on the concurrency. * Introduced in: - ##### max\_hdfs\_scanner\_num[​](#max_hdfs_scanner_num "Direct link to max_hdfs_scanner_num") * Default: 50 * Type: Int * Unit: - * Is mutable: No * Description: Limits the maximum number of concurrently running connector (HDFS/remote) scanners that a ConnectorScanNode can have. During scan startup the node computes an estimated concurrency (based on memory, chunk size and scanner\_row\_num) and then caps it with this value to determine how many scanners and chunks to reserve and how many scanner threads to start. It is also consulted when scheduling pending scanners at runtime (to avoid oversubscription) and when deciding how many pending scanners can be re-submitted considering file-handle limits. Lowering this reduces threads, memory and open-file pressure at the cost of potential throughput; increasing it raises concurrency and resource usage. * Introduced in: v3.2.0 ##### query\_max\_memory\_limit\_percent[​](#query_max_memory_limit_percent "Direct link to query_max_memory_limit_percent") * Default: 90 * Type: Int * Unit: - * Is mutable: No * Description: The maximum memory that the Query Pool can use. It is expressed as a percentage of the Process memory limit. * Introduced in: v3.1.0 ##### rocksdb\_max\_write\_buffer\_memory\_bytes[​](#rocksdb_max_write_buffer_memory_bytes "Direct link to rocksdb_max_write_buffer_memory_bytes") * Default: 1073741824 * Type: Int64 * Unit: - * Is mutable: No * Description: It is the max size of the write buffer for meta in rocksdb. Default is 1GB. * Introduced in: v3.5.0 ##### rocksdb\_write\_buffer\_memory\_percent[​](#rocksdb_write_buffer_memory_percent "Direct link to rocksdb_write_buffer_memory_percent") * Default: 5 * Type: Int64 * Unit: - * Is mutable: No * Description: It is the memory percent of write buffer for meta in rocksdb. default is 5% of system memory. However, aside from this, the final calculated size of the write buffer memory will not be less than 64MB nor exceed 1G (rocksdb\_max\_write\_buffer\_memory\_bytes) * Introduced in: v3.5.0 #### Other[​](#other "Direct link to Other") ##### default\_mv\_resource\_group\_concurrency\_limit[​](#default_mv_resource_group_concurrency_limit "Direct link to default_mv_resource_group_concurrency_limit") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum concurrency (per BE node) of the materialized view refresh tasks in the resource group `default_mv_wg`. The default value `0` indicates no limits. * Introduced in: v3.1 ##### default\_mv\_resource\_group\_cpu\_limit[​](#default_mv_resource_group_cpu_limit "Direct link to default_mv_resource_group_cpu_limit") * Default: 1 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of CPU cores (per BE node) that can be used by the materialized view refresh tasks in the resource group `default_mv_wg`. * Introduced in: v3.1 ##### default\_mv\_resource\_group\_memory\_limit[​](#default_mv_resource_group_memory_limit "Direct link to default_mv_resource_group_memory_limit") * Default: 0.8 * Type: Double * Unit: * Is mutable: Yes * Description: The maximum memory proportion (per BE node) that can be used by the materialized view refresh tasks in the resource group `default_mv_wg`. The default value indicates 80% of the memory. * Introduced in: v3.1 ##### default\_mv\_resource\_group\_spill\_mem\_limit\_threshold[​](#default_mv_resource_group_spill_mem_limit_threshold "Direct link to default_mv_resource_group_spill_mem_limit_threshold") * Default: 0.8 * Type: Double * Unit: - * Is mutable: Yes * Description: The memory usage threshold before a materialized view refresh task in the resource group `default_mv_wg` triggers intermediate result spilling. The default value indicates 80% of the memory. * Introduced in: v3.1 ##### enable\_resolve\_hostname\_to\_ip\_in\_load\_error\_url[​](#enable_resolve_hostname_to_ip_in_load_error_url "Direct link to enable_resolve_hostname_to_ip_in_load_error_url") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: For `error_urls` debugging, whether to allow operators to choose between using original hostnames from FE heartbeat or forcing resolution to IP addresses based on their environment needs. * `true`: Resolve hostnames to IPs. * `false` (Default): Keeps the original hostname in the error URL. * Introduced in: v4.0.1 ##### enable\_retry\_apply[​](#enable_retry_apply "Direct link to enable_retry_apply") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: When enabled, Tablet apply failures that are classified as retryable (for example transient memory-limit errors) are rescheduled for retry instead of immediately marking the tablet in error. The retry path in TabletUpdates schedules the next attempt using `retry_apply_interval_second` multiplied by the current failure count and clamped to a 600s maximum, so backoff grows with successive failures. Explicitly non-retryable errors (for example corruption) bypass retries and cause the apply process to enter the error state immediately. Retries continue until an overall timeout/terminal condition is reached, after which the apply will enter the error state. Turning this off disables automatic rescheduling of failed apply tasks and causes failed applies to transition to error state without retries. * Introduced in: v3.2.9 ##### enable\_token\_check[​](#enable_token_check "Direct link to enable_token_check") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: A boolean value to control whether to enable the token check. `true` indicates enabling the token check, and `false` indicates disabling it. * Introduced in: - ##### es\_scroll\_keepalive[​](#es_scroll_keepalive "Direct link to es_scroll_keepalive") * Default: 5m * Type: String * Unit: Minutes (string with suffix, e.g. "5m") * Is mutable: No * Description: The keep-alive duration sent to Elasticsearch for scroll search contexts. The value is used verbatim (for example "5m") when building the initial scroll URL (`?scroll=`) and when sending subsequent scroll requests (via ESScrollQueryBuilder). This controls how long the ES search context is retained before garbage collection on the ES side; setting it longer keeps scroll contexts alive for more time but prolongs resource usage on the ES cluster. The value is read at startup by the ES scan reader and is not changeable at runtime. * Introduced in: v3.2.0 ##### load\_replica\_status\_check\_interval\_ms\_on\_failure[​](#load_replica_status_check_interval_ms_on_failure "Direct link to load_replica_status_check_interval_ms_on_failure") * Default: 2000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: The interval that the secondary replica checks it's status on the primary replica if the last check rpc fails. * Introduced in: v3.5.1 ##### load\_replica\_status\_check\_interval\_ms\_on\_success[​](#load_replica_status_check_interval_ms_on_success "Direct link to load_replica_status_check_interval_ms_on_success") * Default: 15000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: The interval that the secondary replica checks it's status on the primary replica if the last check rpc successes. * Introduced in: v3.5.1 ##### max\_length\_for\_bitmap\_function[​](#max_length_for_bitmap_function "Direct link to max_length_for_bitmap_function") * Default: 1000000 * Type: Int * Unit: Bytes * Is mutable: No * Description: The maximum length of input values for bitmap functions. * Introduced in: - ##### max\_length\_for\_to\_base64[​](#max_length_for_to_base64 "Direct link to max_length_for_to_base64") * Default: 200000 * Type: Int * Unit: Bytes * Is mutable: No * Description: The maximum length of input values for the to\_base64() function. * Introduced in: - ##### memory\_high\_level[​](#memory_high_level "Direct link to memory_high_level") * Default: 75 * Type: Long * Unit: Percent * Is mutable: Yes * Description: High water memory threshold expressed as a percentage of the process memory limit. When total memory consumption rises above this percentage, BE begins to free memory gradually (currently by evicting data cache and update cache) to relieve pressure. The monitor uses this value to compute memory\_high = mem\_limit \* memory\_high\_level / 100 and, if consumption `>` memory\_high, performs controlled eviction guided by the GC advisor; if consumption exceeds memory\_urgent\_level (a separate config), more aggressive immediate reductions occur. This value is also consulted to disable certain memory‑intensive operations (for example, primary-key preload) when the threshold is exceeded. Must satisfy validation with memory\_urgent\_level (memory\_urgent\_level `>` memory\_high\_level, memory\_high\_level `>=` 1, memory\_urgent\_level `<=` 100). * Introduced in: v3.2.0 ##### report\_exec\_rpc\_request\_retry\_num[​](#report_exec_rpc_request_retry_num "Direct link to report_exec_rpc_request_retry_num") * Default: 10 * Type: Int * Unit: - * Is mutable: Yes * Description: The retry times of rpc request to report exec rpc request to FE. The default value is 10, which means that the rpc request will be retried 10 times if it fails only if it's fragment instatnce finish rpc. Report exec rpc request is important for load job, if one fragment instance finish report failed, the load job will be hang until timeout. * Introduced in: - ##### sleep\_one\_second[​](#sleep_one_second "Direct link to sleep_one_second") * Default: 1 * Type: Int * Unit: Seconds * Is mutable: No * Description: A small, global sleep interval (in seconds) used by BE agent worker threads as a one-second pause when the master address/heartbeat is not yet available or when a short retry/backoff is needed. In the codebase it is referenced by several report worker pools (e.g., ReportDiskStateTaskWorkerPool, ReportOlapTableTaskWorkerPool, ReportWorkgroupTaskWorkerPool) to avoid busy-waiting and reduce CPU consumption while retrying. Increasing this value slows the retry frequency and responsiveness to master availability; reducing it increases polling rate and CPU usage. Adjust only with awareness of the trade-off between responsiveness and resource use. * Introduced in: v3.2.0 ##### small\_file\_dir[​](#small_file_dir "Direct link to small_file_dir") * Default: `${STARROCKS_HOME}/lib/small_file/` * Type: String * Unit: - * Is mutable: No * Description: The directory used to store the files downloaded by the file manager. * Introduced in: - ##### upload\_buffer\_size[​](#upload_buffer_size "Direct link to upload_buffer_size") * Default: 4194304 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: Buffer size (in bytes) used by file copy operations when uploading snapshot files to remote storage (broker or direct FileSystem). In the upload path (snapshot\_loader.cpp) this value is passed to fs::copy as the read/write chunk size for each upload stream. The default is 4 MiB. Increasing this value can improve throughput on high-latency or high-bandwidth links but increases memory usage per concurrent upload; decreasing it reduces per-stream memory but may reduce transfer efficiency. Tune together with upload\_worker\_count and overall available memory. * Introduced in: v3.2.13 ##### user\_function\_dir[​](#user_function_dir "Direct link to user_function_dir") * Default: `${STARROCKS_HOME}/lib/udf` * Type: String * Unit: - * Is mutable: No * Description: The directory used to store User-defined Functions (UDFs). * Introduced in: - ##### web\_log\_bytes[​](#web_log_bytes "Direct link to web_log_bytes") * Default: 1048576 (1 MB) * Type: long * Unit: Bytes * Is mutable: No * Description: Maximum number of bytes to read from the INFO logfile and show on the BE debug webserver's log page. The handler uses this value to compute a seek offset (showing the last N bytes) to avoid reading or serving very large log files. If the logfile is smaller than this value the whole file is shown. Note: in the current implementation the code that reads and serves the INFO log is commented out and the handler reports that the INFO log file couldn't be opened, so this parameter may have no effect unless the log-serving code is enabled. * Introduced in: v3.2.0 --- ### BE Configuration - Statistics and Storage Some BE configuration items are dynamic parameters which you can set interactively when BE nodes are online. The rest of them are static parameters. You can only set the static parameters of a BE node by changing them in the corresponding configuration file **be.conf** and restarting the BE node to allow the change to take effect. #### View BE configuration items[​](#view-be-configuration-items "Direct link to View BE configuration items") You can view the BE configuration items using the following command: ```sql SELECT * FROM information_schema.be_configs [WHERE NAME LIKE "%%"] ``` #### Configure BE parameters[​](#configure-be-parameters "Direct link to Configure BE parameters") ##### Configure BE dynamic parameters[​](#configure-be-dynamic-parameters "Direct link to Configure BE dynamic parameters") You can configure a dynamic parameter of a BE node by updating the value in `information_schema.be_configs`. warning Setting an invalid value may cause unknown behaviors. Check twice before you run the command to update the configuration. ```sql -- Replace with the key of the configuration and with the value. UPDATE information_schema.be_configs SET VALUE = WHERE name = ""; ``` ##### Configure BE static parameters[​](#configure-be-static-parameters "Direct link to Configure BE static parameters") You can only set the static parameters of a BE by changing them in the corresponding configuration file **be.conf**, and restarting the BE to allow the changes to take effect. *** This topic introduces the following types of BE configurations: * [Statistic report](#statistic-report) * [Storage](#storage) #### Statistic report[​](#statistic-report "Direct link to Statistic report") ##### enable\_metric\_calculator[​](#enable_metric_calculator "Direct link to enable_metric_calculator") * Default: true * Type: boolean * Unit: - * Is mutable: No * Description: When true, the BE process launches a background "metrics\_daemon" thread (started in Daemon::init on non-Apple platforms) that runs every ~15 seconds to invoke `StarRocksMetrics::instance()->metrics()->trigger_hook()` and compute derived/system metrics (e.g., push/query bytes/sec, max disk I/O util, max network send/receive rates), log memory breakdowns and run table metrics cleanup. When false, those hooks are executed synchronously inside `MetricRegistry::collect` at metric collection time, which can increase metric-scrape latency. Requires process restart to take effect. * Introduced in: v3.2.0 ##### enable\_system\_metrics[​](#enable_system_metrics "Direct link to enable_system_metrics") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: When true, StarRocks initializes system-level monitoring during startup: it discovers disk devices from the configured store paths and enumerates network interfaces, then passes this information into the metrics subsystem to enable collection of disk I/O, network traffic and memory-related system metrics. If device or interface discovery fails, initialization logs a warning and aborts system metrics setup. This flag only controls whether system metrics are initialized; periodic metric aggregation threads are controlled separately by `enable_metric_calculator`, and JVM metrics initialization is controlled by `enable_jvm_metrics`. Changing this value requires a restart. * Introduced in: v3.2.0 ##### profile\_report\_interval[​](#profile_report_interval "Direct link to profile_report_interval") * Default: 30 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Interval in seconds that the ProfileReportWorker uses to (1) decide when to report per-fragment profile information for LOAD queries and (2) sleep between reporting cycles. The worker compares current time against each task's last\_report\_time using (profile\_report\_interval \* 1000) ms to determine if a profile should be re-reported for both non-pipeline and pipeline load tasks. At each loop the worker reads the current value (mutable at runtime); if the configured value is less than or euqual to 0 the worker forces it to 1 and emits a warning. Changing this value affects the next reporting decision and sleep duration. * Introduced in: v3.2.0 ##### report\_disk\_state\_interval\_seconds[​](#report_disk_state_interval_seconds "Direct link to report_disk_state_interval_seconds") * Default: 60 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval at which to report the storage volume state, which includes the size of data within the volume. * Introduced in: - ##### report\_resource\_usage\_interval\_ms[​](#report_resource_usage_interval_ms "Direct link to report_resource_usage_interval_ms") * Default: 1000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: Interval, in milliseconds, between periodic resource-usage reports sent by the BE agent to the FE (master). The agent worker thread collects TResourceUsage (number of running queries, memory used/limit, CPU used permille, and resource-group usages) and calls report\_task, then sleeps for this configured interval (see task\_worker\_pool). Lower values increase reporting timeliness but raise CPU, network, and master load; higher values reduce overhead but make resource information less current. The reporting updates related metrics (report\_resource\_usage\_requests\_total, report\_resource\_usage\_requests\_failed). Tune according to cluster scale and FE load. * Introduced in: v3.2.0 ##### report\_tablet\_interval\_seconds[​](#report_tablet_interval_seconds "Direct link to report_tablet_interval_seconds") * Default: 60 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval at which to report the most updated version of all tablets. * Introduced in: - ##### report\_task\_interval\_seconds[​](#report_task_interval_seconds "Direct link to report_task_interval_seconds") * Default: 10 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval at which to report the state of a task. A task can be creating a table, dropping a table, loading data, or changing a table schema. * Introduced in: - ##### report\_workgroup\_interval\_seconds[​](#report_workgroup_interval_seconds "Direct link to report_workgroup_interval_seconds") * Default: 5 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval at which to report the most updated version of all workgroups. * Introduced in: - #### Storage[​](#storage "Direct link to Storage") ##### alter\_tablet\_worker\_count[​](#alter_tablet_worker_count "Direct link to alter_tablet_worker_count") * Default: 3 * Type: Int * Unit: - * Is mutable: Yes * Description: The number of threads used for Schema Change. * Introduced in: - ##### automatic\_partition\_thread\_pool\_thread\_num[​](#automatic_partition_thread_pool_thread_num "Direct link to automatic_partition_thread_pool_thread_num") * Default: 1000 * Type: Int * Unit: - * Is mutable: No * Description: The number of threads in the automatic partition thread pool used for automatic partition creation during loading. The queue size of the pool is automatically set to 10 times the thread count. * Introduced in: - ##### avro\_ignore\_union\_type\_tag[​](#avro_ignore_union_type_tag "Direct link to avro_ignore_union_type_tag") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to strip the type tag from the JSON string serialized from the Avro Union data type. * Introduced in: v3.3.7, v3.4 ##### base\_compaction\_check\_interval\_seconds[​](#base_compaction_check_interval_seconds "Direct link to base_compaction_check_interval_seconds") * Default: 60 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval of thread polling for a Base Compaction. * Introduced in: - ##### base\_compaction\_interval\_seconds\_since\_last\_operation[​](#base_compaction_interval_seconds_since_last_operation "Direct link to base_compaction_interval_seconds_since_last_operation") * Default: 86400 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval since the last Base Compaction. This configuration item is one of the conditions that trigger a Base Compaction. * Introduced in: - ##### base\_compaction\_num\_threads\_per\_disk[​](#base_compaction_num_threads_per_disk "Direct link to base_compaction_num_threads_per_disk") * Default: 1 * Type: Int * Unit: - * Is mutable: No * Description: The number of threads used for Base Compaction on each storage volume. * Introduced in: - ##### base\_cumulative\_delta\_ratio[​](#base_cumulative_delta_ratio "Direct link to base_cumulative_delta_ratio") * Default: 0.3 * Type: Double * Unit: - * Is mutable: Yes * Description: The ratio of cumulative file size to base file size. The ratio reaching this value is one of the conditions that trigger the Base Compaction. * Introduced in: - ##### chaos\_test\_enable\_random\_compaction\_strategy[​](#chaos_test_enable_random_compaction_strategy "Direct link to chaos_test_enable_random_compaction_strategy") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: When this item is set to `true`, TabletUpdates::compaction() uses the random compaction strategy (compaction\_random) intended for chaos engineering tests. This flag forces compaction to follow a nondeterministic/random policy instead of normal strategies (e.g., size-tiered compaction), and takes precedence during compaction selection for the tablet. It is intended only for controlled testing: enabling it can produce unpredictable compaction order, increased I/O/CPU, and test flakiness. Do not enable in production; use only for fault-injection or chaos-test scenarios. * Introduced in: v3.3.12, 3.4.2, 3.5.0, 4.0.0 ##### check\_consistency\_worker\_count[​](#check_consistency_worker_count "Direct link to check_consistency_worker_count") * Default: 1 * Type: Int * Unit: - * Is mutable: No * Description: The number of threads used for checking the consistency of tablets. * Introduced in: - ##### clear\_expired\_replication\_snapshots\_interval\_seconds[​](#clear_expired_replication_snapshots_interval_seconds "Direct link to clear_expired_replication_snapshots_interval_seconds") * Default: 3600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval at which the system clears the expired snapshots left by abnormal replications. * Introduced in: v3.3.5 ##### compact\_threads[​](#compact_threads "Direct link to compact_threads") * Default: 4 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of threads used for concurrent compaction tasks. This configuration is changed to dynamic from v3.1.7 and v3.2.2 onwards. * Introduced in: v3.0.0 ##### compaction\_max\_memory\_limit[​](#compaction_max_memory_limit "Direct link to compaction_max_memory_limit") * Default: -1 * Type: Long * Unit: Bytes * Is mutable: No * Description: Global upper bound (in bytes) for memory available to compaction tasks on this BE. During BE initialization the final compaction memory limit is computed as min(`compaction_max_memory_limit`, process\_mem\_limit \* `compaction_max_memory_limit_percent` / 100). If `compaction_max_memory_limit` is negative (default `-1`) it falls back to the BE process memory limit derived from `mem_limit`. The percent value is clamped to \[0,100]. If the process memory limit is not set (negative) compaction memory remains unlimited (`-1`). This computed value is used to initialize the `_compaction_mem_tracker`. See also `compaction_max_memory_limit_percent` and `compaction_memory_limit_per_worker`. * Introduced in: v3.2.0 ##### compaction\_max\_memory\_limit\_percent[​](#compaction_max_memory_limit_percent "Direct link to compaction_max_memory_limit_percent") * Default: 100 * Type: Int * Unit: Percent * Is mutable: No * Description: Percentage of the BE process memory that may be used for compaction. The BE computes the compaction memory cap as the minimum of `compaction_max_memory_limit` and (process memory limit × this percent / 100). If this value is < 0 or > 100 it is treated as 100. If `compaction_max_memory_limit` < 0 the process memory limit is used instead. The calculation also considers the BE process memory derived from `mem_limit`. Combined with `compaction_memory_limit_per_worker` (per-worker cap), this setting controls total compaction memory available and therefore affects compaction concurrency and OOM risk. * Introduced in: v3.2.0 ##### compaction\_memory\_limit\_per\_worker[​](#compaction_memory_limit_per_worker "Direct link to compaction_memory_limit_per_worker") * Default: 2147483648 * Type: Int * Unit: Bytes * Is mutable: No * Description: The maximum memory size allowed for each Compaction thread. * Introduced in: - ##### compaction\_chunk\_reset\_memory\_tracker\_threshold\_percent[​](#compaction_chunk_reset_memory_tracker_threshold_percent "Direct link to compaction_chunk_reset_memory_tracker_threshold_percent") * Default: -1 * Type: Int * Unit: Percent * Is mutable: Yes * Description: Controls when compaction releases retained internal chunk capacity. Currently, this parameter takes effect only for Primary Key table compaction in shared-nothing clusters. When the current compaction task memory tracker consumption exceeds `compaction_memory_limit_per_worker * compaction_chunk_reset_memory_tracker_threshold_percent / 100`, StarRocks releases retained chunk capacity while resetting internal chunks. A negative value disables this behavior. * Introduced in: - ##### compaction\_trace\_threshold[​](#compaction_trace_threshold "Direct link to compaction_trace_threshold") * Default: 60 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time threshold for each compaction. If a compaction takes more time than the time threshold, StarRocks prints the corresponding trace. * Introduced in: - ##### create\_tablet\_worker\_count[​](#create_tablet_worker_count "Direct link to create_tablet_worker_count") * Default: 3 * Type: Int * Unit: Threads * Is mutable: Yes * Description: Sets the maximum number of worker threads in the AgentServer thread pool that process TTaskType::CREATE (create-tablet) tasks submitted by FE. At BE startup this value is used as the thread-pool max (the pool is created with min threads = 1 and max queue size = unlimited), and changing it at runtime triggers `ExecEnv::agent_server()->get_thread_pool(TTaskType::CREATE)->update_max_threads(...)`. Increase this to raise concurrent tablet creation throughput (useful during bulk load or partition creation); decreasing it throttles concurrent create operations. Raising the value increases CPU, memory and I/O concurrency and may cause contention; the thread pool enforces at least one thread, so values less than 1 have no practical effect. * Introduced in: v3.2.0 ##### cumulative\_compaction\_check\_interval\_seconds[​](#cumulative_compaction_check_interval_seconds "Direct link to cumulative_compaction_check_interval_seconds") * Default: 1 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval of thread polling for a Cumulative Compaction. * Introduced in: - ##### cumulative\_compaction\_num\_threads\_per\_disk[​](#cumulative_compaction_num_threads_per_disk "Direct link to cumulative_compaction_num_threads_per_disk") * Default: 1 * Type: Int * Unit: - * Is mutable: No * Description: The number of Cumulative Compaction threads per disk. * Introduced in: - ##### data\_page\_size[​](#data_page_size "Direct link to data_page_size") * Default: 65536 * Type: Int * Unit: Bytes * Is mutable: No * Description: Target uncompressed page size (in bytes) used when building column data and index pages. This value is copied into ColumnWriterOptions.data\_page\_size and IndexedColumnWriterOptions.index\_page\_size and is consulted by page builders (e.g., BinaryPlainPageBuilder::is\_page\_full and buffer reservation logic) to decide when to finish a page and how much memory to reserve. A value of 0 disables the page-size limit in builders. Changing this value affects page count, metadata overhead, memory reservation and I/O/compression trade-offs (smaller pages → more pages and metadata; larger pages → fewer pages, potentially better compression but larger memory spikes). * Introduced in: v3.2.4 ##### default\_num\_rows\_per\_column\_file\_block[​](#default_num_rows_per_column_file_block "Direct link to default_num_rows_per_column_file_block") * Default: 1024 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of rows that can be stored in each row block. * Introduced in: - ##### delete\_worker\_count\_high\_priority[​](#delete_worker_count_high_priority "Direct link to delete_worker_count_high_priority") * Default: 1 * Type: Int * Unit: Threads * Is mutable: No * Description: Number of worker threads in the DeleteTaskWorkerPool that are allocated as HIGH-priority delete threads. On startup AgentServer creates the delete pool with total threads = delete\_worker\_count\_normal\_priority + delete\_worker\_count\_high\_priority; the first delete\_worker\_count\_high\_priority threads are marked to exclusively try to pop TPriority::HIGH tasks (they poll for high-priority delete tasks and sleep/loop if none are available). Increasing this value increases concurrency for high-priority delete requests; decreasing it reduces dedicated capacity and may increase latency for high-priority deletes. * Introduced in: v3.2.0 ##### dictionary\_encoding\_ratio[​](#dictionary_encoding_ratio "Direct link to dictionary_encoding_ratio") * Default: 0.7 * Type: Double * Unit: - * Is mutable: No * Description: Fraction (0.0–1.0) used by StringColumnWriter during the encode-speculation phase to decide between dictionary (DICT\_ENCODING) and plain (PLAIN\_ENCODING) encoding for a chunk. The code computes max\_card = row\_count \* `dictionary_encoding_ratio` and scans the chunk’s distinct key count; if the distinct count exceeds max\_card the writer chooses PLAIN\_ENCODING. The check is performed only when the chunk size passes `dictionary_speculate_min_chunk_size` (and when row\_count > dictionary\_min\_rowcount). Setting the value higher favors dictionary encoding (tolerates more distinct keys); setting it lower causes earlier fallback to plain encoding. A value of 1.0 effectively forces dictionary encoding (distinct count can never exceed row\_count). * Introduced in: v3.2.0 ##### dictionary\_encoding\_ratio\_for\_non\_string\_column[​](#dictionary_encoding_ratio_for_non_string_column "Direct link to dictionary_encoding_ratio_for_non_string_column") * Default: 0 * Type: double * Unit: - * Is mutable: No * Description: Ratio threshold used to decide whether to use dictionary encoding for non-string columns (numeric, date/time, decimal types). When enabled (value > 0.0001) the writer computes max\_card = row\_count \* dictionary\_encoding\_ratio\_for\_non\_string\_column and, for samples with row\_count > `dictionary_min_rowcount`, chooses DICT\_ENCODING only if distinct\_count ≤ max\_card; otherwise it falls back to BIT\_SHUFFLE. A value of `0` (default) disables non-string dictionary encoding. This parameter is analogous to `dictionary_encoding_ratio` but applies to non-string columns. Use values in (0,1] — smaller values restrict dictionary encoding to lower-cardinality columns and reduce dictionary memory/IO overhead. * Introduced in: v3.3.0, v3.4.0, v3.5.0 ##### dictionary\_page\_size[​](#dictionary_page_size "Direct link to dictionary_page_size") * Default: 1048576 * Type: Int * Unit: Bytes * Is mutable: No * Description: Size in bytes of dictionary pages used when building rowset segments. This value is read into `PageBuilderOptions::dict_page_size` in the BE rowset code and controls how many dictionary entries can be stored in a single dictionary page. Increasing this value can improve compression ratio for dictionary-encoded columns by allowing larger dictionaries, but larger pages consume more memory during write/encode and can increase I/O and latency when reading or materializing pages. Set conservatively for large-memory, write-heavy workloads and avoid excessively large values to prevent runtime performance degradation. * Introduced in: v3.3.0, v3.4.0, v3.5.0 ##### disk\_stat\_monitor\_interval[​](#disk_stat_monitor_interval "Direct link to disk_stat_monitor_interval") * Default: 5 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval at which to monitor health status of disks. * Introduced in: - ##### download\_low\_speed\_limit\_kbps[​](#download_low_speed_limit_kbps "Direct link to download_low_speed_limit_kbps") * Default: 50 * Type: Int * Unit: KB/Second * Is mutable: Yes * Description: The download speed lower limit of each HTTP request. An HTTP request aborts when it constantly runs with a lower speed than this value within the time span specified in the configuration item `download_low_speed_time`. * Introduced in: - ##### download\_low\_speed\_time[​](#download_low_speed_time "Direct link to download_low_speed_time") * Default: 300 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The maximum time that an HTTP request can run with a download speed lower than the limit. An HTTP request aborts when it constantly runs with a lower speed than the value of `download_low_speed_limit_kbps` within the time span specified in this configuration item. * Introduced in: - ##### download\_worker\_count[​](#download_worker_count "Direct link to download_worker_count") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of threads for the download tasks of restore jobs on a BE node. `0` indicates setting the value to the number of CPU cores on the machine where the BE resides. * Introduced in: - ##### drop\_tablet\_worker\_count[​](#drop_tablet_worker_count "Direct link to drop_tablet_worker_count") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: The number of threads used to drop a tablet. `0` indicates half of the CPU cores in the node. * Introduced in: - ##### enable\_check\_string\_lengths[​](#enable_check_string_lengths "Direct link to enable_check_string_lengths") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to check the data length during loading to solve compaction failures caused by out-of-bound VARCHAR data. * Introduced in: - ##### enable\_event\_based\_compaction\_framework[​](#enable_event_based_compaction_framework "Direct link to enable_event_based_compaction_framework") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to enable the Event-based Compaction Framework. `true` indicates Event-based Compaction Framework is enabled, and `false` indicates it is disabled. Enabling Event-based Compaction Framework can greatly reduce the overhead of compaction in scenarios where there are many tablets or a single tablet has a large amount of data. * Introduced in: - ##### enable\_lazy\_delta\_column\_compaction[​](#enable_lazy_delta_column_compaction "Direct link to enable_lazy_delta_column_compaction") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: When enabled, compaction will prefer a "lazy" strategy for delta columns produced by partial column updates: StarRocks will avoid eagerly merging delta-column files back into their main segment files to save compaction I/O. In practice the compaction selection code checks for partial column-update rowsets and multiple candidates; if found and this flag is true, the engine will either stop adding further inputs to the compaction or only merge empty rowsets (level -1), leaving delta columns separate. This reduces immediate I/O and CPU during compaction at the cost of delayed consolidation (potentially more segments and temporary storage overhead). Correctness and query semantics are unchanged. * Introduced in: v3.2.3 ##### enable\_new\_load\_on\_memory\_limit\_exceeded[​](#enable_new_load_on_memory_limit_exceeded "Direct link to enable_new_load_on_memory_limit_exceeded") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to allow new loading processes when the hard memory resource limit is reached. `true` indicates new loading processes will be allowed, and `false` indicates they will be rejected. * Introduced in: v3.3.2 ##### enable\_pk\_index\_parallel\_compaction[​](#enable_pk_index_parallel_compaction "Direct link to enable_pk_index_parallel_compaction") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable parallel Compaction for Primary Key index in a shared-data cluster. * Introduced in: - ##### enable\_pk\_index\_parallel\_execution[​](#enable_pk_index_parallel_execution "Direct link to enable_pk_index_parallel_execution") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable parallel execution for Primary Key index operations in a shared-data cluster. When enabled, the system uses a thread pool to process segments concurrently during publish operations, significantly improving performance for large tablets. * Introduced in: - ##### enable\_pk\_size\_tiered\_compaction\_strategy[​](#enable_pk_size_tiered_compaction_strategy "Direct link to enable_pk_size_tiered_compaction_strategy") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to enable the Size-tiered Compaction policy for Primary Key tables. `true` indicates the Size-tiered Compaction strategy is enabled, and `false` indicates it is disabled. * Introduced in: This item takes effect for shared-data clusters from v3.2.4 and v3.1.10 onwards, and for shared-nothing clusters from v3.2.5 and v3.1.10 onwards. ##### enable\_rowset\_verify[​](#enable_rowset_verify "Direct link to enable_rowset_verify") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to verify the correctness of generated rowsets. When enabled, the correctness of the generated rowsets will be checked after Compaction and Schema Change. * Introduced in: - ##### enable\_size\_tiered\_compaction\_strategy[​](#enable_size_tiered_compaction_strategy "Direct link to enable_size_tiered_compaction_strategy") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to enable the Size-tiered Compaction policy (excluding Primary Key tables). `true` indicates the Size-tiered Compaction strategy is enabled, and `false` indicates it is disabled. * Introduced in: - ##### enable\_strict\_delvec\_crc\_check[​](#enable_strict_delvec_crc_check "Direct link to enable_strict_delvec_crc_check") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: When enable\_strict\_delvec\_crc\_check is set to true, we will perform a strict CRC32 check on the delete vector, and if a mismatch is detected, a failure will be returned. * Introduced in: - ##### enable\_transparent\_data\_encryption[​](#enable_transparent_data_encryption "Direct link to enable_transparent_data_encryption") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: When enabled, StarRocks will create encrypted on‑disk artifacts for newly written storage objects (segment files, delete/update files, rowset segments, lake SSTs, persistent index files, etc.). Writers (RowsetWriter/SegmentWriter, lake UpdateManager/LakePersistentIndex and related code paths) will request encryption info from the KeyCache, attach encryption\_info to writable files and persist encryption\_meta into rowset / segment / sstable metadata (segment\_encryption\_metas, delete/update encryption metadata). The Frontend and Backend/CN encryption flags must match — a mismatch causes the BE to abort on heartbeat (LOG(FATAL)). This flag is not runtime‑mutable; enable it before deployment and ensure key management (KEK) and KeyCache are properly configured and synchronized across the cluster. * Introduced in: v3.3.1, 3.4.0, 3.5.0, 4.0.0 ##### enable\_zero\_copy\_from\_page\_cache[​](#enable_zero_copy_from_page_cache "Direct link to enable_zero_copy_from_page_cache") * Default: true * Type: boolean * Unit: - * Is mutable: Yes * Description: When enabled, FixedLengthColumnBase may avoid copying bytes when appending data that originates from a page-cache-backed buffer. In append\_numbers the code will acquire the incoming ContainerResource and set the column's internal resource pointer (zero-copy) if all conditions are met: the config is true, the incoming resource is owned, the resource memory is aligned for the column element type, the column is empty, and the resource length is a multiple of the element size. Enabling this reduces CPU and memory-copy overhead and can improve ingestion/scan throughput. Drawbacks: it couples the column lifetime to the acquired buffer and relies on correct ownership/alignment; disable to force safe copying. * Introduced in: - ##### file\_descriptor\_cache\_clean\_interval[​](#file_descriptor_cache_clean_interval "Direct link to file_descriptor_cache_clean_interval") * Default: 3600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval at which to clean file descriptors that have not been used for a certain period of time. * Introduced in: - ##### ignore\_broken\_disk[​](#ignore_broken_disk "Direct link to ignore_broken_disk") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: Controls startup behavior when configured storage paths fail read/write checks or fail to parse. When `false` (default), BE treats any broken entry in `storage_root_path` or `spill_local_storage_dir` as fatal and will abort startup. When `true`, StarRocks will skip (log a warning and remove) any storage path that fails `check_datapath_rw` or fails parsing so the BE can continue starting with the remaining healthy paths. Note: if all configured paths are removed, BE will still exit. Enabling this can mask misconfigured or failed disks and cause data on ignored paths to be unavailable; monitor logs and disk health accordingly. * Introduced in: v3.2.0 ##### inc\_rowset\_expired\_sec[​](#inc_rowset_expired_sec "Direct link to inc_rowset_expired_sec") * Default: 1800 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The expiration time of the incoming data. This configuration item is used in incremental clone. * Introduced in: - ##### load\_process\_max\_memory\_hard\_limit\_ratio[​](#load_process_max_memory_hard_limit_ratio "Direct link to load_process_max_memory_hard_limit_ratio") * Default: 2 * Type: Int * Unit: - * Is mutable: Yes * Description: The hard limit (ratio) of memory resources that can be taken up by all load processes on a BE node. When `enable_new_load_on_memory_limit_exceeded` is set to `false`, and the memory consumption of all loading processes exceeds `load_process_max_memory_limit_percent * load_process_max_memory_hard_limit_ratio`, new loading processes will be rejected. * Introduced in: v3.3.2 ##### load\_process\_max\_memory\_limit\_percent[​](#load_process_max_memory_limit_percent "Direct link to load_process_max_memory_limit_percent") * Default: 30 * Type: Int * Unit: - * Is mutable: No * Description: The soft limit (in percentage) of memory resources that can be taken up by all load processes on a BE node. * Introduced in: - ##### lz4\_acceleration[​](#lz4_acceleration "Direct link to lz4_acceleration") * Default: 1 * Type: Int * Unit: - * Is mutable: Yes * Description: Controls the LZ4 "acceleration" parameter used by the built-in LZ4 compressor (passed to LZ4\_compress\_fast\_continue). Higher values prioritize compression speed at the cost of compression ratio; lower values (1) produce better compression but are slower. Valid range: MIN=1, MAX=65537. This setting affects all LZ4-based codecs in BlockCompression (e.g., LZ4 and Hadoop-LZ4) and only changes how compression is performed — it does not change the LZ4 format or decompression compatibility. Tune upward (e.g., 4, 8, ...) for CPU-bound or low-latency workloads where larger output is acceptable; keep at 1 for storage- or IO-sensitive workloads. Test with representative data before changing, since throughput vs. size trade-offs are highly data-dependent. * Introduced in: v3.4.1, 3.5.0, 4.0.0 ##### lz4\_expected\_compression\_ratio[​](#lz4_expected_compression_ratio "Direct link to lz4_expected_compression_ratio") * Default: 2.1 * Type: double * Unit: Dimensionless (compression ratio) * Is mutable: Yes * Description: Threshold used by the serialization compression strategy to judge whether observed LZ4 compression is "good". In compress\_strategy.cpp this value divides the observed compress\_ratio when computing a reward metric together with lz4\_expected\_compression\_speed\_mbps; if the combined reward `>` 1.0 the strategy records positive feedback. Increasing this value raises the expected compression ratio (making the condition harder to satisfy), while lowering it makes it easier for observed compression to be considered satisfactory. Tune to match typical data compressibility. Valid range: MIN=1, MAX=65537. * Introduced in: v3.4.1, 3.5.0, 4.0.0 ##### lz4\_expected\_compression\_speed\_mbps[​](#lz4_expected_compression_speed_mbps "Direct link to lz4_expected_compression_speed_mbps") * Default: 600 * Type: double * Unit: MB/s * Is mutable: Yes * Description: Expected LZ4 compression throughput in megabytes per second used by the adaptive compression policy (CompressStrategy). The feedback routine computes a reward\_ratio = (observed\_compression\_ratio / lz4\_expected\_compression\_ratio) \* (observed\_speed / lz4\_expected\_compression\_speed\_mbps). A reward\_ratio `>` 1.0 increments the positive counter (alpha), otherwise the negative counter (beta); this influences whether future data will be compressed. Tune this value to reflect typical LZ4 throughput on your hardware — raising it makes the policy harder to classify a run as "good" (requires higher observed speed), lowering it makes classification easier. Must be a positive finite number. * Introduced in: v3.4.1, 3.5.0, 4.0.0 ##### make\_snapshot\_worker\_count[​](#make_snapshot_worker_count "Direct link to make_snapshot_worker_count") * Default: 5 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of threads for the make snapshot tasks on a BE node. * Introduced in: - ##### manual\_compaction\_threads[​](#manual_compaction_threads "Direct link to manual_compaction_threads") * Default: 4 * Type: Int * Unit: - * Is mutable: No * Description: Number of threads for Manual Compaction. * Introduced in: - ##### max\_base\_compaction\_num\_singleton\_deltas[​](#max_base_compaction_num_singleton_deltas "Direct link to max_base_compaction_num_singleton_deltas") * Default: 100 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of segments that can be compacted in each Base Compaction. * Introduced in: - ##### max\_compaction\_candidate\_num[​](#max_compaction_candidate_num "Direct link to max_compaction_candidate_num") * Default: 40960 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of candidate tablets for compaction. If the value is too large, it will cause high memory usage and high CPU load. * Introduced in: - ##### max\_compaction\_concurrency[​](#max_compaction_concurrency "Direct link to max_compaction_concurrency") * Default: -1 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum concurrency of compactions (including both Base Compaction and Cumulative Compaction). The value `-1` indicates that no limit is imposed on the concurrency. `0` indicates disabling compaction. This parameter is mutable when the Event-based Compaction Framework is enabled. * Introduced in: - ##### max\_cumulative\_compaction\_num\_singleton\_deltas[​](#max_cumulative_compaction_num_singleton_deltas "Direct link to max_cumulative_compaction_num_singleton_deltas") * Default: 1000 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of segments that can be merged in a single Cumulative Compaction. You can reduce this value if OOM occurs during compaction. * Introduced in: - ##### max\_download\_speed\_kbps[​](#max_download_speed_kbps "Direct link to max_download_speed_kbps") * Default: 50000 * Type: Int * Unit: KB/Second * Is mutable: Yes * Description: The maximum download speed of each HTTP request. This value affects the performance of data replica synchronization across BE nodes. * Introduced in: - ##### max\_garbage\_sweep\_interval[​](#max_garbage_sweep_interval "Direct link to max_garbage_sweep_interval") * Default: 3600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The maximum time interval for garbage collection on storage volumes. This configuration is changed to dynamic from v3.0 onwards. * Introduced in: - ##### max\_percentage\_of\_error\_disk[​](#max_percentage_of_error_disk "Direct link to max_percentage_of_error_disk") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum percentage of error that is tolerable in a storage volume before the corresponding BE node quits. * Introduced in: - ##### max\_queueing\_memtable\_per\_tablet[​](#max_queueing_memtable_per_tablet "Direct link to max_queueing_memtable_per_tablet") * Default: 2 * Type: Long * Unit: Count * Is mutable: Yes * Description: Controls per-tablet backpressure for write paths: when a tablet's number of queueing (not-yet-flushing) memtables reaches or exceeds `max_queueing_memtable_per_tablet`, writers in `LocalTabletsChannel` and `LakeTabletsChannel` will block (sleep/retry) before submitting more write work. This reduces simultaneous memtable flush concurrency and peak memory use at the cost of increased latency or RPC timeouts for heavy load. Set higher to allow more concurrent memtables (more memory and I/O burst); set lower to limit memory pressure and increase write throttling. * Introduced in: v3.2.0 ##### max\_row\_source\_mask\_memory\_bytes[​](#max_row_source_mask_memory_bytes "Direct link to max_row_source_mask_memory_bytes") * Default: 209715200 * Type: Int * Unit: Bytes * Is mutable: No * Description: The maximum memory size of the row source mask buffer. When the buffer is larger than this value, data will be persisted to a temporary file on the disk. This value should be set lower than the value of `compaction_memory_limit_per_worker`. * Introduced in: - ##### max\_tablet\_write\_chunk\_bytes[​](#max_tablet_write_chunk_bytes "Direct link to max_tablet_write_chunk_bytes") * Default: 536870912 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: Maximum allowed memory (in bytes) for the current in-memory tablet write chunk before it is treated as full and enqueued for sending. Increase this value to reduce the frequency of RPCs when loading wide tables (many columns), which can improve throughput at the cost of higher memory usage and larger RPC payloads. Tune to balance fewer RPCs against memory and serialization/BRPC limits. * Introduced in: v3.2.12 ##### max\_update\_compaction\_num\_singleton\_deltas[​](#max_update_compaction_num_singleton_deltas "Direct link to max_update_compaction_num_singleton_deltas") * Default: 1000 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of rowsets that can be merged in a single Compaction for Primary Key tables. * Introduced in: - ##### memory\_limitation\_per\_thread\_for\_schema\_change[​](#memory_limitation_per_thread_for_schema_change "Direct link to memory_limitation_per_thread_for_schema_change") * Default: 2 * Type: Int * Unit: GB * Is mutable: Yes * Description: The maximum memory size allowed for each schema change task. * Introduced in: - ##### memory\_ratio\_for\_sorting\_schema\_change[​](#memory_ratio_for_sorting_schema_change "Direct link to memory_ratio_for_sorting_schema_change") * Default: 0.8 * Type: Double * Unit: - (unitless ratio) * Is mutable: Yes * Description: Fraction of the per-thread schema-change memory limit used as the memtable maximum buffer size during sorting schema-change operations. The ratio is multiplied by memory\_limitation\_per\_thread\_for\_schema\_change (configured in GB and converted to bytes) to compute max\_buffer\_size, and that result is capped at 4GB. Used by SchemaChangeWithSorting and SortedSchemaChange when creating MemTable/DeltaWriter. Increasing this ratio allows larger in-memory buffers (fewer flushes/merges) but raises risk of memory pressure; reducing it causes more frequent flushes and higher I/O/merge overhead. * Introduced in: v3.2.0 ##### min\_base\_compaction\_num\_singleton\_deltas[​](#min_base_compaction_num_singleton_deltas "Direct link to min_base_compaction_num_singleton_deltas") * Default: 5 * Type: Int * Unit: - * Is mutable: Yes * Description: The minimum number of segments that trigger a Base Compaction. * Introduced in: - ##### min\_compaction\_failure\_interval\_sec[​](#min_compaction_failure_interval_sec "Direct link to min_compaction_failure_interval_sec") * Default: 120 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The minimum time interval at which a tablet compaction can be scheduled since the previous compaction failure. * Introduced in: - ##### min\_cumulative\_compaction\_failure\_interval\_sec[​](#min_cumulative_compaction_failure_interval_sec "Direct link to min_cumulative_compaction_failure_interval_sec") * Default: 30 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The minimum time interval at which Cumulative Compaction retries upon failures. * Introduced in: - ##### min\_cumulative\_compaction\_num\_singleton\_deltas[​](#min_cumulative_compaction_num_singleton_deltas "Direct link to min_cumulative_compaction_num_singleton_deltas") * Default: 5 * Type: Int * Unit: - * Is mutable: Yes * Description: The minimum number of segments to trigger Cumulative Compaction. * Introduced in: - ##### min\_garbage\_sweep\_interval[​](#min_garbage_sweep_interval "Direct link to min_garbage_sweep_interval") * Default: 180 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The minimum time interval for garbage collection on storage volumes. This configuration is changed to dynamic from v3.0 onwards. * Introduced in: - ##### parallel\_clone\_task\_per\_path[​](#parallel_clone_task_per_path "Direct link to parallel_clone_task_per_path") * Default: 8 * Type: Int * Unit: Threads * Is mutable: Yes * Description: Number of parallel clone worker threads allocated per storage path on a BE. At BE startup the clone thread-pool max threads is computed as max(number\_of\_store\_paths \* parallel\_clone\_task\_per\_path, MIN\_CLONE\_TASK\_THREADS\_IN\_POOL). For example, with 4 storage paths and default=8 the clone pool max = 32. This setting directly controls concurrency of CLONE tasks (tablet replica copies) handled by the BE: increasing it raises parallel clone throughput but also increases CPU, disk and network contention; decreasing it limits simultaneous clone tasks and can throttle FE-scheduled clone operations. The value is applied to the dynamic clone thread pool and can be changed at runtime via the update-config path (causes agent\_server to update the clone pool max threads). * Introduced in: v3.2.0 ##### partial\_update\_memory\_limit\_per\_worker[​](#partial_update_memory_limit_per_worker "Direct link to partial_update_memory_limit_per_worker") * Default: 2147483648 * Type: long * Unit: Bytes * Is mutable: Yes * Description: Maximum memory (in bytes) a single worker may use for assembling a source chunk when performing partial column updates (used in compaction / rowset update processing). The reader estimates per-row update memory (total\_update\_row\_size / num\_rows\_upt) and multiplies it by the number of rows read; when that product exceeds this limit the current chunk is flushed and processed to avoid additional memory growth. Set this to match the available memory per update worker—too low increases I/O/processing overhead (many small chunks); too high risks memory pressure or OOM. If the per-row estimate is zero (legacy rowsets), this config does not impose a byte-based limit (only the INT32\_MAX row count limit applies). * Introduced in: v3.2.10 ##### path\_gc\_check[​](#path_gc_check "Direct link to path_gc_check") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: When enabled, StorageEngine starts per-data-dir background threads that perform periodic path scanning and garbage collection. On startup `start_bg_threads()` spawns `_path_scan_thread_callback` (calls `DataDir::perform_path_scan` and `perform_tmp_path_scan`) and `_path_gc_thread_callback` (calls `DataDir::perform_path_gc_by_tablet`, `DataDir::perform_path_gc_by_rowsetid`, `DataDir::perform_delta_column_files_gc`, and `DataDir::perform_crm_gc`). The scan and GC intervals are controlled by `path_scan_interval_second` and `path_gc_check_interval_second`; CRM file cleanup uses `unused_crm_file_threshold_second`. Disable this to prevent automatic path-level cleanup (you must then manage orphaned/temp files manually). Changing this flag requires restarting the process. * Introduced in: v3.2.0 ##### path\_gc\_check\_interval\_second[​](#path_gc_check_interval_second "Direct link to path_gc_check_interval_second") * Default: 86400 * Type: Int * Unit: Seconds * Is mutable: No * Description: Interval in seconds between runs of the storage engine's path garbage-collection background thread. Each wake triggers DataDir to perform path GC by tablet, by rowset id, delta column file GC and CRM GC (the CRM GC call uses `unused_crm_file_threshold_second`). If set to a non-positive value the code forces the interval to 1800 seconds (half hour) and emits a warning. Tune this to control how frequently on-disk temporary or downloaded files are scanned and removed. * Introduced in: v3.2.0 ##### pending\_data\_expire\_time\_sec[​](#pending_data_expire_time_sec "Direct link to pending_data_expire_time_sec") * Default: 1800 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The expiration time of the pending data in the storage engine. * Introduced in: - ##### pindex\_major\_compaction\_limit\_per\_disk[​](#pindex_major_compaction_limit_per_disk "Direct link to pindex_major_compaction_limit_per_disk") * Default: 1 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum concurrency of compaction on a disk. This addresses the issue of uneven I/O across disks due to compaction. This issue can cause excessively high I/O for certain disks. * Introduced in: v3.0.9 ##### pk\_index\_compaction\_score\_ratio[​](#pk_index_compaction_score_ratio "Direct link to pk_index_compaction_score_ratio") * Default: 1.5 * Type: Double * Unit: - * Is mutable: Yes * Description: Compaction score ratio for Primary Key index in a shared-data cluster. For example, if there are N filesets, the Compaction score will be `N * pk_index_compaction_score_ratio`. * Introduced in: - ##### pk\_index\_early\_sst\_compaction\_threshold[​](#pk_index_early_sst_compaction_threshold "Direct link to pk_index_early_sst_compaction_threshold") * Default: 5 * Type: Int * Unit: - * Is mutable: Yes * Description: early sst compaction threshold for primary key index in a shared-data cluster. * Introduced in: - ##### pk\_index\_map\_shard\_size[​](#pk_index_map_shard_size "Direct link to pk_index_map_shard_size") * Default: 4096 * Type: Int * Unit: - * Is mutable: No * Description: Number of shards used by the Primary Key index shard map in the lake UpdateManager. UpdateManager allocates a vector of `PkIndexShard` of this size and maps a tablet ID to a shard via a bitmask. Increasing this value reduces lock contention among tablets that would otherwise share the same shard, at the cost of more mutex objects and slightly higher memory usage. The value must be a power of two because the code relies on bitmask indexing. For sizing guidance see `tablet_map_shard_size` heuristic: `total_num_of_tablets_in_BE / 512`. * Introduced in: v3.2.0 ##### pk\_index\_memtable\_flush\_threadpool\_max\_threads[​](#pk_index_memtable_flush_threadpool_max_threads "Direct link to pk_index_memtable_flush_threadpool_max_threads") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of threads in the thread pool for Primary Key index MemTable flush in a shared-data cluster. `0` means automatically set to half of the number of CPU cores. * Introduced in: - ##### pk\_index\_memtable\_flush\_threadpool\_size[​](#pk_index_memtable_flush_threadpool_size "Direct link to pk_index_memtable_flush_threadpool_size") * Default: 1048576 * Type: Int * Unit: - * Is mutable: Yes * Description: Controls the maximum queue size (number of pending tasks) for the Primary Key index memtable flush thread pool used in shared-data (cloud-native / lake) mode. The thread pool is created as "cloud\_native\_pk\_index\_flush" in ExecEnv; its max thread count is governed by `pk_index_memtable_flush_threadpool_max_threads`. Increasing this value permits more memtable flush tasks to be buffered before execution, which can reduce immediate backpressure but increases memory consumed by queued task objects. Decreasing it limits buffered tasks and can cause earlier backpressure or task rejections depending on thread-pool behavior. Tune according to available memory and expected concurrent flush workload. * Introduced in: - ##### pk\_index\_memtable\_max\_count[​](#pk_index_memtable_max_count "Direct link to pk_index_memtable_max_count") * Default: 2 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of MemTables for Primary Key index in a shared-data cluster. * Introduced in: - ##### pk\_index\_memtable\_max\_wait\_flush\_timeout\_ms[​](#pk_index_memtable_max_wait_flush_timeout_ms "Direct link to pk_index_memtable_max_wait_flush_timeout_ms") * Default: 30000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: The maximum timeout for waiting for Primary Key index MemTable flush completion in a shared-data cluster. When synchronously flushing all MemTables (for example, before an ingest SST operation), the system waits up to this timeout. The default is 30 seconds. * Introduced in: - ##### pk\_index\_parallel\_compaction\_task\_split\_threshold\_bytes[​](#pk_index_parallel_compaction_task_split_threshold_bytes "Direct link to pk_index_parallel_compaction_task_split_threshold_bytes") * Default: 33554432 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: The splitting threshold for Primary Key index Compaction tasks. When the total size of the files involved in a task is smaller than this threshold, the task will not be split. * Introduced in: - ##### pk\_index\_parallel\_compaction\_threadpool\_max\_threads[​](#pk_index_parallel_compaction_threadpool_max_threads "Direct link to pk_index_parallel_compaction_threadpool_max_threads") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of threads in the thread pool for cloud native Primary Key index parallel Compaction in a shared-data cluster. `0` means automatically set to half of the number of CPU cores. * Introduced in: - ##### pk\_index\_parallel\_compaction\_threadpool\_size[​](#pk_index_parallel_compaction_threadpool_size "Direct link to pk_index_parallel_compaction_threadpool_size") * Default: 1048576 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum queue size (number of pending tasks) for the thread pool used by cloud-native Primary Key index parallel compaction in shared-data mode. This setting controls how many Compaction tasks can be enqueued before the thread pool rejects new submissions. The effective parallelism is bounded by `pk_index_parallel_compaction_threadpool_max_threads`; increase this value to avoid task rejections when you expect many concurrent Compaction tasks, but be aware larger queues can increase memory and latency for queued work. * Introduced in: - ##### pk\_index\_parallel\_execution\_min\_rows[​](#pk_index_parallel_execution_min_rows "Direct link to pk_index_parallel_execution_min_rows") * Default: 16384 * Type: Int * Unit: - * Is mutable: Yes * Description: The minimum rows threshold to enable parallel execution for Primary Key index operations in a shared-data cluster. * Introduced in: - ##### pk\_index\_parallel\_execution\_threadpool\_max\_threads[​](#pk_index_parallel_execution_threadpool_max_threads "Direct link to pk_index_parallel_execution_threadpool_max_threads") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of threads in the thread pool for Primary Key index parallel execution in a shared-data cluster. `0` means automatically set to half of the number of CPU cores. * Introduced in: - ##### pk\_index\_parallel\_rebuild\_mem\_ratio[​](#pk_index_parallel_rebuild_mem_ratio "Direct link to pk_index_parallel_rebuild_mem_ratio") * Default: 50 * Type: Int * Unit: percent (0-100) * Is mutable: Yes * Description: In a shared-data cluster, the memory-pressure gate for the parallel prefetch paths used while rebuilding the Primary Key index. When the update mem tracker is already past this percent of its limit, the rebuild falls back to a single-pass loop that holds only one decoded column at a time, trading the cold-start latency win for bounded peak memory. It gates parallel reads of delete files, segment files, and other files during the rebuild. Set to a higher value to allow the optimization under more memory pressure; set to `100` to disable the memory gate (always run the parallel path when `enable_pk_index_parallel_execution=true`). * Introduced in: - ##### lake\_partial\_update\_thread\_pool\_max\_threads[​](#lake_partial_update_thread_pool_max_threads "Direct link to lake_partial_update_thread_pool_max_threads") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of threads in the thread pool for lake partial update segment-level parallelism in a shared-data cluster. This thread pool is used by both row-mode and column-mode partial updates to parallelize I/O-heavy segment operations (load\_segment + rewrite\_segment for row-mode, DCG generation for column-mode). `0` means automatically set to half of the number of CPU cores. Runtime on/off is controlled by `enable_pk_index_parallel_execution`. * Introduced in: v4.1 ##### lake\_partial\_update\_thread\_pool\_queue\_size[​](#lake_partial_update_thread_pool_queue_size "Direct link to lake_partial_update_thread_pool_queue_size") * Default: 2048 * Type: Int * Unit: - * Is mutable: Yes * Description: The task queue size for the lake partial update thread pool. * Introduced in: v4.1 ##### pk\_index\_size\_tiered\_level\_multiplier[​](#pk_index_size_tiered_level_multiplier "Direct link to pk_index_size_tiered_level_multiplier") * Default: 10 * Type: Int * Unit: - * Is mutable: Yes * Description: The level multiplier parameter for Primary Key index size-tiered Compaction strategy. * Introduced in: - ##### pk\_index\_size\_tiered\_max\_level[​](#pk_index_size_tiered_max_level "Direct link to pk_index_size_tiered_max_level") * Default: 5 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum level for Primary Key index size-tiered Compaction strategy. * Introduced in: - ##### pk\_index\_size\_tiered\_min\_level\_size[​](#pk_index_size_tiered_min_level_size "Direct link to pk_index_size_tiered_min_level_size") * Default: 131072 * Type: Int * Unit: - * Is mutable: Yes * Description: The minimum level for Primary Key index size-tiered Compaction strategy. * Introduced in: - ##### pk\_index\_sstable\_sample\_interval\_bytes[​](#pk_index_sstable_sample_interval_bytes "Direct link to pk_index_sstable_sample_interval_bytes") * Default: 16777216 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: The sampling interval size for SSTable files in a shared-data cluster. When the size of an SSTable file exceeds this threshold, the system samples keys from the SSTable at this interval to optimize the boundary partitioning of Compaction tasks. For SSTables smaller than this threshold, only the start key is used as the boundary key. The default is 16 MB. * Introduced in: - ##### pk\_index\_target\_file\_size[​](#pk_index_target_file_size "Direct link to pk_index_target_file_size") * Default: 67108864 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: The target file size for Primary Key index in a shared-data cluster. * Introduced in: - ##### pk\_index\_eager\_build\_threshold\_bytes[​](#pk_index_eager_build_threshold_bytes "Direct link to pk_index_eager_build_threshold_bytes") * Default: 104857600 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: The minimum size of data generated during import or compaction for the system to eagerly build PK index files. Default is 100MB. * Introduced in: - ##### primary\_key\_limit\_size[​](#primary_key_limit_size "Direct link to primary_key_limit_size") * Default: 128 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: The maximum size of a key column in Primary Key tables. * Introduced in: v2.5 ##### release\_snapshot\_worker\_count[​](#release_snapshot_worker_count "Direct link to release_snapshot_worker_count") * Default: 5 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of threads for the release snapshot tasks on a BE node. * Introduced in: - ##### repair\_compaction\_interval\_seconds[​](#repair_compaction_interval_seconds "Direct link to repair_compaction_interval_seconds") * Default: 600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval to poll Repair Compaction threads. * Introduced in: - ##### replication\_max\_speed\_limit\_kbps[​](#replication_max_speed_limit_kbps "Direct link to replication_max_speed_limit_kbps") * Default: 50000 * Type: Int * Unit: KB/s * Is mutable: Yes * Description: The maximum speed of each replication thread. * Introduced in: v3.3.5 ##### replication\_min\_speed\_limit\_kbps[​](#replication_min_speed_limit_kbps "Direct link to replication_min_speed_limit_kbps") * Default: 50 * Type: Int * Unit: KB/s * Is mutable: Yes * Description: The minimum speed of each replication thread. * Introduced in: v3.3.5 ##### replication\_min\_speed\_time\_seconds[​](#replication_min_speed_time_seconds "Direct link to replication_min_speed_time_seconds") * Default: 300 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time duration allowed for a replication thread to be under the minimum speed. Replication will fail if the time when the actual speed is lower than `replication_min_speed_limit_kbps` exceeds this value. * Introduced in: v3.3.5 ##### replication\_threads[​](#replication_threads "Direct link to replication_threads") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of threads used for replication. `0` indicates setting the thread number to four times the BE CPU core count. * Introduced in: v3.3.5 ##### size\_tiered\_level\_multiple[​](#size_tiered_level_multiple "Direct link to size_tiered_level_multiple") * Default: 5 * Type: Int * Unit: - * Is mutable: Yes * Description: The multiple of data size between two contiguous levels in the Size-tiered Compaction policy. * Introduced in: - ##### size\_tiered\_level\_multiple\_dupkey[​](#size_tiered_level_multiple_dupkey "Direct link to size_tiered_level_multiple_dupkey") * Default: 10 * Type: Int * Unit: - * Is mutable: Yes * Description: In the Size-tiered Compaction policy, the multiple of the data amount difference between two adjacent levels for Duplicate Key tables. * Introduced in: - ##### size\_tiered\_level\_num[​](#size_tiered_level_num "Direct link to size_tiered_level_num") * Default: 7 * Type: Int * Unit: - * Is mutable: Yes * Description: The number of levels for the Size-tiered Compaction policy. At most one rowset is reserved for each level. Therefore, under a stable condition, there are, at most, as many rowsets as the level number specified in this configuration item. * Introduced in: - ##### size\_tiered\_max\_compaction\_level[​](#size_tiered_max_compaction_level "Direct link to size_tiered_max_compaction_level") * Default: 3 * Type: Int * Unit: Levels * Is mutable: Yes * Description: Limits how many size-tiered levels may be merged into a single primary-key real-time compaction task. During the PK size-tiered compaction selection, StarRocks builds ordered "levels" of rowsets by size and will add successive levels into the chosen compaction input until this limit is reached (the code uses compaction\_level `<=` size\_tiered\_max\_compaction\_level). The value is inclusive and counts the number of distinct size tiers merged (the top level is counted as 1). Effective only when the PK size-tiered compaction strategy is enabled; raising it lets a compaction task include more levels (larger, more I/O- and CPU-intensive merges, potential higher write amplification), while lowering it restricts merges and reduces task size and resource usage. * Introduced in: v4.0.0 ##### size\_tiered\_min\_level\_size[​](#size_tiered_min_level_size "Direct link to size_tiered_min_level_size") * Default: 131072 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: The data size of the minimum level in the Size-tiered Compaction policy. Rowsets smaller than this value immediately trigger the data compaction. * Introduced in: - ##### small\_dictionary\_page\_size[​](#small_dictionary_page_size "Direct link to small_dictionary_page_size") * Default: 4096 * Type: Int * Unit: Bytes * Is mutable: No * Description: Threshold (in bytes) used by BinaryPlainPageDecoder to decide whether to eagerly parse a dictionary (binary/plain) page. If a page's encoded size is < `small_dictionary_page_size`, the decoder pre-parses all string entries into an in-memory vector (`_parsed_datas`) to accelerate random access and batch reads. Raising this value causes more pages to be pre-parsed (which can reduce per-access decoding overhead and may increase effective compression for larger dictionaries) but increases memory usage and CPU spent parsing; excessively large values can degrade overall performance. Tune only after measuring memory and access-latency trade-offs. * Introduced in: v3.4.1, v3.5.0 ##### snapshot\_expire\_time\_sec[​](#snapshot_expire_time_sec "Direct link to snapshot_expire_time_sec") * Default: 172800 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The expiration time of snapshot files. * Introduced in: - ##### stale\_memtable\_flush\_time\_sec[​](#stale_memtable_flush_time_sec "Direct link to stale_memtable_flush_time_sec") * Default: 0 * Type: long * Unit: Seconds * Is mutable: Yes * Description: When a sender job's memory usage is high, memtables that have not been updated for longer than `stale_memtable_flush_time_sec` seconds will be flushed to reduce memory pressure. This behavior is only considered when memory limits are approaching (`limit_exceeded_by_ratio(70)` or higher). In `LocalTabletsChannel`, an additional path at very high memory usage (`limit_exceeded_by_ratio(95)`) may flush memtables whose size exceeds `write_buffer_size / 4`. A value of `0` disables this age-based stale-memtable flushing (immutable-partition memtables still flush immediately when idle or on high memory). * Introduced in: v3.2.0 ##### storage\_flood\_stage\_left\_capacity\_bytes[​](#storage_flood_stage_left_capacity_bytes "Direct link to storage_flood_stage_left_capacity_bytes") * Default: 107374182400 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: Hard limit of the remaining storage space in all BE directories. If the remaining storage space of the BE storage directory is less than this value and the storage usage (in percentage) exceeds `storage_flood_stage_usage_percent`, Load and Restore jobs are rejected. You need to set this item together with the FE configuration item `storage_usage_hard_limit_reserve_bytes` to allow the configurations to take effect. * Introduced in: - ##### storage\_flood\_stage\_usage\_percent[​](#storage_flood_stage_usage_percent "Direct link to storage_flood_stage_usage_percent") * Default: 95 * Type: Int * Unit: - * Is mutable: Yes * Description: Hard limit of the storage usage percentage in all BE directories. If the storage usage (in percentage) of the BE storage directory exceeds this value and the remaining storage space is less than `storage_flood_stage_left_capacity_bytes`, Load and Restore jobs are rejected. You need to set this item together with the FE configuration item `storage_usage_hard_limit_percent` to allow the configurations to take effect. * Introduced in: - ##### storage\_high\_usage\_disk\_protect\_ratio[​](#storage_high_usage_disk_protect_ratio "Direct link to storage_high_usage_disk_protect_ratio") * Default: 0.1 * Type: double * Unit: - * Is mutable: Yes * Description: When selecting a storage root for tablet creation, StorageEngine sorts candidate disks by `disk_usage(0)` and computes the average usage. Any disk whose usage is greater than (average usage + `storage_high_usage_disk_protect_ratio`) is excluded from the preferential selection pool (it will not participate in the randomized, preferrential shuffle and thus is deferred from being chosen initially). Set to 0 to disable this protection. Values are fractional (typical range 0.0–1.0); larger values make the scheduler more tolerant of higher-than-average disks. * Introduced in: v3.2.0 ##### storage\_medium\_migrate\_count[​](#storage_medium_migrate_count "Direct link to storage_medium_migrate_count") * Default: 3 * Type: Int * Unit: - * Is mutable: No * Description: The number of threads used for storage medium migration (from SATA to SSD). * Introduced in: - ##### storage\_root\_path[​](#storage_root_path "Direct link to storage_root_path") * Default: `${STARROCKS_HOME}/storage` * Type: String * Unit: - * Is mutable: No * Description: The directory and medium of the storage volume. Example: `/data1,medium:hdd;/data2,medium:ssd`. * Multiple volumes are separated by semicolons (`;`). * If the storage medium is SSD, add `,medium:ssd` at the end of the directory. * If the storage medium is HDD, add `,medium:hdd` at the end of the directory. * Introduced in: - ##### sync\_tablet\_meta[​](#sync_tablet_meta "Direct link to sync_tablet_meta") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: A boolean value to control whether to enable the synchronization of the tablet metadata. `true` indicates enabling synchronization, and `false` indicates disabling it. * Introduced in: - ##### tablet\_map\_shard\_size[​](#tablet_map_shard_size "Direct link to tablet_map_shard_size") * Default: 1024 * Type: Int * Unit: - * Is mutable: No * Description: The tablet map shard size. The value must be a power of two. * Introduced in: - ##### tablet\_max\_pending\_versions[​](#tablet_max_pending_versions "Direct link to tablet_max_pending_versions") * Default: 1000 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of pending versions that are tolerable on a Primary Key tablet. Pending versions refer to versions that are committed but not applied yet. * Introduced in: - ##### tablet\_max\_versions[​](#tablet_max_versions "Direct link to tablet_max_versions") * Default: 1000 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of versions allowed on a tablet. If the number of versions exceeds this value, new write requests will fail. * Introduced in: - ##### tablet\_meta\_checkpoint\_min\_interval\_secs[​](#tablet_meta_checkpoint_min_interval_secs "Direct link to tablet_meta_checkpoint_min_interval_secs") * Default: 600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval of thread polling for a TabletMeta Checkpoint. * Introduced in: - ##### tablet\_meta\_checkpoint\_min\_new\_rowsets\_num[​](#tablet_meta_checkpoint_min_new_rowsets_num "Direct link to tablet_meta_checkpoint_min_new_rowsets_num") * Default: 10 * Type: Int * Unit: - * Is mutable: Yes * Description: The minimum number of rowsets to create since the last TabletMeta Checkpoint. * Introduced in: - ##### tablet\_rowset\_stale\_sweep\_time\_sec[​](#tablet_rowset_stale_sweep_time_sec "Direct link to tablet_rowset_stale_sweep_time_sec") * Default: 1800 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval at which to sweep the stale rowsets in tablets. * Introduced in: - ##### tablet\_stat\_cache\_update\_interval\_second[​](#tablet_stat_cache_update_interval_second "Direct link to tablet_stat_cache_update_interval_second") * Default: 300 * Type: Int * Unit: Seconds * Is mutable: 是 * Description: The time interval at which Tablet Stat Cache updates. * Introduced in: - ##### lake\_enable\_accurate\_pk\_row\_count[​](#lake_enable_accurate_pk_row_count "Direct link to lake_enable_accurate_pk_row_count") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to use accurate row counts for lake primary-key tablets. When enabled, StarRocks reads each rowset's delete vector from object storage and subtracts deleted rows, producing more accurate stats but potentially increasing `get_tablet_stats` RPC overhead. When disabled, StarRocks uses the approximate `num_dels` value in rowset metadata to avoid remote I/O, which may slightly overcount rows that were deleted but not yet compacted. * Introduced in: - ##### lake\_tablet\_stat\_slow\_log\_ms[​](#lake_tablet_stat_slow_log_ms "Direct link to lake_tablet_stat_slow_log_ms") * Default: 300000 * Type: Int64 * Unit: Milliseconds * Is mutable: Yes * Description: Threshold (in milliseconds) for logging slow tablet-stat collection tasks. If a single tablet stat task exceeds this value, StarRocks emits a warning log with diagnostics such as `tablet_id`, version, rowset count, accurate mode, and elapsed time. * Introduced in: - ##### lake\_metadata\_fetch\_thread\_count[​](#lake_metadata_fetch_thread_count "Direct link to lake_metadata_fetch_thread_count") * Default: 3 * Type: Int * Unit: - * Is mutable: Yes * Description: The count of threads for shared-data table tablet metadata fetch operations (e.g., `get_tablet_stats`, `get_tablet_metadatas`). * Introduced in: v3.5.16, v4.0.9 ##### tablet\_writer\_open\_rpc\_timeout\_sec[​](#tablet_writer_open_rpc_timeout_sec "Direct link to tablet_writer_open_rpc_timeout_sec") * Default: 300 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Timeout (in seconds) for the RPC that opens a tablet writer on a remote BE. The value is converted to milliseconds and applied to both the request timeout and the brpc control timeout when issuing the open call. The runtime uses the effective timeout as the minimum of `tablet_writer_open_rpc_timeout_sec` and half of the overall load timeout (i.e., min(`tablet_writer_open_rpc_timeout_sec`, `load_timeout_sec` / 2)). Set this to balance timely failure detection (too small may cause premature open failures) and giving BEs enough time to initialize writers (too large delays error handling). * Introduced in: v3.2.0 ##### transaction\_apply\_worker\_count[​](#transaction_apply_worker_count "Direct link to transaction_apply_worker_count") * Default: 0 * Type: Int * Unit: Threads * Is mutable: Yes * Description: Controls the maximum number of worker threads used by the UpdateManager's "update\_apply" thread pool — the pool that applies rowsets for transactions (notably for primary-key tables). A value `>0` sets a fixed maximum thread count; 0 (the default) makes the pool size equal to the number of CPU cores. The configured value is applied at startup (UpdateManager::init) and can be changed at runtime via the update-config HTTP action, which updates the pool's max threads. Tune this to increase apply concurrency (throughput) or limit CPU/memory contention; min threads and idle timeout are governed by transaction\_apply\_thread\_pool\_num\_min and transaction\_apply\_worker\_idle\_time\_ms respectively. * Introduced in: v3.2.0 ##### transaction\_apply\_worker\_idle\_time\_ms[​](#transaction_apply_worker_idle_time_ms "Direct link to transaction_apply_worker_idle_time_ms") * Default: 500 * Type: int * Unit: Milliseconds * Is mutable: No * Description: Sets the idle timeout (in milliseconds) for the UpdateManager's "update\_apply" thread pool used to apply transactions/updates. The value is passed to ThreadPoolBuilder::set\_idle\_timeout via MonoDelta::FromMilliseconds, so worker threads that remain idle longer than this timeout may be terminated (subject to the pool's configured minimum thread count and max threads). Lower values free resources faster but increase thread creation/teardown overhead under bursty load; higher values keep workers warm for short bursts at the cost of higher baseline resource usage. * Introduced in: v3.2.11 ##### trash\_file\_expire\_time\_sec[​](#trash_file_expire_time_sec "Direct link to trash_file_expire_time_sec") * Default: 86400 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval at which to clean trash files. The default value has been changed from 259,200 to 86,400 since v2.5.17, v3.0.9, and v3.1.6. * Introduced in: - ##### unused\_rowset\_monitor\_interval[​](#unused_rowset_monitor_interval "Direct link to unused_rowset_monitor_interval") * Default: 30 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval at which to clean the expired rowsets. * Introduced in: - ##### update\_cache\_expire\_sec[​](#update_cache_expire_sec "Direct link to update_cache_expire_sec") * Default: 360 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The expiration time of Update Cache. * Introduced in: - ##### update\_compaction\_check\_interval\_seconds[​](#update_compaction_check_interval_seconds "Direct link to update_compaction_check_interval_seconds") * Default: 10 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval at which to check compaction for Primary Key tables. * Introduced in: - ##### update\_compaction\_delvec\_file\_io\_amp\_ratio[​](#update_compaction_delvec_file_io_amp_ratio "Direct link to update_compaction_delvec_file_io_amp_ratio") * Default: 2 * Type: Int * Unit: - * Is mutable: Yes * Description: Used to control the priority of compaction for rowsets that contain Delvec files in Primary Key tables. The larger the value, the higher the priority. * Introduced in: - ##### update\_compaction\_num\_threads\_per\_disk[​](#update_compaction_num_threads_per_disk "Direct link to update_compaction_num_threads_per_disk") * Default: 1 * Type: Int * Unit: - * Is mutable: Yes * Description: The number of Compaction threads per disk for Primary Key tables. * Introduced in: - ##### update\_compaction\_per\_tablet\_min\_interval\_seconds[​](#update_compaction_per_tablet_min_interval_seconds "Direct link to update_compaction_per_tablet_min_interval_seconds") * Default: 120 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The minimum time interval at which compaction is triggered for each tablet in a Primary Key table. * Introduced in: - ##### update\_compaction\_ratio\_threshold[​](#update_compaction_ratio_threshold "Direct link to update_compaction_ratio_threshold") * Default: 0.5 * Type: Double * Unit: - * Is mutable: Yes * Description: The maximum proportion of data that a compaction can merge for a Primary Key table in a shared-data cluster. It is recommended to shrink this value if a single tablet becomes excessively large. * Introduced in: v3.1.5 ##### update\_compaction\_result\_bytes[​](#update_compaction_result_bytes "Direct link to update_compaction_result_bytes") * Default: 1073741824 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: The maximum result size of a single compaction for Primary Key tables. * Introduced in: - ##### update\_compaction\_size\_threshold[​](#update_compaction_size_threshold "Direct link to update_compaction_size_threshold") * Default: 268435456 * Type: Int * Unit: - * Is mutable: Yes * Description: The Compaction Score of Primary Key tables is calculated based on the file size, which is different from other table types. This parameter can be used to make the Compaction Score of Primary Key tables similar to that of other table types, making it easier for users to understand. * Introduced in: - ##### upload\_worker\_count[​](#upload_worker_count "Direct link to upload_worker_count") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of threads for the upload tasks of backup jobs on a BE node. `0` indicates setting the value to the number of CPU cores on the machine where the BE resides. * Introduced in: - ##### vertical\_compaction\_max\_columns\_per\_group[​](#vertical_compaction_max_columns_per_group "Direct link to vertical_compaction_max_columns_per_group") * Default: 5 * Type: Int * Unit: - * Is mutable: No * Description: The maximum number of columns per group of Vertical Compactions. * Introduced in: - --- ### Compaction for Shared-data Clusters This topic describes how to manage compaction in StarRocks shared-data clusters. #### Overview[​](#overview "Direct link to Overview") Each data loading operation in StarRocks generates a new version of data files. Compaction merges data files from different versions into larger files, reducing the number of small files and improving query efficiency. #### Compaction Score[​](#compaction-score "Direct link to Compaction Score") ##### Overview[​](#overview-1 "Direct link to Overview") The *Compaction Score* reflects the merging status of data files in a partition. A higher score indicates lower merging progress, meaning the partition has more unmerged data file versions. FE maintains Compaction Score information for each partition, including the Max Compaction Score (the highest score among all tablets in the partition). If a partition's Max Compaction Score is below the FE parameter `lake_compaction_score_selector_min_score` (default: 10), compaction for that partition is considered complete. A Max Compaction Score exceeding 100 indicates an unhealthy compaction state. When the score exceeds the FE parameter `lake_ingest_slowdown_threshold` (default: 100), the system slows down data loading transaction commits for the partition. If it surpasses `lake_compaction_score_upper_bound` (default: 2000), the system rejects import transactions for the partition. ##### Calculation Rules[​](#calculation-rules "Direct link to Calculation Rules") Typically, each data file contributes 1 to the Compaction Score. For example, if a partition has one tablet and 10 data files generated from the first loading operation, the partition’s Max Compaction Score is 10. All data files generated by a transaction within a tablet are grouped as a Rowset. During score calculation, a tablet’s Rowsets are grouped by size, and the group with the highest number of files determines the tablet’s Compaction Score. For example, a tablet undergoes 7 loading operations, generating Rowsets with sizes: 100 MB, 100 MB, 100 MB, 10 MB, 10 MB, 10 MB, and 10 MB. During calculation, the system will make three 100 MB Rowsets into one group and four 10 MB Rowsets into another. The Compaction Score is calculated based on the group with more files. In this case, the second group has bigger compaction score. The compaction prioritizes the higher-scoring group, so after the first compaction, the Rowset diatribution would be: 100 MB, 100 MB, 100 MB, and 40 MB. #### Compaction Workflow[​](#compaction-workflow "Direct link to Compaction Workflow") For shared-data clusters, StarRocks introduces a new FE-controlled compaction mechanism: 1. Score Calculation: The Leader FE node calculates and stores Compaction Scores for partitions based on transaction publish results. 2. Candidate Selection: FE selects partitions with the highest Max Compaction Scores as compaction candidates. 3. Task Generation: FE initiates compaction transactions for selected partitions, generates tablet-level subtasks, and dispatches them to Compute Nodes (CNs) until reaching the limit set by the FE parameter `lake_compaction_max_tasks`. 4. Subtask Execution: CNs execute compaction subtasks in the background. The number of concurrent subtasks per CN is controlled by the CN parameter `compact_threads`. 5. Result Collection: FE aggregates subtask results and commits the compaction transaction. 6. Publish: FE publishes the successfully committed compaction transaction. #### Manage compaction[​](#manage-compaction "Direct link to Manage compaction") ##### View compaction scores[​](#view-compaction-scores "Direct link to View compaction scores") * You can view the compaction scores of partitions in a specific table by using the SHOW PROC statement. Typically, you only need to focus on the `MaxCS` field. If `MaxCS` is below 10, compaction is considered complete. If `MaxCS` is above 100, the Compaction Score is relatively high. If `MaxCS` exceeds 500, the Compaction Score is very high and manual intervention may be required. ```plain SHOW PARTITIONS FROM SHOW PROC '/dbs///partitions' ``` Example: ```plain mysql> SHOW PROC '/dbs/load_benchmark/store_sales/partitions'; +-------------+---------------+----------------+----------------+-------------+--------+--------------+-------+------------------------------+---------+----------+-----------+----------+------------+-------+-------+-------+ | PartitionId | PartitionName | CompactVersion | VisibleVersion | NextVersion | State | PartitionKey | Range | DistributionKey | Buckets | DataSize | RowCount | CacheTTL | AsyncWrite | AvgCS | P50CS | MaxCS | +-------------+---------------+----------------+----------------+-------------+--------+--------------+-------+------------------------------+---------+----------+-----------+----------+------------+-------+-------+-------+ | 38028 | store_sales | 913 | 921 | 923 | NORMAL | | | ss_item_sk, ss_ticket_number | 64 | 15.6GB | 273857126 | 2592000 | false | 10.00 | 10.00 | 10.00 | +-------------+---------------+----------------+----------------+-------------+--------+--------------+-------+------------------------------+---------+----------+-----------+----------+------------+-------+-------+-------+ 1 row in set (0.20 sec) ``` * You can also view the partition compaction scores by querying the system-defined view `information_schema.partitions_meta`. Example: ```plain mysql> SELECT * FROM information_schema.partitions_meta ORDER BY Max_CS LIMIT 10; +--------------+----------------------------+----------------------------+--------------+-----------------+-----------------+----------------------+--------------+---------------+-----------------+-----------------------------------------+---------+-----------------+----------------+---------------------+-----------------------------+--------------+---------+-----------+------------+------------------+----------+--------+--------+-------------------------------------------------------------------+ | DB_NAME | TABLE_NAME | PARTITION_NAME | PARTITION_ID | COMPACT_VERSION | VISIBLE_VERSION | VISIBLE_VERSION_TIME | NEXT_VERSION | PARTITION_KEY | PARTITION_VALUE | DISTRIBUTION_KEY | BUCKETS | REPLICATION_NUM | STORAGE_MEDIUM | COOLDOWN_TIME | LAST_CONSISTENCY_CHECK_TIME | IS_IN_MEMORY | IS_TEMP | DATA_SIZE | ROW_COUNT | ENABLE_DATACACHE | AVG_CS | P50_CS | MAX_CS | STORAGE_PATH | +--------------+----------------------------+----------------------------+--------------+-----------------+-----------------+----------------------+--------------+---------------+-----------------+-----------------------------------------+---------+-----------------+----------------+---------------------+-----------------------------+--------------+---------+-----------+------------+------------------+----------+--------+--------+-------------------------------------------------------------------+ | tpcds_1t | call_center | call_center | 11905 | 0 | 2 | 2024-03-17 08:30:47 | 3 | | | cc_call_center_sk | 1 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 12.3KB | 42 | 0 | 0 | 0 | 0 | s3://XXX/536a3c77-52c3-485a-8217-781734a970b1/db10328/11906/11905 | | tpcds_1t | web_returns | web_returns | 12030 | 3 | 3 | 2024-03-17 08:40:48 | 4 | | | wr_item_sk, wr_order_number | 16 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 3.5GB | 71997522 | 0 | 0 | 0 | 0 | s3://XXX/536a3c77-52c3-485a-8217-781734a970b1/db10328/12031/12030 | | tpcds_1t | warehouse | warehouse | 11847 | 0 | 2 | 2024-03-17 08:30:47 | 3 | | | w_warehouse_sk | 1 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 4.2KB | 20 | 0 | 0 | 0 | 0 | s3://XXX/536a3c77-52c3-485a-8217-781734a970b1/db10328/11848/11847 | | tpcds_1t | ship_mode | ship_mode | 11851 | 0 | 2 | 2024-03-17 08:30:47 | 3 | | | sm_ship_mode_sk | 1 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 1.7KB | 20 | 0 | 0 | 0 | 0 | s3://XXX/536a3c77-52c3-485a-8217-781734a970b1/db10328/11852/11851 | | tpcds_1t | customer_address | customer_address | 11790 | 0 | 2 | 2024-03-17 08:32:19 | 3 | | | ca_address_sk | 16 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 120.9MB | 6000000 | 0 | 0 | 0 | 0 | s3://XXX/536a3c77-52c3-485a-8217-781734a970b1/db10328/11791/11790 | | tpcds_1t | time_dim | time_dim | 11855 | 0 | 2 | 2024-03-17 08:30:48 | 3 | | | t_time_sk | 16 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 864.7KB | 86400 | 0 | 0 | 0 | 0 | s3://XXX/536a3c77-52c3-485a-8217-781734a970b1/db10328/11856/11855 | | tpcds_1t | web_sales | web_sales | 12049 | 3 | 3 | 2024-03-17 10:14:20 | 4 | | | ws_item_sk, ws_order_number | 128 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 47.7GB | 720000376 | 0 | 0 | 0 | 0 | s3://XXX/536a3c77-52c3-485a-8217-781734a970b1/db10328/12050/12049 | | tpcds_1t | store | store | 11901 | 0 | 2 | 2024-03-17 08:30:47 | 3 | | | s_store_sk | 1 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 95.6KB | 1002 | 0 | 0 | 0 | 0 | s3://XXX/536a3c77-52c3-485a-8217-781734a970b1/db10328/11902/11901 | | tpcds_1t | web_site | web_site | 11928 | 0 | 2 | 2024-03-17 08:30:47 | 3 | | | web_site_sk | 1 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 13.4KB | 54 | 0 | 0 | 0 | 0 | s3://XXX/536a3c77-52c3-485a-8217-781734a970b1/db10328/11929/11928 | | tpcds_1t | household_demographics | household_demographics | 11932 | 0 | 2 | 2024-03-17 08:30:47 | 3 | | | hd_demo_sk | 1 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 2.1KB | 7200 | 0 | 0 | 0 | 0 | s3://XXX/536a3c77-52c3-485a-8217-781734a970b1/db10328/11933/11932 | +--------------+----------------------------+----------------------------+--------------+-----------------+-----------------+----------------------+--------------+---------------+-----------------+-----------------------------------------+---------+-----------------+----------------+---------------------+-----------------------------+--------------+---------+-----------+------------+------------------+----------+--------+--------+-------------------------------------------------------------------+ ``` ##### View compaction tasks[​](#view-compaction-tasks "Direct link to View compaction tasks") As new data is loading to the system, FE constantly schedules compaction tasks to be executed on different CN nodes. You can first view the general status of compaction tasks on FE, and then view the execution details of each tasks on CN. ###### View general status of compaction tasks[​](#view-general-status-of-compaction-tasks "Direct link to View general status of compaction tasks") You can view the general status of compaction tasks using the SHOW PROC statement. ```sql SHOW PROC '/compactions'; ``` Example: ```plain mysql> SHOW PROC '/compactions'; +---------------------+-------+---------------------+---------------------+---------------------+-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Partition | TxnID | StartTime | CommitTime | FinishTime | Error | Profile | +---------------------+-------+---------------------+---------------------+---------------------+-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | ssb.lineorder.10081 | 15 | 2026-01-10 03:29:07 | 2026-01-10 03:29:11 | 2026-01-10 03:29:12 | NULL | {"sub_task_count":12,"read_local_sec":0,"read_local_mb":218,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":120,"write_segment_count":12,"write_segment_mb":219,"write_remote_sec":4,"in_queue_sec":18,"score_before":{"avg":10.0,"p50":10.0,"max":10.0},"score_after":{"avg":8.0,"p50":8.0,"max":8.0},"partial_success":false} | | ssb.lineorder.10068 | 16 | 2026-01-10 03:29:07 | 2026-01-10 03:29:13 | 2026-01-10 03:29:14 | NULL | {"sub_task_count":12,"read_local_sec":0,"read_local_mb":218,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":120,"write_segment_count":12,"write_segment_mb":218,"write_remote_sec":4,"in_queue_sec":38,"score_before":{"avg":10.0,"p50":10.0,"max":10.0},"score_after":{"avg":8.0,"p50":8.0,"max":8.0},"partial_success":false} | | ssb.lineorder.10055 | 20 | 2026-01-10 03:29:11 | 2026-01-10 03:29:15 | 2026-01-10 03:29:17 | NULL | {"sub_task_count":12,"read_local_sec":0,"read_local_mb":218,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":120,"write_segment_count":12,"write_segment_mb":218,"write_remote_sec":4,"in_queue_sec":23,"score_before":{"avg":10.0,"p50":10.0,"max":10.0},"score_after":{"avg":8.0,"p50":8.0,"max":8.0},"partial_success":false} | +---------------------+-------+---------------------+---------------------+---------------------+-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` The following fields are returned: * `Partition`: The partition to which the compaction task belongs. * `TxnID`: The transaction ID assigned to the compaction task. * `StartTime`: The time when the compaction task starts. `NULL` indicates that the task has not yet been initiated. * `CommitTime`: The time when the compaction task commits the data. `NULL` indicates that the data has not yet been committed. * `FinishTime`: The time when the compaction task publishes the data. `NULL` indicates that the data has not yet been published. * `Error`: The error message (if any) of the compaction task. * `Profile`: (supported from v3.2.12 and v3.3.4) The Profile of the compaction task after finished. * `sub_task_count`: The number of sub-tasks (equivalent to tablets) in the partition. * `read_local_sec`: The total time consumption of all sub-tasks on reading data from the local cache. Unit: Seconds. * `read_local_mb`: The total size of data read from the local cache by all sub-tasks. Unit: MB. * `read_remote_sec`: The total time consumption of all sub-tasks on reading data from the remote storage. Unit: Seconds. * `read_remote_mb`: The total size of data read from the remote storage by all sub-tasks. Unit: MB. * `read_segment_count`: The total number of files read by all sub-tasks. * `write_segment_count`: The total number of new files generated by all sub-tasks. * `write_segment_mb`: The total size of new files generated by all sub-tasks. Unit: MB. * `write_remote_sec`: The total time consumption of all sub-tasks on writing data to the remote storage. Unit: Seconds. * `in_queue_sec`: The total time of all sub-tasks staying in the queue. Unit: Seconds. * `score_before`: The compaction score of the partition before the compaction. Includes `avg`, `p50`, and `max` fields. * `score_after`: The compaction score of the partition after the compaction. Includes `avg`, `p50`, and `max` fields. * `partial_success`: Whether the compaction job is partially successful (some tablets failed). ###### View execution details of compaction tasks[​](#view-execution-details-of-compaction-tasks "Direct link to View execution details of compaction tasks") Each compaction task is divided into multiple sub-tasks, each of which corresponds to a tablet. You can view the execution details of each sub-task by querying the system-defined view `information_schema.be_cloud_native_compactions`. Example: ```plain mysql> SELECT * FROM information_schema.be_cloud_native_compactions; +-------+--------+-----------+---------+---------+------+---------------------+-------------+----------+--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | BE_ID | TXN_ID | TABLET_ID | VERSION | SKIPPED | RUNS | START_TIME | FINISH_TIME | PROGRESS | STATUS | PROFILE | +-------+--------+-----------+---------+---------+------+---------------------+-------------+----------+--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 10001 | 51047 | 43034 | 12 | 0 | 1 | 2024-09-24 19:15:15 | NULL | 82 | | {"read_local_sec":0,"read_local_mb":31,"read_remote_sec":0,"read_remote_mb":0,"read_remote_count":0,"read_local_count":1900,"segment_init_sec":0,"column_iterator_init_sec":0,"in_queue_sec":0} | | 10001 | 51048 | 43032 | 12 | 0 | 1 | 2024-09-24 19:15:15 | NULL | 82 | | {"read_local_sec":0,"read_local_mb":32,"read_remote_sec":0,"read_remote_mb":0,"read_remote_count":0,"read_local_count":1900,"segment_init_sec":0,"column_iterator_init_sec":0,"in_queue_sec":0} | | 10001 | 51049 | 43033 | 12 | 0 | 1 | 2024-09-24 19:15:15 | NULL | 82 | | {"read_local_sec":0,"read_local_mb":31,"read_remote_sec":0,"read_remote_mb":0,"read_remote_count":0,"read_local_count":1900,"segment_init_sec":0,"column_iterator_init_sec":0,"in_queue_sec":0} | | 10001 | 51051 | 43038 | 9 | 0 | 1 | 2024-09-24 19:15:15 | NULL | 84 | | {"read_local_sec":0,"read_local_mb":31,"read_remote_sec":0,"read_remote_mb":0,"read_remote_count":0,"read_local_count":1900,"segment_init_sec":0,"column_iterator_init_sec":0,"in_queue_sec":0} | | 10001 | 51052 | 43036 | 12 | 0 | 0 | NULL | NULL | 0 | | | | 10001 | 51053 | 43035 | 12 | 0 | 1 | 2024-09-24 19:15:16 | NULL | 2 | | {"read_local_sec":0,"read_local_mb":1,"read_remote_sec":0,"read_remote_mb":0,"read_remote_count":0,"read_local_count":100,"segment_init_sec":0,"column_iterator_init_sec":0,"in_queue_sec":0} | +-------+--------+-----------+---------+---------+------+---------------------+-------------+----------+--------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` The following fields are returned: * `BE_ID`: The ID of the CN. * `TXN_ID`: The ID of transaction to which the sub-task belongs. * `TABLET_ID`: The ID of tablet to which the sub-task belongs. * `VERSION`: The version of the tablet. * `RUNS`: The number of times the sub-task has been executed. * `START_TIME`: The time when the sub-task starts. * `FINISH_TIME`: The time when the sub-task finishes. * `PROGRESS`: The compaction progress of the tablet in percentage. * `STATUS`: The status of the sub-task. Error messages will be returned in this field if there is an error. * `PROFILE`: (supported from v3.2.12 and v3.3.4) The runtime profile of the sub-task. * `read_local_sec`: The time consumption of the sub-task on reading data from the local cache. Unit: Seconds. * `read_local_mb`: The size of data read from the local cache by the sub-task. Unit: MB. * `read_remote_sec`: The time consumption of the sub-task on reading data from the remote storage. Unit: Seconds. * `read_remote_mb`: The size of data read from the remote storage by the sub-task. Unit: MB. * `read_local_count`: The number of times the sub-task reads data from the local cache. * `read_remote_count`: The number of times the sub-task reads data from the remote storage. * `in_queue_sec`: The time of the sub-task staying in queue. Unit: Seconds. ##### Configure compaction tasks[​](#configure-compaction-tasks "Direct link to Configure compaction tasks") You can configure compaction tasks using these FE and CN (BE) parameters. ###### FE parameter[​](#fe-parameter "Direct link to FE parameter") You can configure the following FE parameter dynamically. ```sql ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "-1"); ``` ###### lake\_compaction\_max\_tasks[​](#lake_compaction_max_tasks "Direct link to lake_compaction_max_tasks") * Default: -1 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of concurrent Compaction tasks allowed in a shared-data cluster. Setting this item to `-1` indicates calculating the concurrent task number in an adaptive manner, that is, the number of surviving CN nodes multiplied by 16. Setting this value to `0` will disable compaction. * Introduced in: v3.1.0 ```sql ADMIN SET FRONTEND CONFIG ("lake_compaction_disable_tables" = "11111;22222"); ``` ###### lake\_compaction\_disable\_tables[​](#lake_compaction_disable_tables "Direct link to lake_compaction_disable_tables") * Default："" * Type：String * Unit：- * Is mutable：Yes * Description：Disable compaction for certain tables. This will not affect compaction that has started. The value of this item is table ID. Multiple values are separated by ';'. * Introduced in：v3.2.7 ###### CN parameters[​](#cn-parameters "Direct link to CN parameters") You can configure the following CN parameter dynamically. ```sql UPDATE information_schema.be_configs SET VALUE = 8 WHERE name = "compact_threads"; ``` ###### compact\_threads[​](#compact_threads "Direct link to compact_threads") * Default: 4 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of threads used for concurrent compaction tasks. This configuration is changed to dynamic from v3.1.7 and v3.2.2 onwards. * Introduced in: v3.0.0 > **NOTE** > > In production, it is recommended to set `compact_threads` to 25% of the BE/CN CPU core count. ###### max\_cumulative\_compaction\_num\_singleton\_deltas[​](#max_cumulative_compaction_num_singleton_deltas "Direct link to max_cumulative_compaction_num_singleton_deltas") * Default: 500 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of segments that can be merged in a single Cumulative Compaction. You can reduce this value if OOM occurs during compaction. * Introduced in: - > **NOTE** > > In production, it is recommended to set `max_cumulative_compaction_num_singleton_deltas` to `100` to accelerate the compaction tasks and reduce their recource consumption. ###### lake\_pk\_compaction\_max\_input\_rowsets[​](#lake_pk_compaction_max_input_rowsets "Direct link to lake_pk_compaction_max_input_rowsets") * Default: 500 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of input rowsets allowed in a Primary Key table compaction task in a shared-data cluster. The default value of this parameter is changed from `5` to `1000` since v3.2.4 and v3.1.10, and to `500` since since v3.3.1 and v3.2.9. After the Sized-tiered Compaction policy is enabled for Primary Key tables (by setting `enable_pk_size_tiered_compaction_strategy` to `true`), StarRocks does not need to limit the number of rowsets for each compaction to reduce write amplification. Therefore, the default value of this parameter is increased. * Introduced in: v3.1.8, v3.2.3 > **NOTE** > > In production, it is recommended to set `max_cumulative_compaction_num_singleton_deltas` to `100` to accelerate the Compaction tasks and reduce their resource consumption. ##### Manually trigger compaction tasks[​](#manually-trigger-compaction-tasks "Direct link to Manually trigger compaction tasks") ```sql -- Trigger compaction for the whole table. ALTER TABLE COMPACT; -- Trigger compaction for a specific partition. ALTER TABLE COMPACT ; -- Trigger compaction for multiple partitions. ALTER TABLE COMPACT (, , ...); ``` ##### Cancel compaction tasks[​](#cancel-compaction-tasks "Direct link to Cancel compaction tasks") You can manually cancel a compaction task using the transaction ID of the task. ```sql CANCEL COMPACTION WHERE TXN_ID = ; ``` > **NOTE** > > * The CANCEL COMPACTION statement must be submitted from the Leader FE node. > * The CANCEL COMPACTION statement only applies to transactions that have not committed, that is, `CommitTime` is NULL in the return of `SHOW PROC '/compactions'`. > * CANCEL COMPACTION is an asynchronous process. You can check if the task is cancelled by executing `SHOW PROC '/compactions'`. #### Best practices[​](#best-practices "Direct link to Best practices") Since Compaction is crucial for query performance, it is recommended to regularly monitor the data merging status of tables and partitions. Here are some best practices and guidelines: * Try to increase the time interval between loading (avoid scenarios with intervals less than 10 seconds) and increase the batch size per load (avoid batch sizes smaller than 100 rows of data). * Adjust the number of parallel compaction worker threads on CN to accelerate task execution. It is recommended to set `compact_threads` to 25% of the BE/CN CPU core count in a production environment. * Monitor the Compaction task status using `show proc '/compactions'` and `select * from information_schema.be_cloud_native_compactions;`. * Monitor the Compaction Score, and configure alerts based on it. StarRocks' built-in Grafana monitoring template includes this metric. * Pay attention to the resource consumption during compaction, especially memory usage. The Grafana monitoring template also includes this metric. #### Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") ##### Slow queries[​](#slow-queries "Direct link to Slow queries") To identify slow queries caused by untimely Compaction, you can check, in the SQL Profile, the value of `SegmentsReadCount` divided by `TabletCount` within a single Fragment. If it is an large value, such as tens or more, untimely Compaction may be the cause of the slow query. ##### High Max Compaction Score in the cluster[​](#high-max-compaction-score-in-the-cluster "Direct link to High Max Compaction Score in the cluster") 1. Check whether the Compaction-related parameters are within reasonable ranges using `ADMIN SHOW FRONTEND CONFIG LIKE "%lake_compaction%"` and `SELECT * FROM information_schema.be_configs WHERE name = "compact_threads"`. 2. Check if Compaction is stuck using `SHOW PROC '/compactions'`: * If `CommitTime` remains NULL, check the system view `information_schema.be_cloud_native_compactions` for the reason why Compaction is stuck. * If `FinishTime` remains NULL, search for the Publish failure reason in the Leader FE log using `TxnID`. 3. Check if compaction is running slowly using `SHOW PROC '/compactions'`: * If `sub_task_count` is too large (check the size of each tablet in this partition using `SHOW PARTITIONS`), the table may be created improperly. * If `read_remote_mb` is too large (more than 30% of the total read data), check the server disk size and also check the cache quota through `SHOW BACKENDS` for field `DataCacheMetrics`. * If `write_remote_sec` is too large (more than 90% of the total Compaction time), write to the remote storage may be too slow. This can be verified by checking the shared-data-specific monitoring metrics with keywords `single upload latency` and `multi upload latency`. * If `in_queue_sec` is too large (average waiting time per tablet exceeds 60 seconds), the parameter settings may be unreasonable or other running Compactions are too slow. --- ### Enable FQDN access This topic describes how to enable cluster access by using a fully qualified domain name (FQDN). An FQDN is a **complete domain name** for a specific entity that can be accessed over the Internet. The FQDN consists of two parts: the hostname and the domain name. Before 2.4, StarRocks supports access to FEs and BEs via IP address only. Even if an FQDN is used to add a node to a cluster, it is transformed into an IP address eventually. This causes a huge inconvenience for DBAs because changing the IP addresses of certain nodes in a StarRocks cluster can lead to access failures to the nodes. In version 2.4, StarRocks decouples each node from its IP address. You can now manage nodes in StarRocks solely via their FQDNs. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") To enable FQDN access for a StarRocks cluster, make sure the following requirements are satisfied: * Each machine in the cluster must have a hostname. * In the file **/etc/hosts** on each machine, you must specify the corresponding IP addresses and FQDNs of other machines in the cluster. * IP addresses in the **/etc/hosts** file must be unique. #### Set up a new cluster with FQDN access[​](#set-up-a-new-cluster-with-fqdn-access "Direct link to Set up a new cluster with FQDN access") By default, FE nodes in a new cluster are started via IP address access. To start a new cluster with FQDN access, you must start the FE nodes by running the following commands **when you start the cluster for the first time**: ```shell ./bin/start_fe.sh --host_type FQDN --daemon ``` The property `--host_type` specifies the access method that is used to start the node. Valid values include `FQDN` and `IP`. You only need to specify this property ONCE when you start the node for the first time. Each BE node identifies itself with `BE Address` defined in the FE metadata. Therefore, you DO NOT need to specify `--host_type` when you start BE nodes. If the `BE Address` defines a BE node with an FQDN, the BE node identifies itself with this FQDN. #### Enable FQDN access in an existing cluster[​](#enable-fqdn-access-in-an-existing-cluster "Direct link to Enable FQDN access in an existing cluster") To enable FQDN access in an existing cluster that was previously started via IP addresses, you must first **upgrade** StarRocks to version 2.4.0 or later. ##### Enable FQDN access for FE nodes[​](#enable-fqdn-access-for-fe-nodes "Direct link to Enable FQDN access for FE nodes") You need to enable FQDN access for all the non-Leader Follower FE nodes before enabling that for the Leader FE node. > **CAUTION** > > Make sure that the cluster has at least three Follower FE nodes before you enable FQDN access for FE nodes. ###### Enable FQDN access for non-Leader Follower FE nodes[​](#enable-fqdn-access-for-non-leader-follower-fe-nodes "Direct link to Enable FQDN access for non-Leader Follower FE nodes") 1. Navigate to the deployment directory of the FE node, and run the following command to stop the FE node: ```shell ./bin/stop_fe.sh ``` 2. Execute the following statement via your MySQL client to check the `Alive` status of the FE node that you have stopped. Wait until the `Alive` status becomes `false`. ```sql SHOW PROC '/frontends'\G ``` 3. Execute the following statement to replace the IP address with FQDN. ```sql ALTER SYSTEM MODIFY FRONTEND HOST "" TO ""; ``` 4. Run the following command to start the FE node with FQDN access. ```shell ./bin/start_fe.sh --host_type FQDN --daemon ``` The property `--host_type` specifies the access method that is used to start the node. Valid values include `FQDN` and `IP`. You only need to specify this property ONCE when you restart the node after you modify the node. 5. Check the `Alive` status of the FE node. Wait until the `Alive` status becomes `true`. ```sql SHOW PROC '/frontends'\G ``` 6. Repeat the above steps to enable FQDN access for the other non-Leader Follower FE nodes one after another when the `Alive` status of the current FE node is `true`. ###### Enable FQDN access for the Leader FE node[​](#enable-fqdn-access-for-the-leader-fe-node "Direct link to Enable FQDN access for the Leader FE node") After all the non-Leader FE nodes have been modified and restarted successfully, you can now enable FQDN access for the Leader FE node. > **NOTE** > > Before the Leader FE node is enabled with FQDN access, the FQDNs used to add nodes to a cluster are still transformed into the corresponding IP addresses. After a Leader FE node with FQDN access enabled is elected for the cluster, the FQDNs will not be transformed into IP addresses. 1. Navigate to the deployment directory of the Leader FE node, and run the following command to stop the Leader FE node. ```shell ./bin/stop_fe.sh ``` 2. Execute the following statement via your MySQL client to check whether a new Leader FE node has been elected for the cluster. ```sql SHOW PROC '/frontends'\G ``` Any FE node with status `Alive` and `isMaster` being `true` is a Leader FE that is running. 3. Execute the following statement to replace the IP address with FQDN. ```sql ALTER SYSTEM MODIFY FRONTEND HOST "" TO ""; ``` 4. Run the following command to start the FE node with FQDN access. ```shell ./bin/start_fe.sh --host_type FQDN --daemon ``` The property `--host_type` specifies the access method that is used to start the node. Valid values include `FQDN` and `IP`. You only need to specify this property ONCE when you restart the node after you modify the node. 5. Check the `Alive` status of the FE node. ```plain SHOW PROC '/frontends'\G ``` If the `Alive` status becomes `true`, the FE node is successfully modified and added to the cluster as a Follower FE node. ##### Enable FQDN access for BE nodes[​](#enable-fqdn-access-for-be-nodes "Direct link to Enable FQDN access for BE nodes") Execute the following statement via your MySQL client to replace the IP address with FQDN to enable FQDN access for the BE node. ```sql ALTER SYSTEM MODIFY BACKEND HOST "" TO ""; ``` > **NOTE** > > You DO NOT need to restart the BE node after FQDN access is enabled. #### Rollback[​](#rollback "Direct link to Rollback") To rollback an FQDN access-enabled StarRocks cluster to an earlier version that does not support FQDN access, you must first enable IP address access for all nodes in the cluster. You can refer [Enable FQDN access in an existing cluster](#enable-fqdn-access-in-an-existing-cluster) for the general guidance except that you need to change the SQL commands to the following ones: * Enable IP address access for an FE node: ```sql ALTER SYSTEM MODIFY FRONTEND HOST "" TO ""; ``` * Enable IP address access for a BE node: ```sql ALTER SYSTEM MODIFY BACKEND HOST "" TO ""; ``` The modification takes effect after your cluster is successfully restarted. #### FAQ[​](#faq "Direct link to FAQ") **Q: An error occurs when I enable FQDN access for an FE node: "required 1 replica. But none were active with this master". What should I do?** A: Make sure the cluster has at least three Follower FE nodes before you enable FQDN access for FE nodes. **Q: Can I add a new node by using IP address to a cluster with FQDN access enabled?** A: Yes. --- ### Develop Static Extensions Static Extensions are StarRocks FE’s extension modules that allow you to add new features or optimize existing functionality without modifying the core code. Compared to dynamic plugins, static extensions are automatically loaded at system startup and provide more registrable extension points, covering the system’s core modules. This feature is supported from v4.1 onwards. #### Usage[​](#usage "Direct link to Usage") The following example demonstrates how to develop a static extension. ##### Prerequisites[​](#prerequisites "Direct link to Prerequisites") Prepare StarRocks FE development environment as follows: ```xml org.apache.maven.plugins maven-jar-plugin 3.3.0 default-jar package jar ${your_extension_directory}/** build-ext-jar package jar ${your_extension_name} ${your_extension_directory}/** ext ``` ##### Extension entry point[​](#extension-entry-point "Direct link to Extension entry point") ```java @SRModule(name = "extension_name") public class MultiWarehouseExtension implements StarRocksExtension { @Override public void onLoad(ExtensionContext ctx) { // Register your own class or perform other initialization ctx.register(WarehouseManager.class, new MyWarehouseManager()); ... } } ``` ##### Logs[​](#logs "Direct link to Logs") After building the extension, place the `${your_extension_name}-ext.jar` file into the `Config.ext_dir` directory (default is `FE/lib`), and then restart the FE. Example FE logs after startup are as follows: ```sh 2025-12-26 12:47:46.047+08:00 INFO (main|1) [ExtensionManager.loadExtensionsFromDir():39] start to load extensions 2025-12-26 12:47:46.152+08:00 INFO (main|1) [ExtensionManager.loadExtensions():63] Loaded extension: extension_name 2025-12-26 12:47:46.152+08:00 INFO (main|1) [ExtensionManager.loadExtensionsFromDir():42] all extensions loaded finished ``` * `start to load extensions`: FE has started scanning the extension directory. * `Loaded extension: extension_name`: The extension was successfully loaded. * `all extensions loaded finished`: All extensions in the directory have been loaded. --- ### FE Configuration FE parameters are classified into dynamic parameters and static parameters. * Dynamic parameters can be configured and adjusted by running SQL commands, which is very convenient. But the configurations become invalid if you restart your FE. Therefore, we recommend that you also modify the configuration items in the **fe.conf** file to prevent the loss of modifications. * Static parameters can only be configured and adjusted in the FE configuration file **fe.conf**. **After you modify this file, you must restart your FE for the changes to take effect.** Whether a parameter is a dynamic parameter is indicated by the `IsMutable` column in the output of [ADMIN SHOW CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SHOW_CONFIG.md). `TRUE` indicates a dynamic parameter. Note that both dynamic and static FE parameters can be configured in the **fe.conf** file. #### View FE configuration items[​](#view-fe-configuration-items "Direct link to View FE configuration items") After your FE is started, you can run the ADMIN SHOW FRONTEND CONFIG command on your MySQL client to check the parameter configurations. If you want to query the configuration of a specific parameter, run the following command: ```sql ADMIN SHOW FRONTEND CONFIG [LIKE "pattern"]; ``` For detailed description of the returned fields, see [`ADMIN SHOW CONFIG`](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SHOW_CONFIG.md). note You must have administrator privileges to run cluster administration-related commands. #### Configure FE parameters[​](#configure-fe-parameters "Direct link to Configure FE parameters") ##### Configure FE dynamic parameters[​](#configure-fe-dynamic-parameters "Direct link to Configure FE dynamic parameters") You can configure or modify the settings of FE dynamic parameters using [`ADMIN SET FRONTEND CONFIG`](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md). ```sql ADMIN SET FRONTEND CONFIG ("key" = "value"); ``` note The configuration changes made with `ADMIN SET FRONTEND` will be restored to the default values in the `fe.conf` file after the FE restarts. Therefore, we recommend that you also modify the configuration items in `fe.conf` if you want the changes to be permanent. ##### Configure FE static parameters[​](#configure-fe-static-parameters "Direct link to Configure FE static parameters") note Static parameters of an FE are set by changing them in the configuration file **fe.conf** and restarting the FE to allow the changes to take effect. #### Parameter groups[​](#parameter-groups "Direct link to Parameter groups") The parameters are grouped in these categories: * [Logging](https://docs.starrocks.io/docs/administration/management/FE_parameters/log_server_meta.md) * [Server](https://docs.starrocks.io/docs/administration/management/FE_parameters/log_server_meta.md) * [Metadata and Cluster management](https://docs.starrocks.io/docs/administration/management/FE_parameters/log_server_meta.md) * [User, role, and privilege](https://docs.starrocks.io/docs/administration/management/FE_parameters/user_query_loading.md) * [Query engine](https://docs.starrocks.io/docs/administration/management/FE_parameters/user_query_loading.md) * [Loading and unloading](https://docs.starrocks.io/docs/administration/management/FE_parameters/user_query_loading.md) * [Statistic report](https://docs.starrocks.io/docs/administration/management/FE_parameters/stats_storage.md) * [Storage](https://docs.starrocks.io/docs/administration/management/FE_parameters/stats_storage.md) * [Shared-data](https://docs.starrocks.io/docs/administration/management/FE_parameters/shared_lake_other.md) * [Data Lake](https://docs.starrocks.io/docs/administration/management/FE_parameters/shared_lake_other.md) * [Other](https://docs.starrocks.io/docs/administration/management/FE_parameters/shared_lake_other.md) --- ### FE Configuration - Logging, Server, and Metadata FE parameters are classified into dynamic parameters and static parameters. * Dynamic parameters can be configured and adjusted by running SQL commands, which is very convenient. But the configurations become invalid if you restart your FE. Therefore, we recommend that you also modify the configuration items in the **fe.conf** file to prevent the loss of modifications. * Static parameters can only be configured and adjusted in the FE configuration file **fe.conf**. **After you modify this file, you must restart your FE for the changes to take effect.** Whether a parameter is a dynamic parameter is indicated by the `IsMutable` column in the output of [ADMIN SHOW CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SHOW_CONFIG.md). `TRUE` indicates a dynamic parameter. Note that both dynamic and static FE parameters can be configured in the **fe.conf** file. #### View FE configuration items[​](#view-fe-configuration-items "Direct link to View FE configuration items") After your FE is started, you can run the ADMIN SHOW FRONTEND CONFIG command on your MySQL client to check the parameter configurations. If you want to query the configuration of a specific parameter, run the following command: ```sql ADMIN SHOW FRONTEND CONFIG [LIKE "pattern"]; ``` For detailed description of the returned fields, see [`ADMIN SHOW CONFIG`](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SHOW_CONFIG.md). note You must have administrator privileges to run cluster administration-related commands. #### Configure FE parameters[​](#configure-fe-parameters "Direct link to Configure FE parameters") ##### Configure FE dynamic parameters[​](#configure-fe-dynamic-parameters "Direct link to Configure FE dynamic parameters") You can configure or modify the settings of FE dynamic parameters using [`ADMIN SET FRONTEND CONFIG`](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md). ```sql ADMIN SET FRONTEND CONFIG ("key" = "value"); ``` note The configuration changes made with `ADMIN SET FRONTEND` will be restored to the default values in the `fe.conf` file after the FE restarts. Therefore, we recommend that you also modify the configuration items in `fe.conf` if you want the changes to be permanent. ##### Configure FE static parameters[​](#configure-fe-static-parameters "Direct link to Configure FE static parameters") note Static parameters of an FE are set by changing them in the configuration file **fe.conf** and restarting the FE to allow the changes to take effect. *** This topic introduces the following types of FE configurations: * [Logging](#logging) * [Server](#server) * [Metadata and Cluster Management](#metadata-and-cluster-management) #### Logging[​](#logging "Direct link to Logging") ##### `audit_log_delete_age`[​](#audit_log_delete_age "Direct link to audit_log_delete_age") * Default: 30d * Type: String * Unit: - * Is mutable: No * Description: The retention period of audit log files. The default value `30d` specifies that each audit log file can be retained for 30 days. StarRocks checks each audit log file and deletes those that were generated 30 days ago. * Introduced in: - ##### `audit_log_dir`[​](#audit_log_dir "Direct link to audit_log_dir") * Default: `StarRocksFE.STARROCKS_HOME_DIR` + "/log" * Type: String * Unit: - * Is mutable: No * Description: The directory that stores audit log files. * Introduced in: - ##### `audit_log_enable_compress`[​](#audit_log_enable_compress "Direct link to audit_log_enable_compress") * Default: false * Type: Boolean * Unit: N/A * Is mutable: No * Description: When true, the generated Log4j2 configuration appends a ".gz" postfix to rotated audit log filenames (fe.audit.log.\*) so that Log4j2 will produce compressed (.gz) archived audit log files on rollover. The setting is read during FE startup in Log4jConfig.initLogging and is applied to the RollingFile appender for audit logs; it only affects rotated/archived files, not the active audit log. Because the value is initialized at startup, changing it requires restarting the FE to take effect. Use alongside audit log rotation settings (`audit_log_dir`, `audit_log_roll_interval`, `audit_roll_maxsize`, `audit_log_roll_num`). * Introduced in: 3.2.12 ##### `audit_log_json_format`[​](#audit_log_json_format "Direct link to audit_log_json_format") * Default: false * Type: Boolean * Unit: N/A * Is mutable: Yes * Description: When true, FE audit events are emitted as structured JSON (Jackson ObjectMapper serializing a Map of annotated AuditEvent fields) instead of the default pipe-separated "key=value" string. The setting affects all built-in audit sinks handled by AuditLogBuilder: connection audit, query audit, big-query audit (big-query threshold fields are added to the JSON when the event qualifies), and slow-audit output. Fields annotated for big-query thresholds and the "features" field are treated specially (excluded from normal audit entries; included in big-query or feature logs as applicable). Enable this to make logs machine-parsable for log collectors or SIEMs; note it changes the log format and may require updating any existing parsers that expect the legacy pipe-separated format. * Introduced in: 3.2.7 ##### `audit_log_modules`[​](#audit_log_modules "Direct link to audit_log_modules") * Default: `slow_query`, query * Type: String\[] * Unit: - * Is mutable: No * Description: The modules for which StarRocks generates audit log entries. By default, StarRocks generates audit logs for the `slow_query` module and the `query` module. The `connection` module is supported from v3.0. Separate the module names with a comma (,) and a space. * Introduced in: - ##### `audit_log_roll_interval`[​](#audit_log_roll_interval "Direct link to audit_log_roll_interval") * Default: DAY * Type: String * Unit: - * Is mutable: No * Description: The time interval at which StarRocks rotates audit log entries. Valid values: `DAY` and `HOUR`. * If this parameter is set to `DAY`, a suffix in the `yyyyMMdd` format is added to the names of audit log files. * If this parameter is set to `HOUR`, a suffix in the `yyyyMMddHH` format is added to the names of audit log files. * Introduced in: - ##### `audit_log_roll_num`[​](#audit_log_roll_num "Direct link to audit_log_roll_num") * Default: 90 * Type: Int * Unit: - * Is mutable: No * Description: The maximum number of audit log files that can be retained within each retention period specified by the `audit_log_roll_interval` parameter. * Introduced in: - ##### `audit_stmt_before_execute`[​](#audit_stmt_before_execute "Direct link to audit_stmt_before_execute") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Controls whether FE emits a `BEFORE_QUERY` audit event before statement execution. When enabled, ConnectProcessor writes one audit record before execution and still writes the normal `AFTER_QUERY` audit record after execution. For multi-statement requests in this branch, audit records remain command-level rather than per sub-statement. * Introduced in: - ##### `bdbje_log_level`[​](#bdbje_log_level "Direct link to bdbje_log_level") * Default: INFO * Type: String * Unit: - * Is mutable: No * Description: Controls the logging level used by Berkeley DB Java Edition (BDB JE) in StarRocks. During BDB environment initialization BDBEnvironment.initConfigs() applies this value to the Java logger for the `com.sleepycat.je` package and to the BDB JE environment file logging level (`EnvironmentConfig.FILE_LOGGING_LEVEL`). Accepts standard java.util.logging.Level names such as SEVERE, WARNING, INFO, CONFIG, FINE, FINER, FINEST, ALL, OFF. Setting to ALL enables all log messages. Increasing verbosity will raise log volume and may impact disk I/O and performance; the value is read when the BDB environment is initialized, so it takes effect only after environment (re)initialization. * Introduced in: v3.2.0 ##### `big_query_log_delete_age`[​](#big_query_log_delete_age "Direct link to big_query_log_delete_age") * Default: 7d * Type: String * Unit: - * Is mutable: No * Description: Controls how long FE big query log files (`fe.big_query.log.*`) are retained before automatic deletion. The value is passed to Log4j's deletion policy as the IfLastModified age — any rotated big query log whose last-modified time is older than this value will be removed. Supports suffixes include `d` (day), `h` (hour), `m` (minute), and `s` (second). Example: `7d` (7 days), `10h` (10 hours), `60m` (60 minutes), and `120s` (120 seconds). This item works together with `big_query_log_roll_interval` and `big_query_log_roll_num` to determine which files are kept or purged. * Introduced in: v3.2.0 ##### `big_query_log_dir`[​](#big_query_log_dir "Direct link to big_query_log_dir") * Default: `Config.STARROCKS_HOME_DIR + "/log"` * Type: String * Unit: - * Is mutable: No * Description: Directory where the FE writes big query dump logs (`fe.big_query.log.*`). The Log4j configuration uses this path to create a RollingFile appender for `fe.big_query.log` and its rotated files. Rotation and retention are governed by `big_query_log_roll_interval` (time-based suffix), `log_roll_size_mb` (size trigger), `big_query_log_roll_num` (max files), and `big_query_log_delete_age` (age-based deletion). Big query records are logged for queries that exceed user-defined thresholds such as `big_query_log_cpu_second_threshold`, `big_query_log_scan_rows_threshold`, or `big_query_log_scan_bytes_threshold`. Use `big_query_log_modules` to control which modules log to this file. * Introduced in: v3.2.0 ##### `big_query_log_modules`[​](#big_query_log_modules "Direct link to big_query_log_modules") * Default: `{"query"}` * Type: String\[] * Unit: - * Is mutable: No * Description: List of module name suffixes that enable per-module big query logging. Typical values are logical component names. For example, the default `query` produces `big_query.query`. * Introduced in: v3.2.0 ##### `big_query_log_roll_interval`[​](#big_query_log_roll_interval "Direct link to big_query_log_roll_interval") * Default: `"DAY"` * Type: String * Unit: - * Is mutable: No * Description: Specifies the time interval used to construct the date component of the rolling file name for the `big_query` log appender. Valid values (case-insensitive) are `DAY` (default) and `HOUR`. `DAY` produces a daily pattern (`"%d{yyyyMMdd}"`) and `HOUR` produces an hourly pattern (`"%d{yyyyMMddHH}"`). The value is combined with size-based rollover (`big_query_roll_maxsize`) and index-based rollover (`big_query_log_roll_num`) to form the RollingFile filePattern. An invalid value causes log configuration generation to fail (IOException) and may prevent log initialization or reconfiguration. Use alongside `big_query_log_dir`, `big_query_roll_maxsize`, `big_query_log_roll_num`, and `big_query_log_delete_age`. * Introduced in: v3.2.0 ##### `big_query_log_roll_num`[​](#big_query_log_roll_num "Direct link to big_query_log_roll_num") * Default: 10 * Type: Int * Unit: - * Is mutable: No * Description: Maximum number of rotated FE big query log files to retain per `big_query_log_roll_interval`. This value is bound to the RollingFile appender's DefaultRolloverStrategy `max` attribute for `fe.big_query.log`; when logs roll (by time or by `log_roll_size_mb`), StarRocks keeps up to `big_query_log_roll_num` indexed files (filePattern uses a time suffix plus index). Files older than this count may be removed by rollover, and `big_query_log_delete_age` can additionally delete files by last-modified age. * Introduced in: v3.2.0 ##### `dump_log_delete_age`[​](#dump_log_delete_age "Direct link to dump_log_delete_age") * Default: 7d * Type: String * Unit: - * Is mutable: No * Description: The retention period of dump log files. The default value `7d` specifies that each dump log file can be retained for 7 days. StarRocks checks each dump log file and deletes those that were generated 7 days ago. * Introduced in: - ##### `dump_log_dir`[​](#dump_log_dir "Direct link to dump_log_dir") * Default: `StarRocksFE.STARROCKS_HOME_DIR` + "/log" * Type: String * Unit: - * Is mutable: No * Description: The directory that stores dump log files. * Introduced in: - ##### `dump_log_modules`[​](#dump_log_modules "Direct link to dump_log_modules") * Default: query * Type: String\[] * Unit: - * Is mutable: No * Description: The modules for which StarRocks generates dump log entries. By default, StarRocks generates dump logs for the query module. Separate the module names with a comma (,) and a space. * Introduced in: - ##### `dump_log_roll_interval`[​](#dump_log_roll_interval "Direct link to dump_log_roll_interval") * Default: DAY * Type: String * Unit: - * Is mutable: No * Description: The time interval at which StarRocks rotates dump log entries. Valid values: `DAY` and `HOUR`. * If this parameter is set to `DAY`, a suffix in the `yyyyMMdd` format is added to the names of dump log files. * If this parameter is set to `HOUR`, a suffix in the `yyyyMMddHH` format is added to the names of dump log files. * Introduced in: - ##### `dump_log_roll_num`[​](#dump_log_roll_num "Direct link to dump_log_roll_num") * Default: 10 * Type: Int * Unit: - * Is mutable: No * Description: The maximum number of dump log files that can be retained within each retention period specified by the `dump_log_roll_interval` parameter. * Introduced in: - ##### `edit_log_write_slow_log_threshold_ms`[​](#edit_log_write_slow_log_threshold_ms "Direct link to edit_log_write_slow_log_threshold_ms") * Default: 2000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: Threshold (in ms) used by JournalWriter to detect and log slow edit-log batch writes. After a batch commit, if the batch duration exceeds this value, JournalWriter emits a WARN with batch size, duration and current journal queue size (rate-limited to once every ~2s). This setting only controls logging/alerts for potential IO or replication latency on the FE leader; it does not change commit or roll behavior (see `edit_log_roll_num` and commit-related settings). Metric updates still occur regardless of this threshold. * Introduced in: v3.2.3 ##### `enable_audit_sql`[​](#enable_audit_sql "Direct link to enable_audit_sql") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: When this item is set to `true`, the FE audit subsystem records the SQL text of statements into FE audit logs (`fe.audit.log`) processed by ConnectProcessor. The stored statement respects other controls: encrypted statements are redacted (`AuditEncryptionChecker`), sensitive credentials may be redacted or desensitized if `enable_sql_desensitize_in_log` is set, and digest recording is controlled by `enable_sql_digest`. When it is set to `false`, ConnectProcessor replaces the statement text with "?" in audit events — other audit fields (user, host, duration, status, slow-query detection via `qe_slow_log_ms`, and metrics) are still recorded. Enabling SQL audit increases forensic and troubleshooting visibility but may expose sensitive SQL content and increase log volume and I/O; disabling it improves privacy at the cost of losing full-statement visibility in audit logs. * Introduced in: - ##### `enable_print_load_profile_to_log`[​](#enable_print_load_profile_to_log "Direct link to enable_print_load_profile_to_log") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: When set to `true`, load profiles (such as Stream Load, Routine Load, Broker Load, and Merge Commit) are additionally written to the profile log (`fe.profile.log`) at INFO level when they are pushed to `ProfileManager`, as a single-line JSON record in the same format as the query profile log. This makes load profiles recoverable from the log even after they are evicted from `ProfileManager` due to the `profile_info_reserved_num` limit. The profile log is used (rather than `fe.log`) because its JSON layout caps strings at `sys_log_json_profile_max_string_length` instead of the much smaller `sys_log_json_max_string_length`, so large load profiles are not truncated; the file is rotated and retained by the `profile_log_*` parameters. Only profiles whose query type is `Load` are printed; query profiles are not affected. A load profile is printed only when it is actually collected (for example, when `enable_profile` is enabled or the load exceeds the big-load profile threshold). * Introduced in: - ##### `enable_profile_log`[​](#enable_profile_log "Direct link to enable_profile_log") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable profile logging. When this feature is enabled, the FE writes per-query profile logs (the serialized `queryDetail` JSON produced by `ProfileManager`) to the profile log sink. This logging is performed only if `enable_collect_query_detail_info` is also enabled; when `enable_profile_log_compress` is enabled, the JSON may be gzipped before logging. Profile log files are managed by `profile_log_dir`, `profile_log_roll_num`, `profile_log_roll_interval` and rotated/deleted according to `profile_log_delete_age` (supports formats like `7d`, `10h`, `60m`, `120s`). Disabling this feature stops writing profile logs (reducing disk I/O, compression CPU and storage usage). Which queries are logged can be further filtered by `profile_log_latency_threshold_ms`. * Introduced in: v3.2.5 ##### `enable_qe_slow_log`[​](#enable_qe_slow_log "Direct link to enable_qe_slow_log") * Default: true * Type: Boolean * Unit: N/A * Is mutable: Yes * Description: When enabled, the FE builtin audit plugin (AuditLogBuilder) will write query events whose measured execution time ("Time" field) exceeds the threshold configured by `qe_slow_log_ms` into the slow-query audit log (AuditLog.getSlowAudit). If disabled, those slow-query entries are suppressed (regular query and connection audit logs are unaffected). The slow-audit entries follow the global `audit_log_json_format` setting (JSON vs. plain string). Use this flag to control generation of slow-query audit volume independently of regular audit logging; turning it off may reduce log I/O when `qe_slow_log_ms` is low or workloads produce many long-running queries. * Introduced in: 3.2.11 ##### `enable_sql_desensitize_in_log`[​](#enable_sql_desensitize_in_log "Direct link to enable_sql_desensitize_in_log") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: When this item is set to `true`, the system replaces or hides sensitive SQL content before it is written to logs, query-detail records, and query profiles. Code paths that honor this configuration include ConnectProcessor.formatStmt (audit logs), StmtExecutor.addRunningQueryDetail (query details), SimpleExecutor.formatSQL (internal executor logs), and StmtExecutor.buildTopLevelProfile / processProfileAsync (the `Sql Statement` and `ExplainPlan` info-strings stored in a profile's `Summary` section). With the feature enabled, invalid SQLs may be replaced with a fixed desensitized message, credentials (user/password) are hidden, and the SQL formatter is required to produce a sanitized representation (it can also enable digest-style output). For the `ExplainPlan` field added by the `enable_explain_in_profile` session variable, this config also forces literal-digest rendering of the embedded `EXPLAIN COSTS` text, so the profile does not leak the literals that the persisted `Sql Statement` would have hidden. This reduces leakage of sensitive literals and credentials in audit/internal logs and profiles, but also means logs, query details, and profiles no longer contain the original full SQL text (which can affect replay or debugging). * Introduced in: - ##### `internal_log_delete_age`[​](#internal_log_delete_age "Direct link to internal_log_delete_age") * Default: 7d * Type: String * Unit: - * Is mutable: No * Description: Specifies the retention period for FE internal log files (written to `internal_log_dir`). The value is a duration string. Supported suffixes: `d` (day), `h` (hour), `m` (minute), `s` (second). Examples: `7d` (7 days), `10h` (10 hours), `60m` (60 minutes), `120s` (120 seconds). This item is substituted into the log4j configuration as the `` predicate used by the RollingFile Delete policy. Files whose last-modified time is earlier than this duration will be removed during log rollover. Increase this value to free disk space sooner, or decrease it to retain internal materialized view or statistics logs longer. * Introduced in: v3.2.4 ##### `internal_log_dir`[​](#internal_log_dir "Direct link to internal_log_dir") * Default: `Config.STARROCKS_HOME_DIR` + "/log" * Type: String * Unit: - * Is mutable: No * Description: Directory used by the FE logging subsystem for storing internal logs (`fe.internal.log`). This configuration is substituted into the Log4j configuration and determines where the InternalFile appender writes internal/materialized view/statistics logs and where per-module loggers under `internal.` place their files. Ensure the directory exists, is writable, and has sufficient disk space. Log rotation and retention for files in this directory are controlled by `log_roll_size_mb`, `internal_log_roll_num`, `internal_log_delete_age`, and `internal_log_roll_interval`. If `sys_log_to_console` is enabled, internal logs may be written to console instead of this directory. * Introduced in: v3.2.4 ##### `internal_log_json_format`[​](#internal_log_json_format "Direct link to internal_log_json_format") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: When this item is set to `true`, internal statistic/audit entries are written as compact JSON objects to the statistic audit logger. The JSON contains keys "executeType" (InternalType: QUERY or DML), "queryId", "sql", and "time" (elapsed milliseconds). When it is set to `false`, the same information is logged as a single formatted text line ("statistic execute: ... | QueryId: \[...] | SQL: ..."). Enabling JSON improves machine parsing and integration with log processors but also causes raw SQL text to be included in logs, which may expose sensitive information and increase log size. * Introduced in: - ##### `internal_log_modules`[​](#internal_log_modules "Direct link to internal_log_modules") * Default: `{"base", "statistic"}` * Type: String\[] * Unit: - * Is mutable: No * Description: A list of module identifiers that will receive dedicated internal logging. For each entry X, Log4j creates a logger named `internal.` with level INFO and additivity="false". Those loggers are routed to the internal appender (written to `fe.internal.log`) or to console when `sys_log_to_console` is enabled. Use short names or package fragments as needed — the exact logger name becomes `internal.` + the configured string. Internal log file rotation and retention follow `internal_log_dir`, `internal_log_roll_num`, `internal_log_delete_age`, `internal_log_roll_interval`, and `log_roll_size_mb`. Adding a module causes its runtime messages to be separated into the internal logger stream for easier debugging and audit. * Introduced in: v3.2.4 ##### `internal_log_roll_interval`[​](#internal_log_roll_interval "Direct link to internal_log_roll_interval") * Default: DAY * Type: String * Unit: - * Is mutable: No * Description: Controls the time-based roll interval for the FE internal log appender. Accepted values (case-insensitive) are `HOUR` and `DAY`. `HOUR` produces an hourly file pattern (`"%d{yyyyMMddHH}"`) and `DAY` produces a daily file pattern (`"%d{yyyyMMdd}"`), which are used by the RollingFile TimeBasedTriggeringPolicy to name rotated `fe.internal.log` files. An invalid value causes initialization to fail (an IOException is thrown when building the active Log4j configuration). Roll behavior also depends on related settings such as `internal_log_dir`, `internal_roll_maxsize`, `internal_log_roll_num`, and `internal_log_delete_age`. * Introduced in: v3.2.4 ##### `internal_log_roll_num`[​](#internal_log_roll_num "Direct link to internal_log_roll_num") * Default: 90 * Type: Int * Unit: - * Is mutable: No * Description: Maximum number of rolled internal FE log files to retain for the internal appender (`fe.internal.log`). This value is used as the Log4j DefaultRolloverStrategy `max` attribute; when rollovers occur, StarRocks keeps up to `internal_log_roll_num` archived files and removes older ones (also governed by `internal_log_delete_age`). A lower value reduces disk usage but shortens log history; a higher value preserves more historical internal logs. This item works together with `internal_log_dir`, `internal_log_roll_interval`, and `internal_roll_maxsize`. * Introduced in: v3.2.4 ##### `log_cleaner_audit_log_min_retention_days`[​](#log_cleaner_audit_log_min_retention_days "Direct link to log_cleaner_audit_log_min_retention_days") * Default: 3 * Type: Int * Unit: Days * Is mutable: Yes * Description: Minimum retention days for audit log files. Audit log files newer than this will not be deleted even if disk usage is high. This ensures that audit logs are preserved for compliance and troubleshooting purposes. * Introduced in: - ##### `log_cleaner_check_interval_second`[​](#log_cleaner_check_interval_second "Direct link to log_cleaner_check_interval_second") * Default: 300 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Interval in seconds to check disk usage and clean logs. The cleaner periodically checks each log directory's disk usage and triggers cleaning when necessary. Default is 300 seconds (5 minutes). * Introduced in: - ##### `log_cleaner_disk_usage_target`[​](#log_cleaner_disk_usage_target "Direct link to log_cleaner_disk_usage_target") * Default: 60 * Type: Int * Unit: Percentage * Is mutable: Yes * Description: Target disk usage (percentage) after log cleaning. Log cleaning will continue until disk usage drops below this threshold. The cleaner deletes the oldest log files one by one until the target is reached. * Introduced in: - ##### `log_cleaner_disk_usage_threshold`[​](#log_cleaner_disk_usage_threshold "Direct link to log_cleaner_disk_usage_threshold") * Default: 80 * Type: Int * Unit: Percentage * Is mutable: Yes * Description: Disk usage threshold (percentage) to trigger log cleaning. When disk usage exceeds this threshold, log cleaning will start. The cleaner checks each configured log directory independently and processes directories that exceed this threshold. * Introduced in: - ##### `log_cleaner_disk_util_based_enable`[​](#log_cleaner_disk_util_based_enable "Direct link to log_cleaner_disk_util_based_enable") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Enable automatic log cleaning based on disk usage. When enabled, logs will be cleaned when disk usage exceeds the threshold. The log cleaner runs as a background daemon on the FE node and helps prevent disk space exhaustion from log file accumulation. * Introduced in: - ##### `log_plan_cancelled_by_crash_be`[​](#log_plan_cancelled_by_crash_be "Direct link to log_plan_cancelled_by_crash_be") * Default: true * Type: boolean * Unit: - * Is mutable: Yes * Description: Whether to enable the query execution plan logging when a query is cancelled due to BE crash or an RPC exception. When this feature is enabled, StarRocks logs the query execution plan (at `TExplainLevel.COSTS`) as a WARN entry when a query is cancelled due to BE crash or an `RpcException`. The log entry includes QueryId, SQL and the COSTS plan; in the ExecuteExceptionHandler path, the exception stacktrace is also logged. The logging is skipped when `enable_collect_query_detail_info` is enabled (the plan is then stored in the query detail) — in code paths, the check is performed by verifying the query detail is null. Note that, in ExecuteExceptionHandler, the plan is logged only on the first retry (`retryTime == 0`). Enabling this may increase log volume because full COSTS plans can be large. * Introduced in: v3.2.0 ##### `log_register_and_unregister_query_id`[​](#log_register_and_unregister_query_id "Direct link to log_register_and_unregister_query_id") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to allow FE to log query registration and deregistration messages (e.g., `"register query id = {}"` and `"deregister query id = {}"`) from QeProcessorImpl. The log is emitted only when the query has a non-null ConnectContext and either the command is not `COM_STMT_EXECUTE` or the session variable `isAuditExecuteStmt()` is true. Because these messages are written for every query lifecycle event, enabling this feature can produce high log volume and become a throughput bottleneck in high concurrency environments. Enable it for debugging or auditing; and disable it to reduce logging overhead and improve performance. * Introduced in: v3.3.0, v3.4.0, v3.5.0 ##### `log_roll_size_mb`[​](#log_roll_size_mb "Direct link to log_roll_size_mb") * Default: 1024 * Type: Int * Unit: MB * Is mutable: No * Description: The maximum size of a system log file or an audit log file. * Introduced in: - ##### `proc_profile_file_retained_days`[​](#proc_profile_file_retained_days "Direct link to proc_profile_file_retained_days") * Default: 1 * Type: Int * Unit: Days * Is mutable: Yes * Description: Number of days to retain process profiling files (CPU and memory) generated under `sys_log_dir/proc_profile`. The ProcProfileCollector computes a cutoff by subtracting `proc_profile_file_retained_days` days from the current time (formatted as yyyyMMdd-HHmmss) and deletes profile files whose timestamp portion is lexicographically earlier than that cutoff (that is, `timePart.compareTo(timeToDelete) < 0`). File deletion also respects the size-based cutoff controlled by `proc_profile_file_retained_size_bytes`. Profile files use the prefixes `cpu-profile-` and `mem-profile-` and are compressed after collection. * Introduced in: v3.2.12 ##### `proc_profile_file_retained_size_bytes`[​](#proc_profile_file_retained_size_bytes "Direct link to proc_profile_file_retained_size_bytes") * Default: 2L \* 1024 \* 1024 \* 1024 (2147483648) * Type: Long * Unit: Bytes * Is mutable: Yes * Description: Maximum total bytes of collected CPU and memory profile files (files named with prefixes `cpu-profile-` and `mem-profile-`) to keep under the profile directory. When the sum of valid profile files exceeds `proc_profile_file_retained_size_bytes`, the collector deletes the oldest profile files until the remaining total size is less than or equal to `proc_profile_file_retained_size_bytes`. Files older than `proc_profile_file_retained_days` are also removed regardless of size. This setting controls disk usage for profile archives and interacts with `proc_profile_file_retained_days` to determine deletion order and retention. * Introduced in: v3.2.12 ##### `profile_log_delete_age`[​](#profile_log_delete_age "Direct link to profile_log_delete_age") * Default: 1d * Type: String * Unit: - * Is mutable: No * Description: Controls how long FE profile log files are retained before they are eligible for deletion. The value is injected into Log4j's `` policy (via `Log4jConfig`) and is applied together with rotation settings such as `profile_log_roll_interval` and `profile_log_roll_num`. Supported suffixes: `d` (day), `h` (hour), `m` (minute), `s` (second). For example: `7d` (7 days), `10h` (10 hours), `60m` (60 minutes), `120s` (120 seconds). * Introduced in: v3.2.5 ##### `profile_log_dir`[​](#profile_log_dir "Direct link to profile_log_dir") * Default: `Config.STARROCKS_HOME_DIR` + "/log" * Type: String * Unit: - * Is mutable: No * Description: Directory where FE profile logs are written. Log4jConfig uses this value to place profile-related appenders (creates files like `fe.profile.log` and `fe.features.log` under this directory). Rotation and retention for these files are governed by `profile_log_roll_size_mb`, `profile_log_roll_num` and `profile_log_delete_age`; the timestamp suffix format is controlled by `profile_log_roll_interval` (supports DAY or HOUR). Because the default directory is under `STARROCKS_HOME_DIR`, ensure the FE process has write and rotation/delete permissions on this directory. * Introduced in: v3.2.5 ##### `profile_log_latency_threshold_ms`[​](#profile_log_latency_threshold_ms "Direct link to profile_log_latency_threshold_ms") * Default: 0 * Type: Long * Unit: Milliseconds * Is mutable: Yes * Description: Minimum query latency (in milliseconds) for a profile to be written to `fe.profile.log`. Only queries whose execution time is greater than or equal to this value are logged. Set to 0 to log all profiles (no threshold). Use a positive value to reduce log volume by logging only slower queries. * Introduced in: - ##### `profile_log_roll_interval`[​](#profile_log_roll_interval "Direct link to profile_log_roll_interval") * Default: DAY * Type: String * Unit: - * Is mutable: No * Description: Controls the time granularity used to generate the date part of profile log filenames. Valid values (case-insensitive) are `HOUR` and `DAY`. `HOUR` produces a pattern of `"%d{yyyyMMddHH}"` (hourly time bucket) and `DAY` produces `"%d{yyyyMMdd}"` (daily time bucket). This value is used when computing `profile_file_pattern` in the Log4j configuration and only affects the time-based component of rollover file names; size-based rollover is still controlled by `profile_log_roll_size_mb` and retention by `profile_log_roll_num` / `profile_log_delete_age`. Invalid values cause an IOException during logging initialization (error message: `"profile_log_roll_interval config error: "`). Choose `HOUR` for high-volume profiling to limit per-file size per hour, or `DAY` for daily aggregation. * Introduced in: v3.2.5 ##### `profile_log_roll_num`[​](#profile_log_roll_num "Direct link to profile_log_roll_num") * Default: 5 * Type: Int * Unit: - * Is mutable: No * Description: Specifies the maximum number of rotated profile log files retained by Log4j's DefaultRolloverStrategy for the profile logger. This value is injected into the logging XML as `${profile_log_roll_num}` (e.g. ``). Rotations are triggered by `profile_log_roll_size_mb` or `profile_log_roll_interval`; when rotation occurs, Log4j keeps at most these indexed files and older index files become eligible for removal. Actual retention on disk is also affected by `profile_log_delete_age` and the `profile_log_dir` location. Lower values reduce disk usage but limit retained history; higher values preserve more historical profile logs. * Introduced in: v3.2.5 ##### `profile_log_roll_size_mb`[​](#profile_log_roll_size_mb "Direct link to profile_log_roll_size_mb") * Default: 1024 * Type: Int * Unit: MB * Is mutable: No * Description: Sets the size threshold (in megabytes) that triggers a size-based rollover of the FE profile log file. This value is used by the Log4j RollingFile SizeBasedTriggeringPolicy for the `ProfileFile` appender; when a profile log exceeds `profile_log_roll_size_mb` it will be rotated. Rotation can also occur by time when `profile_log_roll_interval` is reached — either condition will trigger rollover. Combined with `profile_log_roll_num` and `profile_log_delete_age`, this item controls how many historical profile files are retained and when old files are deleted. Compression of rotated files is controlled by `enable_profile_log_compress`. * Introduced in: v3.2.5 ##### `qe_slow_log_ms`[​](#qe_slow_log_ms "Direct link to qe_slow_log_ms") * Default: 5000 * Type: Long * Unit: Milliseconds * Is mutable: Yes * Description: The threshold used to determine whether a query is a slow query. If the response time of a query exceeds this threshold, it is recorded as a slow query in **fe.audit.log**. * Introduced in: - ##### `slow_lock_log_l2_info_interval_ms`[​](#slow_lock_log_l2_info_interval_ms "Direct link to slow_lock_log_l2_info_interval_ms") * Default: 3000L * Type: Long * Unit: Milliseconds * Is mutable: Yes * Alias: `slow_lock_log_every_ms` (the original name, retained for backward compatibility — both names refer to the same parameter). * Description: Minimum interval (in ms) for the **L2** slow-lock log tier — a full lock-info JSON line **without** stack traces. Slow-lock logging degrades across three tiers, throttled progressively (strictest first): **L1** = full info + stacks (`slow_lock_log_l1_stack_interval_ms`), **L2** = full info, no stacks (this parameter), **L3** = a plain-text brief line (`slow_lock_log_l3_brief_interval_ms`). For one slow-lock event the richest tier whose throttle currently admits is emitted; choosing a higher tier also consumes the looser tiers' windows, so total log volume never exceeds the loosest admitted tier's rate. The throttle scope depends on the emitting layer: **GLOBAL** in `LockManager.logSlowLockTrace` (one static gate across all rids), **per-instance** in `QueryableReentrantReadWriteLock` (each lock object — e.g. each `RoutineLoadJob` — has its own gate), and **per-Database** in the legacy `LockUtils` path. Set to `0` (or negative) to disable the L2 gate (always admit). Use a larger value to reduce log volume during prolonged contention or a smaller value for more frequent full-info diagnostics. * Introduced in: v3.2.0 (as `slow_lock_log_every_ms`); renamed to `slow_lock_log_l2_info_interval_ms` in v4.1. ##### `slow_lock_print_stack`[​](#slow_lock_print_stack "Direct link to slow_lock_print_stack") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Master switch for capturing owner / current-thread stack traces inside slow-lock warnings. Applies to both `LockManager.logSlowLockTrace` (per-owner `"stack"` field) and `QueryableReentrantReadWriteLock.getLockInfoToJson` (owner / oldest-reader / current-thread `"stack"` field used by the legacy db-lock path and `RoutineLoadJob`'s per-job lock). Enabling this feature helps debugging by giving precise thread stacks that hold the lock; disabling it reduces log volume and CPU/memory overhead caused by capturing and serializing stack traces in high concurrency environments. When enabled, the capture frequency is additionally rate-limited by `slow_lock_log_l1_stack_interval_ms`. * Introduced in: v3.3.16, v3.4.5, v3.5.1 ##### `slow_lock_log_l1_stack_interval_ms`[​](#slow_lock_log_l1_stack_interval_ms "Direct link to slow_lock_log_l1_stack_interval_ms") * Default: 30000 * Type: Long * Unit: Milliseconds * Is mutable: Yes * Description: Minimum interval between stack-trace captures across slow-lock log events. Only applies when `slow_lock_print_stack` is `true`. The throttle scope depends on the layer: **GLOBAL** in `LockManager.logSlowLockTrace` (one static gate across all rids) and **per-instance** in `QueryableReentrantReadWriteLock.getLockInfoToJson` (each lock object has its own gate). When the switch is on but the interval has not elapsed since the last capture, the `"stack"` field is replaced with the marker `"throttled"` (LockManager path) or omitted (QueryableReentrantReadWriteLock path), and the rest of the warn log (rid, owners, waiters, queryIds, timings) is still emitted if the outer event gate (`slow_lock_log_l2_info_interval_ms`) lets it through. Set to `0` (or negative) to disable rate limiting and restore the prior behavior of capturing stacks on every slow-lock event. `Thread.getStackTrace` triggers a JVM safepoint that becomes expensive in large clusters where slow-lock events are frequent — this gate caps that cost without suppressing the diagnostic log itself. * Introduced in: v4.1 ##### `slow_lock_max_waiter_count_to_log`[​](#slow_lock_max_waiter_count_to_log "Direct link to slow_lock_max_waiter_count_to_log") * Default: 30 * Type: Int * Unit: - * Is mutable: Yes * Description: Maximum number of waiter entries serialized into a single slow-lock log event. Applies to both `LockManager.logSlowLockTrace` (the `"waiter"` array) and `QueryableReentrantReadWriteLock.getLockInfoToJson` (the `"queuedReaders"` / `"queuedWriters"` arrays consumed by the legacy db-lock path and `RoutineLoadJob`'s per-job lock). When the actual waiter count exceeds this cap, the first N waiters are listed individually and the remainder is summarized as a single trailer entry `{"omitted": "remain M waiters omitted"}` appended to the array. Bounds Gson serialization cost and log-line size under extreme contention without losing the count diagnostic. Set to `0` (or negative) to disable the cap and serialize every waiter. * Introduced in: v4.1 ##### `slow_lock_log_l3_brief_interval_ms`[​](#slow_lock_log_l3_brief_interval_ms "Direct link to slow_lock_log_l3_brief_interval_ms") * Default: 1000 * Type: Long * Unit: Milliseconds * Is mutable: Yes * Description: Minimum interval for the **L3** slow-lock log tier — a single plain-text warn line (no JSON, no stacks) emitted when the richer tiers (`slow_lock_log_l1_stack_interval_ms` for L1, `slow_lock_log_l2_info_interval_ms` for L2) are throttled. This is the loosest of the three tiers. The brief line is emitted **at most once per this interval**: slow-lock events that arrive while the L3 gate is still closed are suppressed (no line). It does **not** guarantee a log line per event — it bounds the worst-case silence to one brief interval during sustained contention. Tune it smaller than the other two: `slow_lock_log_l3_brief_interval_ms < slow_lock_log_l2_info_interval_ms < slow_lock_log_l1_stack_interval_ms`. Set to `0` (or negative) to make the brief line unthrottled — then every otherwise-throttled event leaves a line (predictable but potentially many per second under a storm). Same scope rules as the other slow-lock throttles (GLOBAL in `LockManager`, per-instance in `QueryableReentrantReadWriteLock`). * Introduced in: v4.1 ##### `slow_lock_threshold_ms`[​](#slow_lock_threshold_ms "Direct link to slow_lock_threshold_ms") * Default: 3000L * Type: long * Unit: Milliseconds * Is mutable: Yes * Description: Threshold (in ms) used to classify a lock operation or a held lock as "slow". When the elapsed wait or hold time for a lock exceeds this value, StarRocks will (depending on context) emit diagnostic logs, include stack traces or waiter/owner info, and—in LockManager—start deadlock detection after this delay. It's used by LockUtils (slow-lock logging), QueryableReentrantReadWriteLock (filtering slow readers), LockManager (deadlock-detection delay and slow-lock trace), LockChecker (periodic slow-lock detection), and other callers (e.g., DiskAndTabletLoadReBalancer logging). Lowering the value increases sensitivity and logging/diagnostic overhead; setting it to 0 or negative disables the initial wait-based deadlock-detection delay behavior. Tune together with `slow_lock_log_l2_info_interval_ms`, `slow_lock_print_stack`, and `slow_lock_stack_trace_reserve_levels`. * Introduced in: 3.2.0 ##### `sys_log_delete_age`[​](#sys_log_delete_age "Direct link to sys_log_delete_age") * Default: 7d * Type: String * Unit: - * Is mutable: No * Description: The retention period of system log files. The default value `7d` specifies that each system log file can be retained for 7 days. StarRocks checks each system log file and deletes those that were generated 7 days ago. * Introduced in: - ##### `sys_log_dir`[​](#sys_log_dir "Direct link to sys_log_dir") * Default: `StarRocksFE.STARROCKS_HOME_DIR` + "/log" * Type: String * Unit: - * Is mutable: No * Description: The directory that stores system log files. * Introduced in: - ##### `sys_log_enable_compress`[​](#sys_log_enable_compress "Direct link to sys_log_enable_compress") * Default: false * Type: boolean * Unit: - * Is mutable: No * Description: When this item is set to `true`, the system appends a ".gz" postfix to rotated system log filenames so Log4j will produce gzip-compressed rotated FE system logs (for example, fe.log.\*). This value is read during Log4j configuration generation (Log4jConfig.initLogging / generateActiveLog4jXmlConfig) and controls the `sys_file_postfix` property used in the RollingFile filePattern. Enabling this feature reduces disk usage for retained logs but increases CPU and I/O during rollovers and changes log filenames, so that tools or scripts that read logs must be able to handle .gz files. Note that audit logs use a separate configuration for compression, that is, `audit_log_enable_compress`. * Introduced in: v3.2.12 ##### `sys_log_format`[​](#sys_log_format "Direct link to sys_log_format") * Default: "plaintext" * Type: String * Unit: - * Is mutable: No * Description: Selects the Log4j layout used for FE logs. Valid values: `"plaintext"` (Default) and `"json"`. The values are case-insensitive. `"plaintext"` configures PatternLayout with human-readable timestamps, level, thread, class.method :line and stack traces for WARN/ERROR. `"json"` configures JsonTemplateLayout and emits structured JSON events (UTC timestamps, level, thread id/name, source file/method/line, message, exception stackTrace) suitable for log aggregators (ELK, Splunk). JSON output abides by `sys_log_json_max_string_length` and `sys_log_json_profile_max_string_length` for maximum string lengths. * Introduced in: v3.2.10 ##### `sys_log_json_max_string_length`[​](#sys_log_json_max_string_length "Direct link to sys_log_json_max_string_length") * Default: 1048576 * Type: Int * Unit: Bytes * Is mutable: No * Description: Sets the JsonTemplateLayout "maxStringLength" value used for the JSON-formatted system logs. When `sys_log_format` is set to `"json"`, string-valued fields (for example "message" and stringified exception stack traces) are truncated if their length exceeds this limit. The value is injected into the generated Log4j XML in `Log4jConfig.generateActiveLog4jXmlConfig()`, and is applied to default, warning, audit, dump and bigquery layouts. The profile layout uses a separate configuration (`sys_log_json_profile_max_string_length`). Lowering this value reduces log size but can truncate useful information. * Introduced in: 3.2.11 ##### `sys_log_json_profile_max_string_length`[​](#sys_log_json_profile_max_string_length "Direct link to sys_log_json_profile_max_string_length") * Default: 104857600 (100 MB) * Type: Int * Unit: Bytes * Is mutable: No * Description: Sets the maxStringLength of JsonTemplateLayout for profile (and related feature) log appenders when `sys_log_format` is "json". String field values in JSON-formatted profile logs will be truncated to this byte length; non-string fields are unaffected. This item is applied in Log4jConfig `JsonTemplateLayout maxStringLength` and is ignored when `plaintext` logging is used. Keep the value large enough for full messages you need, but note larger values increase log size and I/O. * Introduced in: v3.2.11 ##### `sys_log_level`[​](#sys_log_level "Direct link to sys_log_level") * Default: INFO * Type: String * Unit: - * Is mutable: No * Description: The severity levels into which system log entries are classified. Valid values: `INFO`, `WARN`, `ERROR`, and `FATAL`. * Introduced in: - ##### `sys_log_roll_interval`[​](#sys_log_roll_interval "Direct link to sys_log_roll_interval") * Default: DAY * Type: String * Unit: - * Is mutable: No * Description: The time interval at which StarRocks rotates system log entries. Valid values: `DAY` and `HOUR`. * If this parameter is set to `DAY`, a suffix in the `yyyyMMdd` format is added to the names of system log files. * If this parameter is set to `HOUR`, a suffix in the `yyyyMMddHH` format is added to the names of system log files. * Introduced in: - ##### `sys_log_roll_num`[​](#sys_log_roll_num "Direct link to sys_log_roll_num") * Default: 10 * Type: Int * Unit: - * Is mutable: No * Description: The maximum number of system log files that can be retained within each retention period specified by the `sys_log_roll_interval` parameter. * Introduced in: - ##### `sys_log_to_console`[​](#sys_log_to_console "Direct link to sys_log_to_console") * Default: false (unless the environment variable `SYS_LOG_TO_CONSOLE` is set to "1") * Type: Boolean * Unit: - * Is mutable: No * Description: When this item is set to `true`, the system configures Log4j to send all logs to the console (ConsoleErr appender) instead of the file-based appenders. This value is read when generating the active Log4j XML configuration (which affects the root logger and per-module logger appender selection). Its value is captured from the `SYS_LOG_TO_CONSOLE` environment variable at process startup. Changing it at runtime has no effect. This configuration is commonly used in containerized or CI environments where stdout/stderr log collection is preferred over writing log files. * Introduced in: v3.2.0 ##### `sys_log_verbose_modules`[​](#sys_log_verbose_modules "Direct link to sys_log_verbose_modules") * Default: Empty string * Type: String\[] * Unit: - * Is mutable: No * Description: The modules for which StarRocks generates system logs. If this parameter is set to `org.apache.starrocks.catalog`, StarRocks generates system logs only for the catalog module. Separate the module names with a comma (,) and a space. * Introduced in: - ##### `sys_log_warn_modules`[​](#sys_log_warn_modules "Direct link to sys_log_warn_modules") * Default: * Type: String\[] * Unit: - * Is mutable: No * Description: A list of logger names or package prefixes that the system will configure at startup as WARN-level loggers and route to the warning appender (SysWF) — the `fe.warn.log` file. Entries are inserted into the generated Log4j configuration (alongside builtin warn modules such as org.apache.kafka, org.apache.hudi, and org.apache.hadoop.io.compress) and produce logger elements like ``. Fully-qualified package and class prefixes (for example, "com.example.lib") are recommended to suppress noisy INFO/DEBUG output into the regular log and to allow warnings to be captured separately. * Introduced in: v3.2.13 #### Server[​](#server "Direct link to Server") ###### enable\_auth\_check[​](#enable_auth_check "Direct link to enable_auth_check") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: Specifies whether to enable the authentication check feature. Valid values: `TRUE` and `FALSE`. `TRUE` specifies to enable this feature, and `FALSE` specifies to disable this feature. * Introduced in: - ##### `brpc_idle_wait_max_time`[​](#brpc_idle_wait_max_time "Direct link to brpc_idle_wait_max_time") * Default: 10000 * Type: Int * Unit: ms * Is mutable: No * Description: The maximum length of time for which bRPC clients wait as in the idle state. * Introduced in: - ##### `brpc_inner_reuse_pool`[​](#brpc_inner_reuse_pool "Direct link to brpc_inner_reuse_pool") * Default: true * Type: boolean * Unit: - * Is mutable: No * Description: Controls whether the underlying BRPC client uses an internal shared reuse pool for connections/channels. StarRocks reads `brpc_inner_reuse_pool` in BrpcProxy when constructing RpcClientOptions (via `rpcOptions.setInnerResuePool(...)`). When enabled (true) the RPC client reuses internal pools to reduce per-call connection creation, lowering connection churn, memory and file-descriptor usage for FE-to-BE / LakeService RPCs. When disabled (false) the client may create more isolated pools (increasing concurrency isolation at the cost of higher resource usage). Changing this value requires restarting the process to take effect. * Introduced in: v3.3.11, v3.4.1, v3.5.0 ##### `brpc_min_evictable_idle_time_ms`[​](#brpc_min_evictable_idle_time_ms "Direct link to brpc_min_evictable_idle_time_ms") * Default: 120000 * Type: Int * Unit: Milliseconds * Is mutable: No * Description: Time in milliseconds that an idle BRPC connection must remain in the connection pool before it becomes eligible for eviction. Applied to the RpcClientOptions used by `BrpcProxy` (via RpcClientOptions.setMinEvictableIdleTime). Raise this value to keep idle connections longer (reducing reconnect churn); lower it to free unused sockets faster (reducing resource usage). Tune together with `brpc_connection_pool_size` and `brpc_idle_wait_max_time` to balance connection reuse, pool growth, and eviction behavior. * Introduced in: v3.3.11, v3.4.1, v3.5.0 ##### `brpc_reuse_addr`[​](#brpc_reuse_addr "Direct link to brpc_reuse_addr") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: When true, StarRocks sets the socket option to allow local address reuse for client sockets created by the brpc RpcClient (via RpcClientOptions.setReuseAddress). Enabling this reduces bind failures and allows faster rebinding of local ports after sockets are closed, which is helpful for high-rate connection churn or rapid restarts. When false, address/port reuse is disabled, which can reduce the chance of unintended port sharing but may increase transient bind errors. This option interacts with connection behavior configured by `brpc_connection_pool_size` and `brpc_short_connection` because it affects how rapidly client sockets can be rebound and reused. * Introduced in: v3.3.11, v3.4.1, v3.5.0 ###### `brpc_connection_pool_retry_wait_time_ms`[​](#brpc_connection_pool_retry_wait_time_ms "Direct link to brpc_connection_pool_retry_wait_time_ms") * Default: 10 * Type: Int * Unit: ms * Is mutable: Yes * Description: The wait time before retrying when a bRPC connection pool exception occurs (e.g. SYN packet loss during TCP handshake). When `ChannelPool.getChannel()` throws a `NoSuchElementException` (directly or wrapped in a `RuntimeException`), the retry logic sleeps for this duration before attempting to reconnect. * Introduced in: - ##### `cluster_name`[​](#cluster_name "Direct link to cluster_name") * Default: StarRocks Cluster * Type: String * Unit: - * Is mutable: No * Description: The name of the StarRocks cluster to which the FE belongs. The cluster name is displayed for `Title` on the web page. * Introduced in: - ##### `dns_cache_ttl_seconds`[​](#dns_cache_ttl_seconds "Direct link to dns_cache_ttl_seconds") * Default: 60 * Type: Int * Unit: Seconds * Is mutable: No * Description: DNS cache TTL (Time-To-Live) in seconds for successful DNS lookups. This sets the Java security property `networkaddress.cache.ttl` which controls how long the JVM caches successful DNS lookups. Set this item to `-1` to allow the system to always cache the infomration, or `0` to disable caching. This is particularly useful in environments where IP addresses change frequently, such as Kubernetes deployments or when dynamic DNS is used. * Introduced in: v3.5.11, v4.0.4 ##### `enable_http_async_handler`[​](#enable_http_async_handler "Direct link to enable_http_async_handler") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to allow the system to process HTTP requests asynchronously. If this feature is enabled, an HTTP request received by Netty worker threads will then be submitted to a separate thread pool for service logic handling to avoid blocking the HTTP server. If disabled, Netty workers will handle the service logic. * Introduced in: 4.0.0 ##### `enable_http_validate_headers`[​](#enable_http_validate_headers "Direct link to enable_http_validate_headers") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: Controls whether Netty's HttpServerCodec performs strict HTTP header validation. The value is passed to HttpServerCodec when the HTTP pipeline is initialized in `HttpServer` (see UseLocations). Default is false for backward compatibility because newer netty versions enforce stricter header rules (). Set to true to enforce RFC-compliant header checks; doing so may cause malformed or nonconforming requests from legacy clients or proxies to be rejected. Change requires a restart of the HTTP server to take effect. * Introduced in: v3.3.0, v3.4.0, v3.5.0 ##### `enable_https`[​](#enable_https "Direct link to enable_https") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to enable HTTPS server alongside HTTP server in FE nodes. * Introduced in: v4.0 ##### `frontend_address`[​](#frontend_address "Direct link to frontend_address") * Default: 0.0.0.0 * Type: String * Unit: - * Is mutable: No * Description: The IP address of the FE node. * Introduced in: - ##### `http_async_threads_num`[​](#http_async_threads_num "Direct link to http_async_threads_num") * Default: 4096 * Type: Int * Unit: - * Is mutable: Yes * Description: Size of the thread pool for asynchronous HTTP request processing. The alias is `max_http_sql_service_task_threads_num`. * Introduced in: 4.0.0 ##### `http_backlog_num`[​](#http_backlog_num "Direct link to http_backlog_num") * Default: 1024 * Type: Int * Unit: - * Is mutable: No * Description: The length of the backlog queue held by the HTTP server in the FE node. * Introduced in: - ##### `http_max_chunk_size`[​](#http_max_chunk_size "Direct link to http_max_chunk_size") * Default: 8192 * Type: Int * Unit: Bytes * Is mutable: No * Description: Sets the maximum allowed size (in bytes) of a single HTTP chunk handled by Netty's HttpServerCodec in the FE HTTP server. It is passed as the third argument to HttpServerCodec and limits the length of chunks during chunked transfer or streaming requests/responses. If an incoming chunk exceeds this value, Netty will raise a frame-too-large error (e.g., TooLongFrameException) and the request may be rejected. Increase this for legitimate large chunked uploads; keep it small to reduce memory pressure and surface area for DoS attacks. This setting is used alongside `http_max_initial_line_length`, `http_max_header_size`, and `enable_http_validate_headers`. * Introduced in: v3.2.0 ##### `http_max_header_size`[​](#http_max_header_size "Direct link to http_max_header_size") * Default: 32768 * Type: Int * Unit: Bytes * Is mutable: No * Description: Maximum allowed size in bytes for the HTTP request header block parsed by Netty's `HttpServerCodec`. StarRocks passes this value to `HttpServerCodec` (as `Config.http_max_header_size`); if an incoming request's headers (names and values combined) exceed this limit, the codec will reject the request (decoder exception) and the connection/request will fail. Increase only when clients legitimately send very large headers (large cookies or many custom headers); larger values increase per-connection memory use. Tune in conjunction with `http_max_initial_line_length` and `http_max_chunk_size`. Changes require FE restart. * Introduced in: v3.2.0 ##### `http_max_initial_line_length`[​](#http_max_initial_line_length "Direct link to http_max_initial_line_length") * Default: 4096 * Type: Int * Unit: Bytes * Is mutable: No * Description: Sets the maximum allowed length (in bytes) of the HTTP initial request line (method + request-target + HTTP version) accepted by the Netty `HttpServerCodec` used in HttpServer. The value is passed to Netty's decoder and requests with an initial line longer than this will be rejected (TooLongFrameException). Increase this only when you must support very long request URIs; larger values increase memory use and may raise exposure to malformed/request-abuse. Tune together with `http_max_header_size` and `http_max_chunk_size`. * Introduced in: v3.2.0 ##### `http_port`[​](#http_port "Direct link to http_port") * Default: 8030 * Type: Int * Unit: - * Is mutable: No * Description: The port on which the HTTP server in the FE node listens. * Introduced in: - ##### `enable_http_auth`[​](#enable_http_auth "Direct link to enable_http_auth") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Introduced in: v4.2.0 * Description: When true, most external FE HTTP endpoints require HTTP Basic Auth. Credentials are validated against the user store via `AuthenticationHandler.authenticate()`, so LDAP / security-integration login works on the HTTP path the same way it does for the MySQL protocol. The following are exempt: * Public probes / observability: `/api/bootstrap`, `/api/oauth2`. * Peer-FE / control-plane paths that are IP-whitelisted or token-gated inside the handler: `/image`, `/check`, `/journal_id`, `/info`, `/role`, `/dump`, `/dump_starmgr`, `/service_id`, `/static`, `/api/_meta_replay_state`, `/api/get_small_file`. Privileged endpoints additionally require a SYSTEM-level RBAC privilege (`OPERATE` or `NODE`) that is **active** in the caller's session. If the granting role is not the user's default, run `SET DEFAULT ROLE TO ;` or set the global variable `activate_all_roles_on_login=true` so the roles activate at login. LDAP / security-integration group → role mappings activate automatically. ##### `http_web_page_display_hardware`[​](#http_web_page_display_hardware "Direct link to http_web_page_display_hardware") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: When true, the HTTP index page (/index) will include a hardware information section populated via the oshi library (CPU, memory, processes, disks, filesystems, network, etc.). oshi may invoke system utilities or read system files indirectly (for example, it can execute commands such as `getent passwd`), which can surface sensitive system data. If you require stricter security or want to avoid executing those indirect commands on the host, set this configuration to false to disable collection and display of hardware details on the web UI. * Introduced in: v3.2.0 ##### `http_worker_threads_num`[​](#http_worker_threads_num "Direct link to http_worker_threads_num") * Default: 0 * Type: Int * Unit: - * Is mutable: No * Description: Number of worker threads for http server to deal with http requests. For a negative or 0 value, the number of threads will be twice the number of cpu cores. * Introduced in: v2.5.18, v3.0.10, v3.1.7, v3.2.2 ##### `https_port`[​](#https_port "Direct link to https_port") * Default: 8443 * Type: Int * Unit: - * Is mutable: No * Description: The port on which the HTTPS server in the FE node listens. * Introduced in: v4.0 ##### `max_mysql_service_task_threads_num`[​](#max_mysql_service_task_threads_num "Direct link to max_mysql_service_task_threads_num") * Default: 4096 * Type: Int * Unit: - * Is mutable: No * Description: The maximum number of threads that can be run by the MySQL server in the FE node to process tasks. * Introduced in: - ##### `max_task_runs_threads_num`[​](#max_task_runs_threads_num "Direct link to max_task_runs_threads_num") * Default: 512 * Type: Int * Unit: Threads * Is mutable: No * Description: Controls the maximum number of threads in the task-run executor thread pool. This value is the upper bound of concurrent task-run executions; increasing it raises parallelism but also increases CPU, memory, and network usage, while reducing it can cause task-run backlog and higher latency. Tune this value according to expected concurrent scheduled jobs and available system resources. * Introduced in: v3.2.0 ##### `memory_tracker_enable`[​](#memory_tracker_enable "Direct link to memory_tracker_enable") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Enables the FE memory tracker subsystem. When `memory_tracker_enable` is set to `true`, `MemoryUsageTracker` periodically scans registered metadata modules, updates the in-memory `MemoryUsageTracker.MEMORY_USAGE` map, logs totals, and causes `MetricRepo` to expose memory usage and object-count gauges in metrics output. Use `memory_tracker_interval_seconds` to control the sampling interval. Enabling this feature helps monitoring and debugging memory consumption but introduces CPU and I/O overhead and additional metric cardinality. * Introduced in: v3.2.4 ##### `memory_tracker_interval_seconds`[​](#memory_tracker_interval_seconds "Direct link to memory_tracker_interval_seconds") * Default: 60 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Interval in seconds for the FE `MemoryUsageTracker` daemon to poll and record memory usage of the FE process and registered `MemoryTrackable` modules. When `memory_tracker_enable` is set to `true`, the tracker runs on this cadence, updates `MEMORY_USAGE`, and logs aggregated JVM and tracked-module usage. * Introduced in: v3.2.4 ##### `mysql_nio_backlog_num`[​](#mysql_nio_backlog_num "Direct link to mysql_nio_backlog_num") * Default: 1024 * Type: Int * Unit: - * Is mutable: No * Description: The length of the backlog queue held by the MySQL server in the FE node. * Introduced in: - ##### `mysql_send_packet_timeout_ms`[​](#mysql_send_packet_timeout_ms "Direct link to mysql_send_packet_timeout_ms") * Default: 60000 * Type: Long * Unit: Milliseconds * Is mutable: Yes * Description: Per-packet write timeout for the MySQL protocol channel. Bounds how long the FE worker can wait for a slow client's TCP recv buffer to drain when sending result rows. Without this bound the worker can block in `Selector.select()` indefinitely and the query becomes unkillable via `KILL QUERY`. Set to `0` to disable (legacy unbounded wait). * Introduced in: v4.1 ##### `mysql_server_version`[​](#mysql_server_version "Direct link to mysql_server_version") * Default: 8.0.33 * Type: String * Unit: - * Is mutable: Yes * Description: The MySQL server version returned to the client. Modifying this parameter will affect the version information in the following situations: 1. `select version();` 2. Handshake packet version 3. Value of the global variable `version` (`show variables like 'version';`) * Introduced in: - ##### `mysql_service_io_threads_num`[​](#mysql_service_io_threads_num "Direct link to mysql_service_io_threads_num") * Default: 4 * Type: Int * Unit: - * Is mutable: No * Description: The maximum number of threads that can be run by the MySQL server in the FE node to process I/O events. * Introduced in: - ##### `mysql_service_kill_after_disconnect`[​](#mysql_service_kill_after_disconnect "Direct link to mysql_service_kill_after_disconnect") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: Controls how the server handles the session when the MySQL TCP connection is detected closed (EOF on read). If it is set to `true`, the server immediately kills any running query for that connection and performs immediate cleanup. If it is `false`, the server does not kill running queries on disconnection and only performs cleanup when there are no pending request tasks, allowing long-running queries to continue after client disconnects. Note: despite a brief comment suggesting TCP keep‑alive, this parameter specifically governs post-disconnection killing behavior and should be set according to whether you want orphaned queries terminated (recommended behind unreliable/load‑balanced clients) or allowed to finish. * Introduced in: - ##### `mysql_service_nio_enable_keep_alive`[​](#mysql_service_nio_enable_keep_alive "Direct link to mysql_service_nio_enable_keep_alive") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: Enable TCP Keep-Alive for MySQL connections. Useful for long-idled connections behind load balancers. * Introduced in: - ##### `net_use_ipv6_when_priority_networks_empty`[​](#net_use_ipv6_when_priority_networks_empty "Direct link to net_use_ipv6_when_priority_networks_empty") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: A boolean value to control whether to use IPv6 addresses preferentially when `priority_networks` is not specified. `true` indicates to allow the system to use an IPv6 address preferentially when the server that hosts the node has both IPv4 and IPv6 addresses and `priority_networks` is not specified. * Introduced in: v3.3.0 ##### `priority_networks`[​](#priority_networks "Direct link to priority_networks") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: Declares a selection strategy for servers that have multiple IP addresses. Note that at most one IP address must match the list specified by this parameter. The value of this parameter is a list that consists of entries, which are separated with semicolons (;) in CIDR notation, such as 10.10.10.0/24. If no IP address matches the entries in this list, an available IP address of the server will be randomly selected. From v3.3.0, StarRocks supports deployment based on IPv6. If the server has both IPv4 and IPv6 addresses, and this parameter is not specified, the system uses an IPv4 address by default. You can change this behavior by setting `net_use_ipv6_when_priority_networks_empty` to `true`. * Introduced in: - ##### `proc_profile_cpu_enable`[​](#proc_profile_cpu_enable "Direct link to proc_profile_cpu_enable") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: When this item is set to `true`, the background `ProcProfileCollector` will collect CPU profiles using `AsyncProfiler` and write HTML reports under `sys_log_dir/proc_profile`. Each collection run records CPU stacks for the duration configured by `proc_profile_collect_time_s` and uses `proc_profile_jstack_depth` for Java stack depth. Generated profiles are compressed and old files are pruned according to `proc_profile_file_retained_days` and `proc_profile_file_retained_size_bytes`. `AsyncProfiler` requires the native library (`libasyncProfiler.so`); `one.profiler.extractPath` is set to `STARROCKS_HOME_DIR/bin` to avoid noexec issues on `/tmp`. * Introduced in: v3.2.12 ##### `qe_max_connection`[​](#qe_max_connection "Direct link to qe_max_connection") * Default: 4096 * Type: Int * Unit: - * Is mutable: No * Description: The maximum number of connections that can be established by all users to the FE node. From v3.1.12 and v3.2.7 onwards, the default value has been changed from `1024` to `4096`. * Introduced in: - ##### `query_port`[​](#query_port "Direct link to query_port") * Default: 9030 * Type: Int * Unit: - * Is mutable: No * Description: The port on which the MySQL server in the FE node listens. * Introduced in: - ##### `rpc_port`[​](#rpc_port "Direct link to rpc_port") * Default: 9020 * Type: Int * Unit: - * Is mutable: No * Description: The port on which the Thrift server in the FE node listens. * Introduced in: - ##### `slow_lock_stack_trace_reserve_levels`[​](#slow_lock_stack_trace_reserve_levels "Direct link to slow_lock_stack_trace_reserve_levels") * Default: 15 * Type: Int * Unit: - * Is mutable: Yes * Description: Controls how many stack-trace frames are captured and emitted when StarRocks dumps lock debug information for slow or held locks. This value is passed to `LogUtil.getStackTraceToJsonArray` by `QueryableReentrantReadWriteLock` when producing JSON for the exclusive lock owner, current thread, and oldest/shared readers. Increasing this value provides more context for diagnosing slow-lock or deadlock issues at the cost of larger JSON payloads and slightly higher CPU/memory for stack capture; decreasing it reduces overhead. Note: this cap applies only to the `QueryableReentrantReadWriteLock` stack-dump path; the `LockManager` slow-lock path captures full stack depth and is not bounded by this value. Reader entries can be filtered by `slow_lock_threshold_ms` when only logging slow locks. * Introduced in: v3.4.0, v3.5.0 ##### `ssl_cipher_blacklist`[​](#ssl_cipher_blacklist "Direct link to ssl_cipher_blacklist") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: Comma separated list, with regex support to blacklist ssl cipher suites by IANA names. If both whitelist and blacklist are set, blacklist takes precedence. * Introduced in: v4.0 ##### `ssl_cipher_whitelist`[​](#ssl_cipher_whitelist "Direct link to ssl_cipher_whitelist") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: Comma separated list, with regex support to whitelist ssl cipher suites by IANA names. If both whitelist and blacklist are set, blacklist takes precedence. * Introduced in: v4.0 ##### `task_runs_concurrency`[​](#task_runs_concurrency "Direct link to task_runs_concurrency") * Default: 4 * Type: Int * Unit: - * Is mutable: Yes * Description: Global limit of concurrently running TaskRun instances. `TaskRunScheduler` stops scheduling new runs when current running count is greater than or equal to `task_runs_concurrency`, so this value caps parallel TaskRun execution across the scheduler. It is also used by `MVPCTRefreshPartitioner` to compute per-TaskRun partition refresh granularity. Increasing the value raises parallelism and resource usage; decreasing it reduces concurrency and makes partition refreshes larger per run. Do not set to 0 or negative unless intentionally disabling scheduling: 0 (or negative) will effectively prevent new TaskRuns from being scheduled by `TaskRunScheduler`. * Introduced in: v3.2.0 ##### `task_runs_queue_length`[​](#task_runs_queue_length "Direct link to task_runs_queue_length") * Default: 500 * Type: Int * Unit: - * Is mutable: Yes * Description: Limits the maximum number of pending TaskRun items kept in the pending queue. `TaskRunManager` checks the current pending count and rejects new submissions when valid pending TaskRun count is greater than or equal to `task_runs_queue_length`. The same limit is rechecked before merged/accepted TaskRuns are added. Tune this value to balance memory and scheduling backlog: set higher for large bursty workloads to avoid rejects, or lower to bound memory and reduce pending backlog. * Introduced in: v3.2.0 ##### `thrift_backlog_num`[​](#thrift_backlog_num "Direct link to thrift_backlog_num") * Default: 1024 * Type: Int * Unit: - * Is mutable: No * Description: The length of the backlog queue held by the Thrift server in the FE node. * Introduced in: - ##### `thrift_client_timeout_ms`[​](#thrift_client_timeout_ms "Direct link to thrift_client_timeout_ms") * Default: 5000 * Type: Int * Unit: Milliseconds * Is mutable: No * Description: The length of time after which idle client connections time out. * Introduced in: - ##### `thrift_rpc_max_body_size`[​](#thrift_rpc_max_body_size "Direct link to thrift_rpc_max_body_size") * Default: -1 * Type: Int * Unit: Bytes * Is mutable: No * Description: Controls the maximum allowed Thrift RPC message body size (in bytes) used when constructing the server's Thrift protocol (passed to TBinaryProtocol.Factory in `ThriftServer`). A value of `-1` disables the limit (unbounded). Setting a positive value enforces an upper bound so that messages larger than this are rejected by the Thrift layer, which helps limit memory usage and mitigate oversized-request or DoS risks. Set this to a size large enough for expected payloads (large structs or batched data) to avoid rejecting legitimate requests. * Introduced in: v3.2.0 ##### `thrift_server_max_worker_threads`[​](#thrift_server_max_worker_threads "Direct link to thrift_server_max_worker_threads") * Default: 4096 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of worker threads that are supported by the Thrift server in the FE node. * Introduced in: - ##### `thrift_server_queue_size`[​](#thrift_server_queue_size "Direct link to thrift_server_queue_size") * Default: 4096 * Type: Int * Unit: - * Is mutable: No * Description: The length of queue where requests are pending. If the number of threads that are being processed in the thrift server exceeds the value specified in `thrift_server_max_worker_threads`, new requests are added to the pending queue. * Introduced in: - #### Metadata and cluster management[​](#metadata-and-cluster-management "Direct link to Metadata and cluster management") ##### `alter_max_worker_queue_size`[​](#alter_max_worker_queue_size "Direct link to alter_max_worker_queue_size") * Default: 4096 * Type: Int * Unit: Tasks * Is mutable: No * Description: Controls the capacity of the internal worker thread pool queue used by the alter subsystem. It is passed to `ThreadPoolManager.newDaemonCacheThreadPool` in `AlterHandler` together with `alter_max_worker_threads`. When the number of pending alter tasks exceeds `alter_max_worker_queue_size`, new submissions will be rejected and a `RejectedExecutionException` can be thrown (see `AlterHandler.handleFinishAlterTask`). Tune this value to balance memory usage and the amount of backlog you permit for concurrent alter tasks. * Introduced in: v3.2.0 ##### `alter_max_worker_threads`[​](#alter_max_worker_threads "Direct link to alter_max_worker_threads") * Default: 4 * Type: Int * Unit: Threads * Is mutable: No * Description: Sets the maximum number of worker threads in the AlterHandler's thread pool. The AlterHandler constructs the executor with this value to run and finalize alter-related tasks (e.g., submitting `AlterReplicaTask` via handleFinishAlterTask). This value bounds concurrent execution of alter operations; raising it increases parallelism and resource usage, lowering it limits concurrent alters and may become a bottleneck. The executor is created together with `alter_max_worker_queue_size`, and the handler scheduling uses `alter_scheduler_interval_millisecond`. * Introduced in: v3.2.0 ##### `automated_cluster_snapshot_interval_seconds`[​](#automated_cluster_snapshot_interval_seconds "Direct link to automated_cluster_snapshot_interval_seconds") * Default: 600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The interval at which the Automated Cluster Snapshot tasks are triggered. * Introduced in: v3.4.2 ##### `background_refresh_metadata_interval_millis`[​](#background_refresh_metadata_interval_millis "Direct link to background_refresh_metadata_interval_millis") * Default: 600000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: The interval between two consecutive Hive metadata cache refreshes. * Introduced in: v2.5.5 ##### `background_refresh_metadata_time_secs_since_last_access_secs`[​](#background_refresh_metadata_time_secs_since_last_access_secs "Direct link to background_refresh_metadata_time_secs_since_last_access_secs") * Default: 3600 \* 24 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The expiration time of a Hive metadata cache refresh task. For the Hive catalog that has been accessed, if it has not been accessed for more than the specified time, StarRocks stops refreshing its cached metadata. For the Hive catalog that has not been accessed, StarRocks will not refresh its cached metadata. * Introduced in: v2.5.5 ##### `bdbje_cleaner_threads`[​](#bdbje_cleaner_threads "Direct link to bdbje_cleaner_threads") * Default: 1 * Type: Int * Unit: - * Is mutable: No * Description: Number of background cleaner threads for the Berkeley DB Java Edition (JE) environment used by StarRocks journal. This value is read during environment initialization in `BDBEnvironment.initConfigs` and applied to `EnvironmentConfig.CLEANER_THREADS` using `Config.bdbje_cleaner_threads`. It controls parallelism for JE log cleaning and space reclamation; increasing it can speed up cleaning at the cost of additional CPU and I/O interference with foreground operations. Changes take effect only when the BDB environment is (re)initialized, so a frontend restart is required to apply a new value. * Introduced in: v3.2.0 ##### `bdbje_heartbeat_timeout_second`[​](#bdbje_heartbeat_timeout_second "Direct link to bdbje_heartbeat_timeout_second") * Default: 30 * Type: Int * Unit: Seconds * Is mutable: No * Description: The amount of time after which the heartbeats among the leader, follower, and observer FEs in the StarRocks cluster time out. * Introduced in: - ##### `bdbje_lock_timeout_second`[​](#bdbje_lock_timeout_second "Direct link to bdbje_lock_timeout_second") * Default: 1 * Type: Int * Unit: Seconds * Is mutable: No * Description: The amount of time after which a lock in the BDB JE-based FE times out. * Introduced in: - ##### `bdbje_replay_cost_percent`[​](#bdbje_replay_cost_percent "Direct link to bdbje_replay_cost_percent") * Default: 150 * Type: Int * Unit: Percent * Is mutable: No * Description: Sets the relative cost (as a percentage) of replaying transactions from a BDB JE log versus obtaining the same data via a network restore. The value is supplied to the underlying JE replication parameter `REPLAY_COST_PERCENT` and is typically `>100` to indicate that replay is usually more expensive than a network restore. When deciding whether to retain cleaned log files for potential replay, the system compares replay cost multiplied by log size against the cost of a network restore; files will be removed if network restore is judged more efficient. A value of 0 disables retention based on this cost comparison. Log files required for replicas within `REP_STREAM_TIMEOUT` or for any active replication are always retained. * Introduced in: v3.2.0 ##### `bdbje_replica_ack_timeout_second`[​](#bdbje_replica_ack_timeout_second "Direct link to bdbje_replica_ack_timeout_second") * Default: 10 * Type: Int * Unit: Seconds * Is mutable: No * Description: The maximum amount of time for which the leader FE can wait for ACK messages from a specified number of follower FEs when metadata is written from the leader FE to the follower FEs. Unit: second. If a large amount of metadata is being written, the follower FEs require a long time before they can return ACK messages to the leader FE, causing ACK timeout. In this situation, metadata writes fail, and the FE process exits. We recommend that you increase the value of this parameter to prevent this situation. * Introduced in: - ##### `bdbje_reserved_disk_size`[​](#bdbje_reserved_disk_size "Direct link to bdbje_reserved_disk_size") * Default: 512 \* 1024 \* 1024 (536870912) * Type: Long * Unit: Bytes * Is mutable: No * Description: Limits the number of bytes Berkeley DB JE will reserve as "unprotected" (deletable) log/data files. StarRocks passes this value to JE via `EnvironmentConfig.RESERVED_DISK` in BDBEnvironment; JE's built-in default is 0 (unlimited). The StarRocks default (512 MiB) prevents JE from reserving excessive disk space for unprotected files while allowing safe cleanup of obsolete files. Tune this value on disk-constrained systems: decreasing it lets JE free more files sooner, increasing it lets JE retain more reserved space. Changes require restarting the process to take effect. * Introduced in: v3.2.0 ##### `bdbje_reset_election_group`[​](#bdbje_reset_election_group "Direct link to bdbje_reset_election_group") * Default: false * Type: String * Unit: - * Is mutable: No * Description: Whether to reset the BDBJE replication group. If this parameter is set to `TRUE`, the FE will reset the BDBJE replication group (that is, remove the information of all electable FE nodes) and start as the leader FE. After the reset, this FE will be the only member in the cluster, and other FEs can rejoin this cluster by using `ALTER SYSTEM ADD/DROP FOLLOWER/OBSERVER 'xxx'`. Use this setting only when no leader FE can be elected because the data of most follower FEs have been damaged. `reset_election_group` is used to replace `metadata_failure_recovery`. * Introduced in: - ##### `black_host_connect_failures_within_time`[​](#black_host_connect_failures_within_time "Direct link to black_host_connect_failures_within_time") * Default: 5 * Type: Int * Unit: - * Is mutable: Yes * Description: The threshold of connection failures allowed for a blacklisted BE node. If a BE node is added to the BE Blacklist automatically, StarRocks will assess its connectivity and judge whether it can be removed from the BE Blacklist. Within `black_host_history_sec`, only if a blacklisted BE node has fewer connection failures than the threshold set in `black_host_connect_failures_within_time`, it can be removed from the BE Blacklist. * Introduced in: v3.3.0 ##### `black_host_history_sec`[​](#black_host_history_sec "Direct link to black_host_history_sec") * Default: 2 \* 60 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time duration for retaining historical connection failures of BE nodes in the BE Blacklist. If a BE node is added to the BE Blacklist automatically, StarRocks will assess its connectivity and judge whether it can be removed from the BE Blacklist. Within `black_host_history_sec`, only if a blacklisted BE node has fewer connection failures than the threshold set in `black_host_connect_failures_within_time`, it can be removed from the BE Blacklist. * Introduced in: v3.3.0 ##### `brpc_connection_pool_size`[​](#brpc_connection_pool_size "Direct link to brpc_connection_pool_size") * Default: 16 * Type: Int * Unit: Connections * Is mutable: No * Description: The maximum number of pooled BRPC connections per endpoint used by the FE's BrpcProxy. This value is applied to RpcClientOptions via `setMaxTotoal` and `setMaxIdleSize`, so it directly limits concurrent outgoing BRPC requests because each request must borrow a connection from the pool. In high concurrency scenarios increase this to avoid request queuing; increasing it raises socket and memory usage and may increase remote server load. When tuning, consider related settings such as `brpc_idle_wait_max_time`, `brpc_short_connection`, `brpc_inner_reuse_pool`, `brpc_reuse_addr`, and `brpc_min_evictable_idle_time_ms`. Changing this value is not hot-reloadable and requires a restart. * Introduced in: v3.2.0 ##### `brpc_short_connection`[​](#brpc_short_connection "Direct link to brpc_short_connection") * Default: false * Type: boolean * Unit: - * Is mutable: No * Description: Controls whether the underlying brpc RpcClient uses short-lived connections. When enabled (`true`), RpcClientOptions.setShortConnection is set and connections are closed after a request completes, reducing the number of long-lived sockets at the cost of higher connection setup overhead and increased latency. When disabled (`false`, the default) persistent connections and connection pooling are used. Enabling this option affects connection-pool behavior and should be considered together with `brpc_connection_pool_size`, `brpc_idle_wait_max_time`, `brpc_min_evictable_idle_time_ms`, `brpc_reuse_addr`, and `brpc_inner_reuse_pool`. Keep it disabled for typical high-throughput deployments; enable only to limit socket lifetime or when short connections are required by network policy. * Introduced in: v3.3.11, v3.4.1, v3.5.0 ##### `catalog_try_lock_timeout_ms`[​](#catalog_try_lock_timeout_ms "Direct link to catalog_try_lock_timeout_ms") * Default: 5000 * Type: Long * Unit: Milliseconds * Is mutable: Yes * Description: The timeout duration to obtain the global lock. * Introduced in: - ##### `checkpoint_only_on_leader`[​](#checkpoint_only_on_leader "Direct link to checkpoint_only_on_leader") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: When `true`, the CheckpointController will only select the leader FE as the checkpoint worker; when `false`, the controller may pick any frontend and prefers nodes with lower heap usage. With `false`, workers are sorted by recent failure time and `heapUsedPercent` (the leader is treated as having infinite usage to avoid selecting it). For operations that require cluster snapshot metadata, the controller already forces leader selection regardless of this flag. Enabling `true` centralizes checkpoint work on the leader (simpler but increases leader CPU/memory and network load); keeping it `false` distributes checkpoint load to less-loaded FEs. This setting affects worker selection and interaction with timeouts such as `checkpoint_timeout_seconds` and RPC settings like `thrift_rpc_timeout_ms`. * Introduced in: v3.4.0, v3.5.0 ##### `checkpoint_timeout_seconds`[​](#checkpoint_timeout_seconds "Direct link to checkpoint_timeout_seconds") * Default: 24 \* 3600 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: Maximum time (in seconds) the leader's CheckpointController will wait for a checkpoint worker to complete a checkpoint. The controller converts this value to nanoseconds and polls the worker result queue; if no successful completion is received within this timeout the checkpoint is treated as failed and createImage returns failure. Increasing this value accommodates longer-running checkpoints but delays failure detection and subsequent image propagation; decreasing it causes faster failover/retries but can produce false timeouts for slow workers. This setting only controls the waiting period in `CheckpointController` during checkpoint creation and does not change the worker's internal checkpointing behavior. * Introduced in: v3.4.0, v3.5.0 ##### `db_used_data_quota_update_interval_secs`[​](#db_used_data_quota_update_interval_secs "Direct link to db_used_data_quota_update_interval_secs") * Default: 300 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The interval at which the database used data quota is updated. StarRocks periodically updates the used data quota for all databases to track storage consumption. This value is used for quota enforcement and metrics collection. The minimum allowed interval is 30 seconds to prevent excessive system load. A value less than 30 will be rejected. * Introduced in: - ##### `drop_backend_after_decommission`[​](#drop_backend_after_decommission "Direct link to drop_backend_after_decommission") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to delete a BE after the BE is decommissioned. `TRUE` indicates that the BE is deleted immediately after it is decommissioned. `FALSE` indicates that the BE is not deleted after it is decommissioned. * Introduced in: - ##### `edit_log_port`[​](#edit_log_port "Direct link to edit_log_port") * Default: 9010 * Type: Int * Unit: - * Is mutable: No * Description: The port that is used for communication among the Leader, Follower, and Observer FEs in the cluster. * Introduced in: - ##### `edit_log_roll_num`[​](#edit_log_roll_num "Direct link to edit_log_roll_num") * Default: 50000 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of metadata log entries that can be written before a log file is created for these log entries. This parameter is used to control the size of log files. The new log file is written to the BDBJE database. * Introduced in: - ##### `edit_log_type`[​](#edit_log_type "Direct link to edit_log_type") * Default: BDB * Type: String * Unit: - * Is mutable: No * Description: The type of edit log that can be generated. Set the value to `BDB`. * Introduced in: - ##### `enable_background_refresh_connector_metadata`[​](#enable_background_refresh_connector_metadata "Direct link to enable_background_refresh_connector_metadata") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable the periodic Hive metadata cache refresh. After it is enabled, StarRocks polls the metastore (Hive Metastore or AWS Glue) of your Hive cluster, and refreshes the cached metadata of the frequently accessed Hive catalogs to perceive data changes. `true` indicates to enable the Hive metadata cache refresh, and `false` indicates to disable it. * Introduced in: v2.5.5 ##### `refresh_other_fe_dispatch_executor_thread_num`[​](#refresh_other_fe_dispatch_executor_thread_num "Direct link to refresh_other_fe_dispatch_executor_thread_num") * Default: 4 * Type: Integer * Unit: - * Is mutable: Yes * Description: The number of threads in the FE-global dispatch executor for asynchronous "refresh other FE" jobs. These threads only schedule background refresh tasks from connector write paths. They do not send peer FE refresh RPCs directly. Changes take effect on running FEs without restart. * Introduced in: - ##### `refresh_other_fe_rpc_executor_thread_num`[​](#refresh_other_fe_rpc_executor_thread_num "Direct link to refresh_other_fe_rpc_executor_thread_num") * Default: 4 * Type: Integer * Unit: - * Is mutable: Yes * Description: The number of threads in the FE-global RPC executor for "refresh other FE" fan-out. This executor bounds the number of concurrent refresh RPCs sent to peer FEs for both synchronous and asynchronous external table refresh flows. Changes take effect on running FEs without restart. * Introduced in: - ##### `enable_collect_query_detail_info`[​](#enable_collect_query_detail_info "Direct link to enable_collect_query_detail_info") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to collect the profile of a query. If this parameter is set to `TRUE`, the system collects the profile of the query. If this parameter is set to `FALSE`, the system does not collect the profile of the query. * Introduced in: - ##### `enable_create_partial_partition_in_batch`[​](#enable_create_partial_partition_in_batch "Direct link to enable_create_partial_partition_in_batch") * Default: false * Type: boolean * Unit: - * Is mutable: Yes * Description: When this item is set to `false` (default), StarRocks enforces that batch-created range partitions align to the standard time unit boundaries. It will reject non‑aligned ranges to avoid creating holes. Setting this item to `true` disables that alignment check and allows creating partial (non‑standard) partitions in batch, which can produce gaps or misaligned partition ranges. You should only set it to `true` when you intentionally need partial batch partitions and accept the associated risks. * Introduced in: v3.2.0 ##### `enable_internal_sql`[​](#enable_internal_sql "Direct link to enable_internal_sql") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: When this item is set to `true`, internal SQL statements executed by internal components (for example, SimpleExecutor) are preserved and written into internal audit or log messages (and can be further desensitized if `enable_sql_desensitize_in_log` is set). When it is set to `false`, internal SQL text is suppressed: formatting code (SimpleExecutor.formatSQL) returns "?" and the actual statement is not emitted to internal audit or log messages. This configuration does not change execution semantics of internal statements — it only controls logging and visibility of internal SQL for privacy or security. * Introduced in: - ##### `enable_legacy_compatibility_for_replication`[​](#enable_legacy_compatibility_for_replication "Direct link to enable_legacy_compatibility_for_replication") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable the Legacy Compatibility for Replication. StarRocks may behave differently between the old and new versions, causing problems during cross-cluster data migration. Therefore, you must enable Legacy Compatibility for the target cluster before data migration and disable it after data migration is completed. `true` indicates enabling this mode. * Introduced in: v3.1.10, v3.2.6 ##### `enable_show_materialized_views_include_all_task_runs`[​](#enable_show_materialized_views_include_all_task_runs "Direct link to enable_show_materialized_views_include_all_task_runs") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Controls how TaskRuns are returned to the SHOW MATERIALIZED VIEWS command. When this item is set to `false`, StarRocks returns only the newest TaskRun per task (legacy behavior for compatibility). When it is set to `true` (default), `TaskManager` may include additional TaskRuns for the same task only when they share the same start TaskRun ID (for example, belong to the same job), preventing unrelated duplicate runs from appearing while allowing multiple statuses tied to one job to be shown. Set this item to `false` to restore single-run output or to surface multi-run job history for debugging and monitoring. * Introduced in: v3.3.0, v3.4.0, v3.5.0 ##### `enable_statistics_collect_profile`[​](#enable_statistics_collect_profile "Direct link to enable_statistics_collect_profile") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to generate profiles for statistics queries. You can set this item to `true` to allow StarRocks to generate query profiles for queries on system statistics. * Introduced in: v3.1.5 ##### `enable_table_name_case_insensitive`[​](#enable_table_name_case_insensitive "Direct link to enable_table_name_case_insensitive") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to enable case-insensitive processing on catalog names, database names, table names, view names, and materialized view names. By default, this feature is disabled and these names are case-sensitive. When enabled, StarRocks stores these names in lowercase and forcibly converts every such name to lowercase during **both query and write (DDL/DML) processing**. This feature can be enabled only when creating a cluster. **We strongly recommend that you keep it disabled unless you have a specific, well-understood reason to enable it**, for the following reasons: * **It can make external tables and external catalogs unusable.** Different external catalog services follow different naming and case-sensitivity conventions. If an external schema, database, or table name is not already in lowercase, StarRocks lowercases the name in your SQL before passing it to the connector and then looks up a name that does not exist in the source, so the query fails with a "not found" error. * **It cannot be changed after the cluster is created.** After the cluster is started, the value cannot be modified by any means; any attempt to modify it results in an error, and FE fails to start if the value is inconsistent with the value used when the cluster was first started. * Only enable this feature on a new cluster where you are certain that all object names — including those in every external data source you plan to access — are already in lowercase. * Introduced in: v4.0 ##### `enable_task_history_archive`[​](#enable_task_history_archive "Direct link to enable_task_history_archive") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: When enabled, finished task-run records are archived to the persistent task-run history table and recorded to the edit log so lookups (e.g., `lookupHistory`, `lookupHistoryByTaskNames`, `lookupLastJobOfTasks`) include archived results. Archiving is performed by the FE leader and is skipped during unit tests (`FeConstants.runningUnitTest`). When enabled, in-memory expiration and forced-GC paths are bypassed (the code returns early from `removeExpiredRuns` and `forceGC`), so retention/eviction is handled by the persistent archive instead of `task_runs_ttl_second` and `task_runs_max_history_number`. When disabled, history stays in memory and is pruned by those configurations. * Introduced in: v3.3.1, v3.4.0, v3.5.0 ##### `enable_task_run_fe_evaluation`[​](#enable_task_run_fe_evaluation "Direct link to enable_task_run_fe_evaluation") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: When enabled, the FE will perform local evaluation for the system table `task_runs` in `TaskRunsSystemTable.supportFeEvaluation`. FE-side evaluation is only allowed for conjunctive equality predicates comparing a column to a constant and is limited to the columns `QUERY_ID` and `TASK_NAME`. Enabling this improves performance for targeted lookups by avoiding broader scans or additional remote processing; disabling it forces the planner to skip FE evaluation for `task_runs`, which may reduce predicate pruning and affect query latency for those filters. * Introduced in: v3.3.13, v3.4.3, v3.5.0 ##### `heartbeat_mgr_blocking_queue_size`[​](#heartbeat_mgr_blocking_queue_size "Direct link to heartbeat_mgr_blocking_queue_size") * Default: 1024 * Type: Int * Unit: - * Is mutable: No * Description: The size of the blocking queue that stores heartbeat tasks run by the Heartbeat Manager. * Introduced in: - ##### `heartbeat_mgr_threads_num`[​](#heartbeat_mgr_threads_num "Direct link to heartbeat_mgr_threads_num") * Default: 8 * Type: Int * Unit: - * Is mutable: No * Description: The number of threads that can be run by the Heartbeat Manager to run heartbeat tasks. * Introduced in: - ##### `ignore_materialized_view_error`[​](#ignore_materialized_view_error "Direct link to ignore_materialized_view_error") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether FE ignores the metadata exception caused by materialized view errors. If FE fails to start due to the metadata exception caused by materialized view errors, you can set this parameter to `true` to allow FE to ignore the exception. * Introduced in: v2.5.10 ##### `ignore_meta_check`[​](#ignore_meta_check "Direct link to ignore_meta_check") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether non-Leader FEs ignore the metadata gap from the Leader FE. If the value is TRUE, non-Leader FEs ignore the metadata gap from the Leader FE and continue providing data reading services. This parameter ensures continuous data reading services even when you stop the Leader FE for a long period of time. If the value is FALSE, non-Leader FEs do not ignore the metadata gap from the Leader FE and stop providing data reading services. * Introduced in: - ##### `ignore_task_run_history_replay_error`[​](#ignore_task_run_history_replay_error "Direct link to ignore_task_run_history_replay_error") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: When StarRocks deserializes TaskRun history rows for `information_schema.task_runs`, a corrupted or invalid JSON row will normally cause deserialization to log a warning and throw a RuntimeException. If this item is set to `true`, the system will catch deserialization errors, skip the malformed record, and continue processing remaining rows instead of failing the query. This will make `information_schema.task_runs` queries tolerant of bad entries in the `_statistics_.task_run_history` table. Note that enabling it will silently drop corrupted history records (potential data loss) instead of surfacing an explicit error. * Introduced in: v3.3.3, v3.4.0, v3.5.0 ##### `lock_checker_interval_second`[​](#lock_checker_interval_second "Direct link to lock_checker_interval_second") * Default: 30 * Type: long * Unit: Seconds * Is mutable: Yes * Description: Interval, in seconds, between executions of the LockChecker frontend daemon (named "deadlock-checker"). The daemon performs deadlock detection and slow-lock scanning; the configured value is multiplied by 1000 to set the timer in milliseconds. Decreasing this value reduces detection latency but increases scheduling and CPU overhead; increasing it reduces overhead but delays detection and slow-lock reporting. Changes take effect at runtime because the daemon resets its interval each run. This setting interacts with `lock_checker_enable_deadlock_check` (enables deadlock checks) and `slow_lock_threshold_ms` (defines what constitutes a slow lock). * Introduced in: v3.2.0 ##### `master_sync_policy`[​](#master_sync_policy "Direct link to master_sync_policy") * Default: SYNC * Type: String * Unit: - * Is mutable: No * Description: The policy based on which the leader FE flushes logs to disk. This parameter is valid only when the current FE is a leader FE. Valid values: * `SYNC`: When a transaction is committed, a log entry is generated and flushed to disk simultaneously. * `NO_SYNC`: The generation and flushing of a log entry do not occur at the same time when a transaction is committed. * `WRITE_NO_SYNC`: When a transaction is committed, a log entry is generated simultaneously but is not flushed to disk. If you have deployed only one follower FE, we recommend that you set this parameter to `SYNC`. If you have deployed three or more follower FEs, we recommend that you set this parameter and the `replica_sync_policy` both to `WRITE_NO_SYNC`. * Introduced in: - ##### `max_bdbje_clock_delta_ms`[​](#max_bdbje_clock_delta_ms "Direct link to max_bdbje_clock_delta_ms") * Default: 5000 * Type: Long * Unit: Milliseconds * Is mutable: No * Description: The maximum clock offset that is allowed between the leader FE and the follower or observer FEs in the StarRocks cluster. * Introduced in: - ##### `meta_delay_toleration_second`[​](#meta_delay_toleration_second "Direct link to meta_delay_toleration_second") * Default: 300 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The maximum duration by which the metadata on the follower and observer FEs can lag behind that on the leader FE. Unit: seconds. If this duration is exceeded, the non-leader FEs stops providing services. * Introduced in: - ##### `meta_dir`[​](#meta_dir "Direct link to meta_dir") * Default: `StarRocksFE.STARROCKS_HOME_DIR` + "/meta" * Type: String * Unit: - * Is mutable: No * Description: The directory that stores metadata. * Introduced in: - ##### `metadata_ignore_unknown_operation_type`[​](#metadata_ignore_unknown_operation_type "Direct link to metadata_ignore_unknown_operation_type") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to ignore an unknown log ID. When an FE is rolled back, the FEs of the earlier version may be unable to recognize some log IDs. If the value is `TRUE`, the FE ignores unknown log IDs. If the value is `FALSE`, the FE exits. * Introduced in: - ##### `profile_info_format`[​](#profile_info_format "Direct link to profile_info_format") * Default: default * Type: String * Unit: - * Is mutable: Yes * Description: The format of the Profile output by the system. Valid values: `default` and `json`. When set to `default`, Profile is of the default format. When set to `json`, the system outputs Profile in JSON format. * Introduced in: v2.5 ##### `replica_ack_policy`[​](#replica_ack_policy "Direct link to replica_ack_policy") * Default: `SIMPLE_MAJORITY` * Type: String * Unit: - * Is mutable: No * Description: The policy based on which a log entry is considered valid. The default value `SIMPLE_MAJORITY` specifies that a log entry is considered valid if a majority of follower FEs return ACK messages. * Introduced in: - ##### `replica_sync_policy`[​](#replica_sync_policy "Direct link to replica_sync_policy") * Default: SYNC * Type: String * Unit: - * Is mutable: No * Description: The policy based on which the follower FE flushes logs to disk. This parameter is valid only when the current FE is a follower FE. Valid values: * `SYNC`: When a transaction is committed, a log entry is generated and flushed to disk simultaneously. * `NO_SYNC`: The generation and flushing of a log entry do not occur at the same time when a transaction is committed. * `WRITE_NO_SYNC`: When a transaction is committed, a log entry is generated simultaneously but is not flushed to disk. * Introduced in: - ##### `start_with_incomplete_meta`[​](#start_with_incomplete_meta "Direct link to start_with_incomplete_meta") * Default: false * Type: boolean * Unit: - * Is mutable: No * Description: When true, the FE will allow startup when image data exists but Berkeley DB JE (BDB) log files are missing or corrupted. `MetaHelper.checkMetaDir()` uses this flag to bypass the safety check that otherwise prevents starting from an image without corresponding BDB logs; starting this way can produce stale or inconsistent metadata and should only be used for emergency recovery. `RestoreClusterSnapshotMgr` temporarily sets this flag to true while restoring a cluster snapshot and then rolls it back; that component also toggles `bdbje_reset_election_group` during restore. Do not enable in normal operation — enable only when recovering from corrupted BDB data or when explicitly restoring an image-based snapshot. * Introduced in: v3.2.0 ##### `table_keeper_interval_second`[​](#table_keeper_interval_second "Direct link to table_keeper_interval_second") * Default: 30 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Interval, in seconds, between executions of the TableKeeper daemon. The TableKeeperDaemon uses this value (multiplied by 1000) to set its internal timer and periodically runs keeper tasks that ensure history tables exist, correct table properties (replication number) and update partition TTLs. The daemon only performs work on the leader node and updates its runtime interval via setInterval when `table_keeper_interval_second` changes. Increase to reduce scheduling frequency and load; decrease for faster reaction to missing or stale history tables. * Introduced in: v3.3.1, v3.4.0, v3.5.0 ##### `task_runs_ttl_second`[​](#task_runs_ttl_second "Direct link to task_runs_ttl_second") * Default: 7 \* 24 \* 3600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Controls the time-to-live (TTL) for task run history. Lowering this value shortens history retention and reduces memory/disk usage; raising it keeps histories longer but increases resource usage. Adjust together with `task_runs_max_history_number` and `enable_task_history_archive` for predictable retention and storage behavior. * Introduced in: v3.2.0 ##### `task_ttl_second`[​](#task_ttl_second "Direct link to task_ttl_second") * Default: 24 \* 3600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Time-to-live (TTL) for tasks. For manual tasks (when no schedule is set), TaskBuilder uses this value to compute the task's `expireTime` (`expireTime = now + task_ttl_second * 1000L`). TaskRun also uses this value as an upper bound when computing a run's execute timeout — the effective execute timeout is `min(task_runs_timeout_second, task_runs_ttl_second, task_ttl_second)`. Adjusting this value changes how long manually created tasks remain valid and can indirectly limit the maximum allowed execution time of task runs. * Introduced in: v3.2.0 ##### `thrift_rpc_retry_times`[​](#thrift_rpc_retry_times "Direct link to thrift_rpc_retry_times") * Default: 3 * Type: Int * Unit: - * Is mutable: Yes * Description: Controls the total number of attempts a Thrift RPC call will make. This value is used by `ThriftRPCRequestExecutor` (and callers such as `NodeMgr` and `VariableMgr`) as the loop count for retries — i.e., a value of 3 allows up to three attempts including the initial try. On `TTransportException` the executor will try to reopen the connection and retry up to this count; it will not retry when the cause is a `SocketTimeoutException` or when reopen fails. Each attempt is subject to the per-attempt timeout configured by `thrift_rpc_timeout_ms`. Increasing this value improves resilience to transient connection failures but can increase overall RPC latency and resource usage. * Introduced in: v3.2.0 ##### `thrift_rpc_strict_mode`[​](#thrift_rpc_strict_mode "Direct link to thrift_rpc_strict_mode") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: Controls the TBinaryProtocol "strict read" mode used by the Thrift server. This value is passed as the first argument to org.apache.thrift.protocol.TBinaryProtocol.Factory in the Thrift server stack and affects how incoming Thrift messages are parsed and validated. When `true` (default), the server enforces strict Thrift encoding/version checks and honors the configured `thrift_rpc_max_body_size` limit; when `false`, the server accepts non-strict (legacy/lenient) message formats, which can improve compatibility with older clients but may bypass some protocol validations. Use caution changing this on a running cluster because it is not mutable and affects interoperability and parsing safety. * Introduced in: v3.2.0 ##### `thrift_rpc_timeout_ms`[​](#thrift_rpc_timeout_ms "Direct link to thrift_rpc_timeout_ms") * Default: 10000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: Timeout (in milliseconds) used as the default network/socket timeout for Thrift RPC calls. It is passed to TSocket when creating Thrift clients in `ThriftConnectionPool` (used by the frontend and backend pools) and is also added to an operation's execution timeout (e.g., ExecTimeout\*1000 + `thrift_rpc_timeout_ms`) when computing RPC call timeouts in places such as `ConfigBase`, `LeaderOpExecutor`, `GlobalStateMgr`, `NodeMgr`, `VariableMgr`, and `CheckpointWorker`. Increasing this value makes RPC calls tolerate longer network or remote processing delays; decreasing it causes faster failover on slow networks. Changing this value affects connection creation and request deadlines across the FE code paths that perform Thrift RPCs. * Introduced in: v3.2.0 ##### `txn_latency_metric_report_groups`[​](#txn_latency_metric_report_groups "Direct link to txn_latency_metric_report_groups") * Default: An empty string * Type: String * Unit: - * Is mutable: Yes * Description: A comma-separated list of transaction latency metric groups to report. Load types are categorized into logical groups for monitoring. When a group is enabled, its name is added as a 'type' label to transaction metrics. Valid values: `stream_load`, `routine_load`, `broker_load`, `insert`, and `compaction` (available only for shared-data clusters). Example: `"stream_load,routine_load"`. * Introduced in: v4.0 ##### `txn_rollback_limit`[​](#txn_rollback_limit "Direct link to txn_rollback_limit") * Default: 100 * Type: Int * Unit: - * Is mutable: No * Description: The maximum number of transactions that can be rolled back. * Introduced in: - --- ### FE Configuration - Shared-data, Data Lake, and Others FE parameters are classified into dynamic parameters and static parameters. * Dynamic parameters can be configured and adjusted by running SQL commands, which is very convenient. But the configurations become invalid if you restart your FE. Therefore, we recommend that you also modify the configuration items in the **fe.conf** file to prevent the loss of modifications. * Static parameters can only be configured and adjusted in the FE configuration file **fe.conf**. **After you modify this file, you must restart your FE for the changes to take effect.** Whether a parameter is a dynamic parameter is indicated by the `IsMutable` column in the output of [ADMIN SHOW CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SHOW_CONFIG.md). `TRUE` indicates a dynamic parameter. Note that both dynamic and static FE parameters can be configured in the **fe.conf** file. #### View FE configuration items[​](#view-fe-configuration-items "Direct link to View FE configuration items") After your FE is started, you can run the ADMIN SHOW FRONTEND CONFIG command on your MySQL client to check the parameter configurations. If you want to query the configuration of a specific parameter, run the following command: ```sql ADMIN SHOW FRONTEND CONFIG [LIKE "pattern"]; ``` For detailed description of the returned fields, see [`ADMIN SHOW CONFIG`](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SHOW_CONFIG.md). note You must have administrator privileges to run cluster administration-related commands. #### Configure FE parameters[​](#configure-fe-parameters "Direct link to Configure FE parameters") ##### Configure FE dynamic parameters[​](#configure-fe-dynamic-parameters "Direct link to Configure FE dynamic parameters") You can configure or modify the settings of FE dynamic parameters using [`ADMIN SET FRONTEND CONFIG`](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md). ```sql ADMIN SET FRONTEND CONFIG ("key" = "value"); ``` note The configuration changes made with `ADMIN SET FRONTEND` will be restored to the default values in the `fe.conf` file after the FE restarts. Therefore, we recommend that you also modify the configuration items in `fe.conf` if you want the changes to be permanent. ##### Configure FE static parameters[​](#configure-fe-static-parameters "Direct link to Configure FE static parameters") note Static parameters of an FE are set by changing them in the configuration file **fe.conf** and restarting the FE to allow the changes to take effect. *** This topic introduces the following types of FE configurations: * [Shared-data](#shared-data) * [Data Lake](#data-lake) * [Other](#other) #### Shared-data[​](#shared-data "Direct link to Shared-data") ##### `aws_s3_access_key`[​](#aws_s3_access_key "Direct link to aws_s3_access_key") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The Access Key ID used to access your S3 bucket. * Introduced in: v3.0 ##### `aws_s3_endpoint`[​](#aws_s3_endpoint "Direct link to aws_s3_endpoint") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The endpoint used to access your S3 bucket, for example, `https://s3.us-west-2.amazonaws.com`. * Introduced in: v3.0 ##### `aws_s3_external_id`[​](#aws_s3_external_id "Direct link to aws_s3_external_id") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The external ID of the AWS account that is used for cross-account access to your S3 bucket. * Introduced in: v3.0 ##### `aws_s3_iam_role_arn`[​](#aws_s3_iam_role_arn "Direct link to aws_s3_iam_role_arn") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The ARN of the IAM role that has privileges on your S3 bucket in which your data files are stored. * Introduced in: v3.0 ##### `aws_s3_path`[​](#aws_s3_path "Direct link to aws_s3_path") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The S3 path used to store data. It consists of the name of your S3 bucket and the sub-path (if any) under it, for example, `testbucket/subpath`. * Introduced in: v3.0 ##### `aws_s3_region`[​](#aws_s3_region "Direct link to aws_s3_region") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The region in which your S3 bucket resides, for example, `us-west-2`. * Introduced in: v3.0 ##### `aws_s3_secret_key`[​](#aws_s3_secret_key "Direct link to aws_s3_secret_key") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The Secret Access Key used to access your S3 bucket. * Introduced in: v3.0 ##### `aws_s3_use_aws_sdk_default_behavior`[​](#aws_s3_use_aws_sdk_default_behavior "Direct link to aws_s3_use_aws_sdk_default_behavior") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to use the default authentication credential of AWS SDK. Valid values: true and false (Default). * Introduced in: v3.0 ##### `aws_s3_use_instance_profile`[​](#aws_s3_use_instance_profile "Direct link to aws_s3_use_instance_profile") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to use Instance Profile and Assumed Role as credential methods for accessing S3. Valid values: true and false (Default). * If you use IAM user-based credential (Access Key and Secret Key) to access S3, you must specify this item as `false`, and specify `aws_s3_access_key` and `aws_s3_secret_key`. * If you use Instance Profile to access S3, you must specify this item as `true`. * If you use Assumed Role to access S3, you must specify this item as `true`, and specify `aws_s3_iam_role_arn`. * And if you use an external AWS account, you must also specify `aws_s3_external_id`. * Introduced in: v3.0 ##### `azure_adls2_endpoint`[​](#azure_adls2_endpoint "Direct link to azure_adls2_endpoint") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The endpoint of your Azure Data Lake Storage Gen2 Account, for example, `https://test.dfs.core.windows.net`. * Introduced in: v3.4.1 ##### `azure_adls2_oauth2_client_id`[​](#azure_adls2_oauth2_client_id "Direct link to azure_adls2_oauth2_client_id") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The Client ID of the Managed Identity used to authorize requests for your Azure Data Lake Storage Gen2. * Introduced in: v3.4.4 ##### `azure_adls2_oauth2_tenant_id`[​](#azure_adls2_oauth2_tenant_id "Direct link to azure_adls2_oauth2_tenant_id") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The Tenant ID of the Managed Identity used to authorize requests for your Azure Data Lake Storage Gen2. * Introduced in: v3.4.4 ##### `azure_adls2_oauth2_use_managed_identity`[​](#azure_adls2_oauth2_use_managed_identity "Direct link to azure_adls2_oauth2_use_managed_identity") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to use Managed Identity to authorize requests for your Azure Data Lake Storage Gen2. * Introduced in: v3.4.4 ##### `azure_adls2_path`[​](#azure_adls2_path "Direct link to azure_adls2_path") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The Azure Data Lake Storage Gen2 path used to store data. It consists of the file system name and the directory name, for example, `testfilesystem/starrocks`. * Introduced in: v3.4.1 ##### `azure_adls2_sas_token`[​](#azure_adls2_sas_token "Direct link to azure_adls2_sas_token") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The shared access signatures (SAS) used to authorize requests for your Azure Data Lake Storage Gen2. * Introduced in: v3.4.1 ##### `azure_adls2_shared_key`[​](#azure_adls2_shared_key "Direct link to azure_adls2_shared_key") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The Shared Key used to authorize requests for your Azure Data Lake Storage Gen2. * Introduced in: v3.4.1 ##### `azure_blob_endpoint`[​](#azure_blob_endpoint "Direct link to azure_blob_endpoint") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The endpoint of your Azure Blob Storage Account, for example, `https://test.blob.core.windows.net`. * Introduced in: v3.1 ##### `azure_blob_path`[​](#azure_blob_path "Direct link to azure_blob_path") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The Azure Blob Storage path used to store data. It consists of the name of the container within your storage account and the sub-path (if any) under the container, for example, `testcontainer/subpath`. * Introduced in: v3.1 ##### `azure_blob_sas_token`[​](#azure_blob_sas_token "Direct link to azure_blob_sas_token") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The shared access signatures (SAS) used to authorize requests for your Azure Blob Storage. * Introduced in: v3.1 ##### `azure_blob_shared_key`[​](#azure_blob_shared_key "Direct link to azure_blob_shared_key") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The Shared Key used to authorize requests for your Azure Blob Storage. * Introduced in: v3.1 ##### `azure_use_native_sdk`[​](#azure_use_native_sdk "Direct link to azure_use_native_sdk") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to use the native SDK to access Azure Blob Storage, thus allowing authentication with Managed Identities and Service Principals. If this item is set to `false`, only authentication with Shared Key and SAS Token is allowed. * Introduced in: v3.4.4 ##### `cloud_native_hdfs_url`[​](#cloud_native_hdfs_url "Direct link to cloud_native_hdfs_url") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The URL of the HDFS storage, for example, `hdfs://127.0.0.1:9000/user/xxx/starrocks/`. * Introduced in: - ##### `cloud_native_meta_port`[​](#cloud_native_meta_port "Direct link to cloud_native_meta_port") * Default: 6090 * Type: Int * Unit: - * Is mutable: No * Description: FE cloud-native metadata server RPC listen port. * Introduced in: - ##### `cloud_native_storage_type`[​](#cloud_native_storage_type "Direct link to cloud_native_storage_type") * Default: S3 * Type: String * Unit: - * Is mutable: No * Description: The type of object storage you use. In shared-data mode, StarRocks supports storing data in HDFS, Azure Blob (supported from v3.1.1 onwards), Azure Data Lake Storage Gen2 (supported from v3.4.1 onwards), Google Storage (with native SDK, supported from v3.5.1 onwards), and object storage systems that are compatible with the S3 protocol (such as AWS S3, and MinIO). Valid value: `S3` (Default), `HDFS`, `AZBLOB`, `ADLS2`, and `GS`. If you specify this parameter as `S3`, you must add the parameters prefixed by `aws_s3`. If you specify this parameter as `AZBLOB`, you must add the parameters prefixed by `azure_blob`. If you specify this parameter as `ADLS2`, you must add the parameters prefixed by `azure_adls2`. If you specify this parameter as `GS`, you must add the parameters prefixed by `gcp_gcs`. If you specify this parameter as `HDFS`, you only need to specify `cloud_native_hdfs_url`. * Introduced in: - ##### `enable_admin_skip_committed_txn`[​](#enable_admin_skip_committed_txn "Direct link to enable_admin_skip_committed_txn") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable the `ADMIN SKIP COMMITTED TRANSACTION` statement. When `false`, the statement is rejected with an error. This is an operator-only escape hatch for unblocking a publish-stuck `COMMITTED` transaction on a shared-data (lake) table; the stuck transaction's data contribution is discarded, while the partition's visible version still advances via a "no-op publish" (a new metadata file is written that carries no data changes from this transaction). Only supports lake tables with `file_bundling=true`; load and lake-compaction transaction types only (alter / schema-change are not yet supported). Enable only when an operator needs to manually unblock a stuck transaction, and disable again afterwards to prevent accidental use. * Introduced in: - ##### `enable_load_volume_from_conf`[​](#enable_load_volume_from_conf "Direct link to enable_load_volume_from_conf") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to allow StarRocks to create the built-in storage volume by using the object storage-related properties specified in the FE configuration file. The default value is changed from `true` to `false` from v3.4.1 onwards. * Introduced in: v3.1.0 ##### `gcp_gcs_impersonation_service_account`[​](#gcp_gcs_impersonation_service_account "Direct link to gcp_gcs_impersonation_service_account") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The Service Account that you want to impersonate if you use the impersonation-based authentication to access Google Storage. * Introduced in: v3.5.1 ##### `gcp_gcs_path`[​](#gcp_gcs_path "Direct link to gcp_gcs_path") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The Google Cloud path used to store data. It consists of the name of your Google Cloud bucket and the sub-path (if any) under it, for example, `testbucket/subpath`. * Introduced in: v3.5.1 ##### `gcp_gcs_service_account_email`[​](#gcp_gcs_service_account_email "Direct link to gcp_gcs_service_account_email") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The email address in the JSON file generated at the creation of the Service Account, for example, `user@hello.iam.gserviceaccount.com`. * Introduced in: v3.5.1 ##### `gcp_gcs_service_account_private_key`[​](#gcp_gcs_service_account_private_key "Direct link to gcp_gcs_service_account_private_key") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The Private Key in the JSON file generated at the creation of the Service Account, for example, `-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n`. * Introduced in: v3.5.1 ##### `gcp_gcs_service_account_private_key_id`[​](#gcp_gcs_service_account_private_key_id "Direct link to gcp_gcs_service_account_private_key_id") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The Private Key ID in the JSON file generated at the creation of the Service Account. * Introduced in: v3.5.1 ##### `gcp_gcs_use_compute_engine_service_account`[​](#gcp_gcs_use_compute_engine_service_account "Direct link to gcp_gcs_use_compute_engine_service_account") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to use the Service Account that is bound to your Compute Engine. * Introduced in: v3.5.1 ##### `hdfs_file_system_expire_seconds`[​](#hdfs_file_system_expire_seconds "Direct link to hdfs_file_system_expire_seconds") * Default: 300 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Time-to-live in seconds for an unused cached HDFS/ObjectStore FileSystem managed by HdfsFsManager. The FileSystemExpirationChecker (runs every 60s) calls each HdfsFs.isExpired(...) using this value; when expired the manager closes the underlying FileSystem and removes it from the cache. Accessor methods (for example `HdfsFs.getDFSFileSystem`, `getUserName`, `getConfiguration`) update the last-access timestamp, so expiry is based on inactivity. Lower values reduce idle resource holding but increase reopen overhead; higher values keep handles longer and may consume more resources. * Introduced in: v3.2.0 ##### `lake_autovacuum_grace_period_minutes`[​](#lake_autovacuum_grace_period_minutes "Direct link to lake_autovacuum_grace_period_minutes") * Default: 30 * Type: Long * Unit: Minutes * Is mutable: Yes * Description: The time range for retaining historical data versions in a shared-data cluster. Historical data versions within this time range are not automatically cleaned via AutoVacuum after Compactions. You need to set this value greater than the maximum query time to avoid that the data accessed by running queries get deleted before the queries finish. The default value has been changed from `5` to `30` since v3.3.0, v3.2.5, and v3.1.10. * Introduced in: v3.1.0 ##### `lake_autovacuum_parallel_partitions`[​](#lake_autovacuum_parallel_partitions "Direct link to lake_autovacuum_parallel_partitions") * Default: 8 * Type: Int * Unit: - * Is mutable: No * Description: The maximum number of partitions that can undergo AutoVacuum simultaneously in a shared-data cluster. AutoVacuum is the Garbage Collection after Compactions. * Introduced in: v3.1.0 ##### `lake_autovacuum_partition_naptime_seconds`[​](#lake_autovacuum_partition_naptime_seconds "Direct link to lake_autovacuum_partition_naptime_seconds") * Default: 180 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The minimum interval between AutoVacuum operations on the same partition in a shared-data cluster. * Introduced in: v3.1.0 ##### `lake_autovacuum_stale_partition_threshold`[​](#lake_autovacuum_stale_partition_threshold "Direct link to lake_autovacuum_stale_partition_threshold") * Default: 12 * Type: Long * Unit: Hours * Is mutable: Yes * Description: If a partition has no updates (loading, DELETE, or Compactions) within this time range, the system will not perform AutoVacuum on this partition. * Introduced in: v3.1.0 ##### `lake_compaction_allow_partial_success`[​](#lake_compaction_allow_partial_success "Direct link to lake_compaction_allow_partial_success") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: If this item is set to `true`, the system will consider the Compaction operation in a shared-data cluster as successful when one of the sub-tasks succeeds. * Introduced in: v3.5.2 ##### `lake_compaction_disable_ids`[​](#lake_compaction_disable_ids "Direct link to lake_compaction_disable_ids") * Default: "" * Type: String * Unit: - * Is mutable: Yes * Description: The table or partition list of which compaction is disabled in shared-data mode. The format is `tableId1;partitionId2`, seperated by semicolon, for example, `12345;98765`. * Introduced in: v3.4.4 ##### `lake_compaction_history_size`[​](#lake_compaction_history_size "Direct link to lake_compaction_history_size") * Default: 20 * Type: Int * Unit: - * Is mutable: Yes * Description: The number of recent successful Compaction task records to keep in the memory of the Leader FE node in a shared-data cluster. You can view recent successful Compaction task records using the `SHOW PROC '/compactions'` command. Note that the Compaction history is stored in the FE process memory, and it will be lost if the FE process is restarted. * Introduced in: v3.1.0 ##### `lake_compaction_max_parallel_default`[​](#lake_compaction_max_parallel_default "Direct link to lake_compaction_max_parallel_default") * Default: 3 * Type: Int * Unit: - * Is mutable: Yes * Description: Default max parallel compaction subtasks per tablet when `lake_compaction_max_parallel` is not specified in table properties. `0` means disable parallel compaction. This config is used as the default value for the table property `lake_compaction_max_parallel`. ##### `lake_compaction_max_tasks`[​](#lake_compaction_max_tasks "Direct link to lake_compaction_max_tasks") * Default: -1 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of concurrent Compaction tasks allowed in a shared-data cluster. Setting this item to `-1` indicates to calculate the concurrent task number in an adaptive manner. Setting this value to `0` will disable compaction. * Introduced in: v3.1.0 ##### `lake_compaction_score_selector_min_score`[​](#lake_compaction_score_selector_min_score "Direct link to lake_compaction_score_selector_min_score") * Default: 10.0 * Type: Double * Unit: - * Is mutable: Yes * Description: The Compaction Score threshold that triggers Compaction operations in a shared-data cluster. When the Compaction Score of a partition is greater than or equal to this value, the system performs Compaction on that partition. * Introduced in: v3.1.0 ##### `lake_compaction_score_upper_bound`[​](#lake_compaction_score_upper_bound "Direct link to lake_compaction_score_upper_bound") * Default: 2000 * Type: Long * Unit: - * Is mutable: Yes * Description: The upper limit of the Compaction Score for a partition in a shared-data cluster. `0` indicates no upper limit. This item only takes effect when `lake_enable_ingest_slowdown` is set to `true`. When the Compaction Score of a partition reaches or exceeds this upper limit, incoming loading tasks will be rejected. From v3.3.6 onwards, the default value is changed from `0` to `2000`. * Introduced in: v3.2.0 ##### `lake_compaction_interval_ms_on_success`[​](#lake_compaction_interval_ms_on_success "Direct link to lake_compaction_interval_ms_on_success") * Default: 10000 * Type: Long * Unit: Milliseconds * Is mutable: Yes * Description: The interval before triggering the next Compaction for a partition in a shared-data cluster after a successful Compaction on that partition. The alias is `lake_min_compaction_interval_ms_on_success`. * Introduced in: v3.2.0 ##### `lake_enable_balance_tablets_between_workers`[​](#lake_enable_balance_tablets_between_workers "Direct link to lake_enable_balance_tablets_between_workers") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to balance the number of tablets among Compute Nodes during the tablet migration of cloud-native tables in a shared-data cluster. `true` indicates to balance the tablets among Compute Nodes, and `false` indicates to disabling this feature. * Introduced in: v3.3.4 ##### `lake_enable_ingest_slowdown`[​](#lake_enable_ingest_slowdown "Direct link to lake_enable_ingest_slowdown") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable Data Ingestion Slowdown in a shared-data cluster. When Data Ingestion Slowdown is enabled, if the Compaction Score of a partition exceeds `lake_ingest_slowdown_threshold`, loading tasks on that partition will be throttled down. This configuration only takes effect when `run_mode` is set to `shared_data`. From v3.3.6 onwards, the default value is chenged from `false` to `true`. * Introduced in: v3.2.0 ##### `lake_ingest_slowdown_threshold`[​](#lake_ingest_slowdown_threshold "Direct link to lake_ingest_slowdown_threshold") * Default: 100 * Type: Long * Unit: - * Is mutable: Yes * Description: The Compaction Score threshold that triggers Data Ingestion Slowdown in a shared-data cluster. This configuration only takes effect when `lake_enable_ingest_slowdown` is set to `true`. * Introduced in: v3.2.0 ##### `lake_publish_version_max_threads`[​](#lake_publish_version_max_threads "Direct link to lake_publish_version_max_threads") * Default: 512 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of threads for Version Publish tasks in a shared-data cluster. * Introduced in: v3.2.0 ##### `slow_publish_partition_log_threshold_ms`[​](#slow_publish_partition_log_threshold_ms "Direct link to slow_publish_partition_log_threshold_ms") * Default: 3000 * Type: Long * Unit: Milliseconds * Is mutable: Yes * Description: The threshold above which `PublishVersionDaemon` logs a per-phase breakdown (`executor_queue` + `db_lock_wait` + `fe_prep` + `rpc`) of a slow partition publish at the WARN level. Lower this value to capture sub-second jitter when investigating publish latency on a live cluster; raise it to silence routine slow-but-acceptable publishes. There is no behavior change at the default value. * Introduced in: v4.2 ##### `meta_sync_force_delete_shard_meta`[​](#meta_sync_force_delete_shard_meta "Direct link to meta_sync_force_delete_shard_meta") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to allow deleting the metadata of the shared-data cluster directly, bypassing cleaning the remote storage files. It is recommended to set this item to `true` only when there is an excessive number of shards to be cleaned, which leads to extreme memory pressure on the FE JVM. Note that the data files belonging to the shards or tablets cannot be automatically cleaned after this feature is enabled. * Introduced in: v3.2.10, v3.3.3 ##### `run_mode`[​](#run_mode "Direct link to run_mode") * Default: `shared_nothing` * Type: String * Unit: - * Is mutable: No * Description: The running mode of the StarRocks cluster. Valid values: `shared_data` and `shared_nothing` (Default). * `shared_data` indicates running StarRocks in shared-data mode. * `shared_nothing` indicates running StarRocks in shared-nothing mode. > **CAUTION** > > * You cannot adopt the `shared_data` and `shared_nothing` modes simultaneously for a StarRocks cluster. Mixed deployment is not supported. > * DO NOT change `run_mode` after the cluster is deployed. Otherwise, the cluster fails to restart. The transformation from a shared-nothing cluster to a shared-data cluster or vice versa is not supported. * Introduced in: - ##### `shard_group_clean_threshold_sec`[​](#shard_group_clean_threshold_sec "Direct link to shard_group_clean_threshold_sec") * Default: 3600 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The time before FE cleans the unused tablet and shard groups in a shared-data cluster. Tablets and shard groups created within this threshold will not be cleaned. * Introduced in: - ##### `star_mgr_meta_sync_interval_sec`[​](#star_mgr_meta_sync_interval_sec "Direct link to star_mgr_meta_sync_interval_sec") * Default: 600 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The interval at which FE runs the periodical metadata synchronization with StarMgr in a shared-data cluster. * Introduced in: - ##### `starmgr_grpc_server_max_worker_threads`[​](#starmgr_grpc_server_max_worker_threads "Direct link to starmgr_grpc_server_max_worker_threads") * Default: 1024 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of worker threads that are used by the grpc server in the FE starmgr module. * Introduced in: v4.0.0, v3.5.8 ##### `starmgr_grpc_timeout_seconds`[​](#starmgr_grpc_timeout_seconds "Direct link to starmgr_grpc_timeout_seconds") * Default: 5 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: * Introduced in: - #### Data Lake[​](#data-lake "Direct link to Data Lake") ##### `files_enable_insert_push_down_column_type`[​](#files_enable_insert_push_down_column_type "Direct link to files_enable_insert_push_down_column_type") * Default: true * Alias: `files_enable_insert_push_down_schema` * Type: Boolean * Unit: - * Is mutable: Yes * Description: When enabled, StarRocks pushes target table column types down to the `files()` table function for `INSERT INTO target_table SELECT ... FROM files()` operations. Only the types of columns already inferred from the files are rewritten; no columns are added or removed. Complex types are skipped. This reduces type-mismatch errors caused by imprecise file-based type inference. For full schema push-down (column names and types), use the INSERT property `enable_push_down_schema`. * Introduced in: v3.4.0, v3.5.0 ##### `hdfs_read_buffer_size_kb`[​](#hdfs_read_buffer_size_kb "Direct link to hdfs_read_buffer_size_kb") * Default: 8192 * Type: Int * Unit: Kilobytes * Is mutable: Yes * Description: Size of the HDFS read buffer in kilobytes. StarRocks converts this value to bytes (`<< 10`) and uses it to initialize HDFS read buffers in `HdfsFsManager` and to populate the thrift field `hdfs_read_buffer_size_kb` sent to BE tasks (e.g., `TBrokerScanRangeParams`, `TDownloadReq`) when broker access is not used. Increasing `hdfs_read_buffer_size_kb` can improve sequential read throughput and reduce syscall overhead at the cost of higher per-stream memory usage; decreasing it reduces memory footprint but may lower IO efficiency. Consider workload (many small streams vs. few large sequential reads) when tuning. * Introduced in: v3.2.0 ##### `hdfs_write_buffer_size_kb`[​](#hdfs_write_buffer_size_kb "Direct link to hdfs_write_buffer_size_kb") * Default: 1024 * Type: Int * Unit: Kilobytes * Is mutable: Yes * Description: Sets the HDFS write buffer size (in KB) used for direct writes to HDFS or object stores when not using a broker. The FE converts this value to bytes (`<< 10`) and initializes the local write buffer in HdfsFsManager, and it is propagated in Thrift requests (e.g., TUploadReq, TExportSink, sink options) so backends/agents use the same buffer size. Increasing this value can improve throughput for large sequential writes at the cost of more memory per writer; decreasing it reduces per-stream memory usage and may lower latency for small writes. Tune alongside `hdfs_read_buffer_size_kb` and consider available memory and concurrent writers. * Introduced in: v3.2.0 ##### `lake_batch_publish_max_version_num`[​](#lake_batch_publish_max_version_num "Direct link to lake_batch_publish_max_version_num") * Default: 10 * Type: Int * Unit: Count * Is mutable: Yes * Description: Sets the upper bound on how many consecutive transaction versions may be grouped together when building a publish batch for lake (cloud‑native) tables. The value is passed to the transaction graph batching routine (see getReadyToPublishTxnListBatch) and works together with `lake_batch_publish_min_version_num` to determine the candidate range size for a TransactionStateBatch. Larger values can increase publish throughput by batching more commits, but increase the scope of an atomic publish (longer visibility latency and larger rollback surface) and may be limited at runtime when versions are not consecutive. Tune according to workload and visibility/latency requirements. * Introduced in: v3.2.0 ##### `lake_batch_publish_min_version_num`[​](#lake_batch_publish_min_version_num "Direct link to lake_batch_publish_min_version_num") * Default: 1 * Type: Int * Unit: - * Is mutable: Yes * Description: Sets the minimum number of consecutive transaction versions required to form a publish batch for lake tables. DatabaseTransactionMgr.getReadyToPublishTxnListBatch passes this value to transactionGraph.getTxnsWithTxnDependencyBatch together with `lake_batch_publish_max_version_num` to select dependent transactions. A value of `1` allows single-transaction publishes (no batching). Values `>1` require at least that many consecutively-versioned, single-table, non-replication transactions to be available; batching is aborted if versions are non-consecutive, a replication transaction appears, or a schema change consumes a version. Increasing this value can improve publish throughput by grouping commits but may delay publishing while waiting for enough consecutive transactions. * Introduced in: v3.2.0 ##### `lake_enable_batch_publish_version`[​](#lake_enable_batch_publish_version "Direct link to lake_enable_batch_publish_version") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: When enabled, PublishVersionDaemon batches ready transactions for the same Lake (shared-data) table/partition and publishes their versions together instead of issuing per-transaction publishes. In RunMode shared-data, the daemon calls getReadyPublishTransactionsBatch() and uses publishVersionForLakeTableBatch(...) to perform grouped publish operations (reducing RPCs and improving throughput). When disabled, the daemon falls back to per-transaction publishing via publishVersionForLakeTable(...). The implementation coordinates in-flight work using internal sets to avoid duplicate publishes when the switch is toggled and is affected by the thread pool sizing via `lake_publish_version_max_threads`. * Introduced in: v3.2.0 ##### `lake_enable_batch_publish_multi_table`[​](#lake_enable_batch_publish_multi_table "Direct link to lake_enable_batch_publish_multi_table") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to allow batch publish to group consecutive multi-table transactions into one publish operation. This benefits workloads that commit small atomic transactions spanning the same group of tables at a high rate (for example, CDC pipelines fanning out to multiple tables), where per-transaction publishing serializes on the shared table dependency chain and inflates the commit-to-visible latency. Effective only when `lake_enable_batch_publish_version` is `true`. Enable this parameter only after all FE nodes are upgraded to a version that supports it: an FE follower running an older version that replays a multi-table transaction batch applies the visible log of the first table only. * Introduced in: v4.1.5 ##### `lake_enable_tablet_creation_optimization`[​](#lake_enable_tablet_creation_optimization "Direct link to lake_enable_tablet_creation_optimization") * Default: false * Type: boolean * Unit: - * Is mutable: Yes * Description: When enabled, StarRocks optimizes tablet creation for cloud-native tables and materialized views in shared-data mode by creating a single shared tablet metadata for all tablets under a physical partition instead of distinct metadata per tablet. This reduces the number of tablet creation tasks and metadata/files produced during table creation, rollup, and schema-change jobs. The optimization is applied only for cloud-native tables/materialized views and is combined with `file_bundling` (the latter reuses the same optimization logic). Note: schema-change and rollup jobs explicitly disable the optimization for tables using `file_bundling` to avoid overwriting files with identical names. Enable cautiously — it changes the granularity of created tablet metadata and can affect how replica creation and file naming behave. * Introduced in: v3.3.1, v3.4.0, v3.5.0 ##### `lake_create_tablet_max_retries`[​](#lake_create_tablet_max_retries "Direct link to lake_create_tablet_max_retries") * Default: 1 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of retry attempts for failed create-tablet tasks in shared-data mode. When a CN is unreachable or down during table creation, the failed tasks are retried on an alternative alive CN. Only send-phase failures (RPC errors, node down) are retried; CN-reported errors and timeouts are not retried. Set to `0` to disable retry. * Introduced in: v4.1 ##### `lake_use_combined_txn_log`[​](#lake_use_combined_txn_log "Direct link to lake_use_combined_txn_log") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: When this item is set to `true`, the system allows Lake tables to use the combined transaction log path for relevant transactions. Available for shared-data clusters only. * Introduced in: v3.3.7, v3.4.0, v3.5.0 ##### `lake_repair_metadata_fetch_max_version_batch_size`[​](#lake_repair_metadata_fetch_max_version_batch_size "Direct link to lake_repair_metadata_fetch_max_version_batch_size") * Default: 160 * Type: Long * Unit: - * Is mutable: Yes * Description: The maximum batch size for version scanning when fetching tablet metadata during lake tablet repair. The batch size starts at 5 and grows exponentially (doubling each iteration) up to this maximum. A larger value allows more versions to be fetched in a single batch, which can improve repair efficiency by leveraging file existence caching across versions. If set to a value less than 5, it will be clamped to 5 at runtime. * Introduced in: v3.5.16, v4.0.9 ##### `lake_enable_drop_tablet_cache`[​](#lake_enable_drop_tablet_cache "Direct link to lake_enable_drop_tablet_cache") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: In shared-data mode, clears the cache on BE/CN before the underlying data is actually deleted. * Introduced in: v4.0 ##### `enable_iceberg_commit_queue`[​](#enable_iceberg_commit_queue "Direct link to enable_iceberg_commit_queue") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable commit queue for Iceberg tables to avoid concurrent commit conflicts. Iceberg uses optimistic concurrency control (OCC) for metadata commits. When multiple threads concurrently commit to the same table, conflicts can occur with errors like: "Cannot commit: Base metadata location is not same as the current table metadata location". When enabled, each Iceberg table has its own single-threaded executor for commit operations, ensuring that commits to the same table are serialized and preventing OCC conflicts. Different tables can commit concurrently, maintaining overall throughput. This is a system-level optimization to improve reliability and should be enabled by default. If disabled, concurrent commits may fail due to optimistic locking conflicts. * Introduced in: v4.1.0 ##### `iceberg_commit_queue_timeout_seconds`[​](#iceberg_commit_queue_timeout_seconds "Direct link to iceberg_commit_queue_timeout_seconds") * Default: 300 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The timeout in seconds for waiting for an Iceberg commit operation to complete. When using the commit queue (`enable_iceberg_commit_queue=true`), each commit operation must complete within this timeout. If a commit takes longer than this timeout, it will be cancelled and an error will be raised. Factors that affect commit time include: number of data files being committed, metadata size of the table, performance of the underlying storage (e.g., S3, HDFS). * Introduced in: v4.1.0 ##### `iceberg_commit_queue_max_size`[​](#iceberg_commit_queue_max_size "Direct link to iceberg_commit_queue_max_size") * Default: 1000 * Type: Int * Unit: Count * Is mutable: No * Description: The maximum number of pending commit operations per Iceberg table. When using the commit queue (`enable_iceberg_commit_queue=true`), this limits the number of commit operations that can be queued for a single table. When the limit is reached, additional commit operations will execute in the caller thread (blocking until capacity available). This configuration is read at FE startup and applies to newly created table executors. Requires FE restart to take effect. Increase this value if you expect many concurrent commits to the same table. If this value is too low, commits may block in the caller thread during high concurrency. * Introduced in: v4.1.0 ##### `iceberg_remove_orphan_files_min_retention_seconds`[​](#iceberg_remove_orphan_files_min_retention_seconds "Direct link to iceberg_remove_orphan_files_min_retention_seconds") * Default: 86400 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The `older_than` argument of the `remove_orphan_files` procedure must be earlier than `current time - this value`. A later `older_than` is rejected, because deleting files that recent can remove data that a concurrent write has not committed yet and leave the table unreadable. Only an explicit `older_than` is bounded; omitting it keeps the procedure's own 7-day default. Lower this value only when nothing writes to the table while the procedure runs. * Introduced in: v4.1.5 ###### lake\_balance\_tablets\_threshold[​](#lake_balance_tablets_threshold "Direct link to lake_balance_tablets_threshold") * Default: 0.15 * Type: Double * Unit: - * Is mutable: Yes * Description: The threshold the system used to judge the tablet balance among workers in a shared-data cluster, The imbalance factor is calculated as `f = (MAX(tablets) - MIN(tablets)) / AVERAGE(tablets)`. If the factor is greater than `lake_balance_tablets_threshold`, a tablet balance will be triggered. This item takes effect only when `lake_enable_balance_tablets_between_workers` is set to `true`. * Introduced in: v3.3.4 #### Other[​](#other "Direct link to Other") ##### `agent_task_resend_wait_time_ms`[​](#agent_task_resend_wait_time_ms "Direct link to agent_task_resend_wait_time_ms") * Default: 5000 * Type: Long * Unit: Milliseconds * Is mutable: Yes * Description: The duration the FE must wait before it can resend an agent task. An agent task can be resent only when the gap between the task creation time and the current time exceeds the value of this parameter. This parameter is used to prevent repetitive sending of agent tasks. * Introduced in: - ##### `allow_system_reserved_names`[​](#allow_system_reserved_names "Direct link to allow_system_reserved_names") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to allow users to create columns whose names are initiated with `__op` and `__row`. To enable this feature, set this parameter to `TRUE`. Please note that these name formats are reserved for special purposes in StarRocks and creating such columns may result in undefined behavior. Therefore this feature is disabled by default. * Introduced in: v3.2.0 ##### `auth_token`[​](#auth_token "Direct link to auth_token") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The token that is used for identity authentication within the StarRocks cluster to which the FE belongs. If this parameter is left unspecified, StarRocks generates a random token for the cluster at the time when the leader FE of the cluster is started for the first time. * Introduced in: - ##### `authentication_ldap_simple_bind_base_dn`[​](#authentication_ldap_simple_bind_base_dn "Direct link to authentication_ldap_simple_bind_base_dn") * Default: Empty string * Type: String * Unit: - * Is mutable: Yes * Description: The base DN, which is the point from which the LDAP server starts to search for users' authentication information. * Introduced in: - ##### `authentication_ldap_simple_bind_dn_pattern`[​](#authentication_ldap_simple_bind_dn_pattern "Direct link to authentication_ldap_simple_bind_dn_pattern") * Default: Empty string * Type: String * Unit: - * Is mutable: Yes * Description: The DN pattern for direct bind authentication. Use `${USER}` as a placeholder for the username. The pattern must produce a valid LDAP Distinguished Name (DN); UPN-style patterns like `${USER}@domain` are not supported. For example, `uid=${USER},ou=People,dc=example,dc=com`. Multiple patterns can be separated by semicolons, and the system will try each pattern in order until one succeeds. When set, the search step is skipped and the system binds directly with the constructed DN. ##### `authentication_ldap_simple_bind_root_dn`[​](#authentication_ldap_simple_bind_root_dn "Direct link to authentication_ldap_simple_bind_root_dn") * Default: Empty string * Type: String * Unit: - * Is mutable: Yes * Description: The administrator DN used to search for users' authentication information. * Introduced in: - ##### `authentication_ldap_simple_bind_root_pwd`[​](#authentication_ldap_simple_bind_root_pwd "Direct link to authentication_ldap_simple_bind_root_pwd") * Default: Empty string * Type: String * Unit: - * Is mutable: Yes * Description: The password of the administrator used to search for users' authentication information. * Introduced in: - ##### `authentication_ldap_simple_server_host`[​](#authentication_ldap_simple_server_host "Direct link to authentication_ldap_simple_server_host") * Default: Empty string * Type: String * Unit: - * Is mutable: Yes * Description: The host on which the LDAP server runs. * Introduced in: - ##### `authentication_ldap_simple_server_port`[​](#authentication_ldap_simple_server_port "Direct link to authentication_ldap_simple_server_port") * Default: 389 * Type: Int * Unit: - * Is mutable: Yes * Description: The port of the LDAP server. * Introduced in: - ##### `authentication_ldap_simple_user_search_attr`[​](#authentication_ldap_simple_user_search_attr "Direct link to authentication_ldap_simple_user_search_attr") * Default: uid * Type: String * Unit: - * Is mutable: Yes * Description: The name of the attribute that identifies users in LDAP objects. * Introduced in: - ##### `backup_job_default_timeout_ms`[​](#backup_job_default_timeout_ms "Direct link to backup_job_default_timeout_ms") * Default: 86400 \* 1000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: The timeout duration of a backup job. If this value is exceeded, the backup job fails. * Introduced in: - ##### `enable_collect_tablet_num_in_show_proc_backend_disk_path`[​](#enable_collect_tablet_num_in_show_proc_backend_disk_path "Direct link to enable_collect_tablet_num_in_show_proc_backend_disk_path") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable the collection of tablet numbers for each disk in the `SHOW PROC /BACKENDS/{id}` command * Introduced in: v4.0.1, v3.5.8 ##### `enable_colocate_restore`[​](#enable_colocate_restore "Direct link to enable_colocate_restore") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable Backup and Restore for Colocate Tables. `true` indicates enabling Backup and Restore for Colocate Tables and `false` indicates disabling it. * Introduced in: v3.2.10, v3.3.3 ##### `enable_external_catalog_information_schema_tables_access_full_metadata`[​](#enable_external_catalog_information_schema_tables_access_full_metadata "Direct link to enable_external_catalog_information_schema_tables_access_full_metadata") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Controls whether `information_schema.tables` is allowed to access external metadata services when resolving tables in external catalogs (such as Hive, Iceberg, JDBC). When set to `false` (default), columns like `TABLE_COMMENT` may be empty for external tables but the query is fast and avoids remote calls. When set to `true`, the FE contacts the corresponding external metadata service and can populate fields like `TABLE_COMMENT` at the cost of additional latency and remote calls per table. * Introduced in: - ##### `enable_materialized_view_concurrent_prepare`[​](#enable_materialized_view_concurrent_prepare "Direct link to enable_materialized_view_concurrent_prepare") * Default: true * Type: Boolean * Unit: * Is mutable: Yes * Description: Whether to prepare materialized view concurrently to improve performance. * Introduced in: v3.4.4 ##### `enable_metric_calculator`[​](#enable_metric_calculator "Direct link to enable_metric_calculator") * Default: true * Type: Boolean * Unit: - * Is mutable: No * Description: Specifies whether to enable the feature that is used to periodically collect metrics. Valid values: `TRUE` and `FALSE`. `TRUE` specifies to enable this feature, and `FALSE` specifies to disable this feature. * Introduced in: - ##### `enable_table_metrics_collect`[​](#enable_table_metrics_collect "Direct link to enable_table_metrics_collect") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to export table-level metrics in FE. When disabled, FE will skip exporting table metrics (such as table scan/load counters and table size metrics), but still records the counters in memory. * Introduced in: - ##### `enable_mv_post_image_reload_cache`[​](#enable_mv_post_image_reload_cache "Direct link to enable_mv_post_image_reload_cache") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to perform reload flag check after FE loaded an image. If the check is performed for a base materialized view, it is not needed for other materialized views that related to it. * Introduced in: v3.5.0 ##### `enable_mv_query_context_cache`[​](#enable_mv_query_context_cache "Direct link to enable_mv_query_context_cache") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable query-level materialized view rewrite cache to improve query rewrite performance. * Introduced in: v3.3 ##### `enable_mv_refresh_collect_profile`[​](#enable_mv_refresh_collect_profile "Direct link to enable_mv_refresh_collect_profile") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable profile in refreshing materialized view by default for all materialized views. * Introduced in: v3.3.0 ##### `enable_mv_refresh_extra_prefix_logging`[​](#enable_mv_refresh_extra_prefix_logging "Direct link to enable_mv_refresh_extra_prefix_logging") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable prefixes with materialized view names in logs for better debug. * Introduced in: v3.4.0 ##### `enable_mv_refresh_query_rewrite`[​](#enable_mv_refresh_query_rewrite "Direct link to enable_mv_refresh_query_rewrite") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable rewrite query during materialized view refresh so that the query can use the rewritten mv directly rather than the base table to improve query performance. * Introduced in: v3.3 ##### `enable_trace_historical_node`[​](#enable_trace_historical_node "Direct link to enable_trace_historical_node") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to allow the system to trace the historical nodes. By setting this item to `true`, you can enable the Cache Sharing feature and allow the system to choose the right cache nodes during elastic scaling. * Introduced in: v3.5.1 ##### `es_state_sync_interval_second`[​](#es_state_sync_interval_second "Direct link to es_state_sync_interval_second") * Default: 10 * Type: Long * Unit: Seconds * Is mutable: No * Description: The time interval at which the FE obtains Elasticsearch indexes and synchronizes the metadata of StarRocks external tables. * Introduced in: - ##### `hive_meta_cache_refresh_interval_s`[​](#hive_meta_cache_refresh_interval_s "Direct link to hive_meta_cache_refresh_interval_s") * Default: 60 * Type: Long * Unit: Seconds * Is mutable: No * Description: The time interval at which the cached metadata of Hive external tables is updated. * Introduced in: - ##### `hive_meta_store_timeout_s`[​](#hive_meta_store_timeout_s "Direct link to hive_meta_store_timeout_s") * Default: 10 * Type: Long * Unit: Seconds * Is mutable: No * Description: The amount of time after which a connection to a Hive metastore times out. * Introduced in: - ##### `jdbc_connection_idle_timeout_ms`[​](#jdbc_connection_idle_timeout_ms "Direct link to jdbc_connection_idle_timeout_ms") * Default: 600000 * Type: Int * Unit: Milliseconds * Is mutable: No * Description: The maximum amount of time after which a connection for accessing a JDBC catalog times out. Timed-out connections are considered idle. * Introduced in: - ##### `jdbc_connection_timeout_ms`[​](#jdbc_connection_timeout_ms "Direct link to jdbc_connection_timeout_ms") * Default: 10000 * Type: Long * Unit: Milliseconds * Is mutable: No * Description: The timeout in milliseconds for HikariCP connection pool to acquire a connection. If a connection cannot be acquired from the pool within this time, the operation will fail. * Introduced in: v3.5.13 ##### `jdbc_query_timeout_ms`[​](#jdbc_query_timeout_ms "Direct link to jdbc_query_timeout_ms") * Default: 30000 * Type: Long * Unit: Milliseconds * Is mutable: Yes * Description: The timeout in milliseconds for JDBC statement query execution. This timeout is applied to all SQL queries executed through JDBC catalogs (e.g., partition metadata queries). The value is converted to seconds when passed to the JDBC driver. * Introduced in: v3.5.13 ##### `jdbc_network_timeout_ms`[​](#jdbc_network_timeout_ms "Direct link to jdbc_network_timeout_ms") * Default: 30000 * Type: Long * Unit: Milliseconds * Is mutable: Yes * Description: The timeout in milliseconds for JDBC network operations (socket read). This timeout applies to database metadata calls (e.g., getSchemas(), getTables(), getColumns()) to prevent indefinite blocking when the external database is unresponsive. * Introduced in: v3.5.13 ##### `jdbc_connection_max_lifetime_ms`[​](#jdbc_connection_max_lifetime_ms "Direct link to jdbc_connection_max_lifetime_ms") * Default: 300000 * Type: Long * Unit: Milliseconds * Is mutable: No * Description: Maximum lifetime of a connection in the JDBC connection pool. Connections are recycled before this timeout to prevent stale connections. Should be shorter than the external database's connection timeout. Minimum allowed value is 30000 (30 seconds). * Introduced in: - ##### `jdbc_connection_keepalive_time_ms`[​](#jdbc_connection_keepalive_time_ms "Direct link to jdbc_connection_keepalive_time_ms") * Default: 30000 * Type: Long * Unit: Milliseconds * Is mutable: No * Description: Keepalive interval for idle JDBC connections. Idle connections are tested at this interval to detect stale connections proactively. Set to 0 to disable keepalive probing. When enabled, must be >= 30000 and less than `jdbc_connection_max_lifetime_ms`. Invalid enabled values are silently disabled (reset to 0). * Introduced in: - ##### `jdbc_connection_leak_detection_threshold_ms`[​](#jdbc_connection_leak_detection_threshold_ms "Direct link to jdbc_connection_leak_detection_threshold_ms") * Default: 0 * Type: Long * Unit: Milliseconds * Is mutable: No * Description: Threshold for JDBC connection leak detection. If a connection is held longer than this, a warning is logged. Set to 0 to disable. This is a debugging aid for identifying code paths that hold connections too long. * Introduced in: - ##### `jdbc_connection_pool_size`[​](#jdbc_connection_pool_size "Direct link to jdbc_connection_pool_size") * Default: 8 * Type: Int * Unit: - * Is mutable: No * Description: The maximum capacity of the JDBC connection pool for accessing JDBC catalogs. * Introduced in: - ##### `jdbc_meta_default_cache_enable`[​](#jdbc_meta_default_cache_enable "Direct link to jdbc_meta_default_cache_enable") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: The default value for whether the JDBC Catalog metadata cache is enabled. When set to True, newly created JDBC Catalogs will default to metadata caching enabled. * Introduced in: - ##### `jdbc_meta_default_cache_expire_sec`[​](#jdbc_meta_default_cache_expire_sec "Direct link to jdbc_meta_default_cache_expire_sec") * Default: 600 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The default expiration time for the JDBC Catalog metadata cache. When `jdbc_meta_default_cache_enable` is set to true, newly created JDBC Catalogs will default to setting the expiration time of the metadata cache. * Introduced in: - ##### `jdbc_row_count_cache_refresh_sec`[​](#jdbc_row_count_cache_refresh_sec "Direct link to jdbc_row_count_cache_refresh_sec") * Default: 600 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: Background refresh interval for the JDBC table row-count cache. After this interval, the stale value is returned immediately while a reload runs asynchronously in the background. Overridable per-catalog via the catalog property `jdbc_row_count_cache_refresh_sec`. * Introduced in: - ##### `jdbc_row_count_cache_expire_sec`[​](#jdbc_row_count_cache_expire_sec "Direct link to jdbc_row_count_cache_expire_sec") * Default: 1200 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: Hard eviction TTL for JDBC table row-count cache entries. Entries not accessed within this window are evicted. Must be greater than `jdbc_row_count_cache_refresh_sec`. Overridable per-catalog via the catalog property `jdbc_row_count_cache_expire_sec`. * Introduced in: - ##### `jdbc_row_count_cache_max_size`[​](#jdbc_row_count_cache_max_size "Direct link to jdbc_row_count_cache_max_size") * Default: 10000 * Type: Long * Unit: - * Is mutable: Yes * Description: Maximum number of entries in the JDBC table row-count cache per catalog. Limits memory growth for catalogs with large numbers of tables. Overridable per-catalog via the catalog property `jdbc_row_count_cache_max_size`. * Introduced in: - ##### `jdbc_minimum_idle_connections`[​](#jdbc_minimum_idle_connections "Direct link to jdbc_minimum_idle_connections") * Default: 1 * Type: Int * Unit: - * Is mutable: No * Description: The minimum number of idle connections in the JDBC connection pool for accessing JDBC catalogs. * Introduced in: - ##### `jwt_jwks_url`[​](#jwt_jwks_url "Direct link to jwt_jwks_url") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The URL to the JSON Web Key Set (JWKS) service or the path to the public key local file under the `fe/conf` directory. * Introduced in: v3.5.0 ##### `jwt_principal_field`[​](#jwt_principal_field "Direct link to jwt_principal_field") * Default: sub * Type: String * Unit: - * Is mutable: No * Description: The string used to identify the field that indicates the subject (`sub`) in the JWT. The default value is `sub`. The value of this field must be identical with the username for logging in to StarRocks. * Introduced in: v3.5.0 ##### `jwt_required_audience`[​](#jwt_required_audience "Direct link to jwt_required_audience") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The list of strings used to identify the audience (`aud`) in the JWT. The JWT is considered valid only if one of the values in the list match the JWT audience. * Introduced in: v3.5.0 ##### `jwt_required_issuer`[​](#jwt_required_issuer "Direct link to jwt_required_issuer") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The list of strings used to identify the issuers (`iss`) in the JWT. The JWT is considered valid only if one of the values in the list match the JWT issuer. * Introduced in: v3.5.0 ##### locale[​](#locale "Direct link to locale") * Default: `zh_CN.UTF-8` * Type: String * Unit: - * Is mutable: No * Description: The character set that is used by the FE. * Introduced in: - ##### `max_agent_task_threads_num`[​](#max_agent_task_threads_num "Direct link to max_agent_task_threads_num") * Default: 4096 * Type: Int * Unit: - * Is mutable: No * Description: The maximum number of threads that are allowed in the agent task thread pool. * Introduced in: - ##### `max_download_task_per_be`[​](#max_download_task_per_be "Direct link to max_download_task_per_be") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: In each RESTORE operation, the maximum number of download tasks StarRocks assigned to a BE node. When this item is set to less than or equal to 0, no limit is imposed on the task number. * Introduced in: v3.1.0 ##### `max_mv_check_base_table_change_retry_times`[​](#max_mv_check_base_table_change_retry_times "Direct link to max_mv_check_base_table_change_retry_times") * Default: 10 * Type: - * Unit: - * Is mutable: Yes * Description: The maximum retry times for detecting base table change when refreshing materialized views. * Introduced in: v3.3.0 ##### `max_mv_refresh_failure_retry_times`[​](#max_mv_refresh_failure_retry_times "Direct link to max_mv_refresh_failure_retry_times") * Default: 1 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum retry times when materialized view fails to refresh. * Introduced in: v3.3.0 ##### `max_mv_refresh_try_lock_failure_retry_times`[​](#max_mv_refresh_try_lock_failure_retry_times "Direct link to max_mv_refresh_try_lock_failure_retry_times") * Default: 3 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum retry times of try lock when materialized view fails to refresh. * Introduced in: v3.3.0 ##### `max_small_file_number`[​](#max_small_file_number "Direct link to max_small_file_number") * Default: 100 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of small files that can be stored on an FE directory. * Introduced in: - ##### `max_small_file_size_bytes`[​](#max_small_file_size_bytes "Direct link to max_small_file_size_bytes") * Default: 1024 \* 1024 * Type: Int * Unit: Bytes * Is mutable: Yes * Description: The maximum size of a small file. * Introduced in: - ##### `max_upload_task_per_be`[​](#max_upload_task_per_be "Direct link to max_upload_task_per_be") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: In each BACKUP operation, the maximum number of upload tasks StarRocks assigned to a BE node. When this item is set to less than or equal to 0, no limit is imposed on the task number. * Introduced in: v3.1.0 ##### `mv_create_partition_batch_interval_ms`[​](#mv_create_partition_batch_interval_ms "Direct link to mv_create_partition_batch_interval_ms") * Default: 1000 * Type: Int * Unit: ms * Is mutable: Yes * Description: During materialized view refresh, if multiple partitions need to be created in bulk, the system divides them into batches of 64 partitions each. To reduce the risk of failures caused by frequent partition creation, a default interval (in milliseconds) is set between each batch to control the creation frequency. * Introduced in: v3.3 ##### `mv_plan_cache_max_size`[​](#mv_plan_cache_max_size "Direct link to mv_plan_cache_max_size") * Default: 1000 * Type: Long * Unit: * Is mutable: Yes * Description: The maximum size of materialized view plan cache (which is used for materialized view rewrite). If there are many materialized views used for transparent query rewrite, you may increase this value. * Introduced in: v3.2 ##### `mv_plan_cache_thread_pool_size`[​](#mv_plan_cache_thread_pool_size "Direct link to mv_plan_cache_thread_pool_size") * Default: 8 * Type: Int * Unit: - * Is mutable: Yes * Description: The default thread pool size of materialized view plan cache (which is used for materialized view rewrite). * Introduced in: v3.2 ##### `mv_refresh_default_planner_optimize_timeout`[​](#mv_refresh_default_planner_optimize_timeout "Direct link to mv_refresh_default_planner_optimize_timeout") * Default: 30000 * Type: - * Unit: - * Is mutable: Yes * Description: The default timeout for the planning phase of the optimizer when refresh materialized views. * Introduced in: v3.3.0 ##### `mv_refresh_fail_on_filter_data`[​](#mv_refresh_fail_on_filter_data "Direct link to mv_refresh_fail_on_filter_data") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Mv refresh fails if there is filtered data in refreshing, true by default, otherwise return success by ignoring the filtered data. * Introduced in: - ##### `mv_refresh_try_lock_timeout_ms`[​](#mv_refresh_try_lock_timeout_ms "Direct link to mv_refresh_try_lock_timeout_ms") * Default: 30000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: The default try lock timeout for materialized view refresh to try the DB lock of its base table/materialized view. * Introduced in: v3.3.0 ##### `oauth2_auth_server_url`[​](#oauth2_auth_server_url "Direct link to oauth2_auth_server_url") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The authorization URL. The URL to which the users’ browser will be redirected in order to begin the OAuth 2.0 authorization process. * Introduced in: v3.5.0 ##### `oauth2_client_id`[​](#oauth2_client_id "Direct link to oauth2_client_id") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The public identifier of the StarRocks client. * Introduced in: v3.5.0 ##### `oauth2_client_secret`[​](#oauth2_client_secret "Direct link to oauth2_client_secret") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The secret used to authorize StarRocks client with the authorization server. * Introduced in: v3.5.0 ##### `oauth2_jwks_url`[​](#oauth2_jwks_url "Direct link to oauth2_jwks_url") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The URL to the JSON Web Key Set (JWKS) service or the path to the local file under the `conf` directory. * Introduced in: v3.5.0 ##### `oauth2_principal_field`[​](#oauth2_principal_field "Direct link to oauth2_principal_field") * Default: sub * Type: String * Unit: - * Is mutable: No * Description: The string used to identify the field that indicates the subject (`sub`) in the JWT. The default value is `sub`. The value of this field must be identical with the username for logging in to StarRocks. * Introduced in: v3.5.0 ##### `oauth2_redirect_url`[​](#oauth2_redirect_url "Direct link to oauth2_redirect_url") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The URL to which the users’ browser will be redirected after the OAuth 2.0 authentication succeeds. The authorization code will be sent to this URL. In most cases, it need to be configured as `http://:/api/oauth2`. * Introduced in: v3.5.0 ##### `oauth2_required_audience`[​](#oauth2_required_audience "Direct link to oauth2_required_audience") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The list of strings used to identify the audience (`aud`) in the JWT. The JWT is considered valid only if one of the values in the list match the JWT audience. * Introduced in: v3.5.0 ##### `oauth2_required_issuer`[​](#oauth2_required_issuer "Direct link to oauth2_required_issuer") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The list of strings used to identify the issuers (`iss`) in the JWT. The JWT is considered valid only if one of the values in the list match the JWT issuer. * Introduced in: v3.5.0 ##### `oauth2_token_server_url`[​](#oauth2_token_server_url "Direct link to oauth2_token_server_url") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The URL of the endpoint on the authorization server from which StarRocks obtains the access token. * Introduced in: v3.5.0 ##### `plugin_dir`[​](#plugin_dir "Direct link to plugin_dir") * Default: `System.getenv("STARROCKS_HOME")` + "/plugins" * Type: String * Unit: - * Is mutable: No * Description: The directory that stores plugin installation packages. * Introduced in: - ##### `plugin_enable`[​](#plugin_enable "Direct link to plugin_enable") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether plugins can be installed on FEs. Plugins can be installed or uninstalled only on the Leader FE. * Introduced in: - ##### `proc_profile_jstack_depth`[​](#proc_profile_jstack_depth "Direct link to proc_profile_jstack_depth") * Default: 128 * Type: Int * Unit: - * Is mutable: Yes * Description: Maximum Java stack depth when the system collects CPU and memory profiles. This value controls how many Java stack frames are captured for each sampled stack: larger values increase trace detail and output size and may add profiling overhead, while smaller values reduce details. This setting is used when the profiler is started for both CPU and memory profiling, so adjust it to balance diagnostic needs and performance impact. * Introduced in: - ##### `proc_profile_mem_enable`[​](#proc_profile_mem_enable "Direct link to proc_profile_mem_enable") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable collection of process memory allocation profiles. When this item is set to `true`, the system generates an HTML profile named `mem-profile-.html` under `sys_log_dir/proc_profile`, sleeps for `proc_profile_collect_time_s` seconds while sampling, and uses `proc_profile_jstack_depth` for Java stack depth. Generated files are compressed and purged according to `proc_profile_file_retained_days` and `proc_profile_file_retained_size_bytes`. The native extraction path uses `STARROCKS_HOME_DIR` to avoid `/tmp` noexec issues. This item is intended for troubleshooting memory-allocation hotspots. Enabling it increases CPU, I/O and disk usage and may produce large files. * Introduced in: v3.2.12 ##### `query_detail_explain_level`[​](#query_detail_explain_level "Direct link to query_detail_explain_level") * Default: COSTS * Type: String * Unit: - * Is mutable: true * Description: The detail level of query plan returned by the EXPLAIN statement. Valid values: COSTS, NORMAL, VERBOSE. * Introduced in: v3.2.12, v3.3.5 ##### `replication_interval_ms`[​](#replication_interval_ms "Direct link to replication_interval_ms") * Default: 100 * Type: Int * Unit: - * Is mutable: No * Description: The minimum time interval at which the replication tasks are scheduled. * Introduced in: v3.3.5 ##### `replication_max_parallel_data_size_mb`[​](#replication_max_parallel_data_size_mb "Direct link to replication_max_parallel_data_size_mb") * Default: 1048576 * Type: Int * Unit: MB * Is mutable: Yes * Description: The maximum size of data allowed for concurrent synchronization. * Introduced in: v3.3.5 ##### `replication_max_parallel_replica_count`[​](#replication_max_parallel_replica_count "Direct link to replication_max_parallel_replica_count") * Default: 10240 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of tablet replicas allowed for concurrent synchronization. * Introduced in: v3.3.5 ##### `replication_max_parallel_table_count`[​](#replication_max_parallel_table_count "Direct link to replication_max_parallel_table_count") * Default: 100 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of concurrent data synchronization tasks allowed. StarRocks creates one synchronization task for each table. * Introduced in: v3.3.5 ##### `replication_transaction_timeout_sec`[​](#replication_transaction_timeout_sec "Direct link to replication_transaction_timeout_sec") * Default: 86400 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The timeout duration for synchronization tasks. * Introduced in: v3.3.5 ##### `skip_whole_phase_lock_mv_limit`[​](#skip_whole_phase_lock_mv_limit "Direct link to skip_whole_phase_lock_mv_limit") * Default: 5 * Type: Int * Unit: - * Is mutable: Yes * Description: Controls when StarRocks applies the "non-lock" optimization for tables that have related materialized views. When this item is set to less than 0, the system always applies non-lock optimization and does not copy related materialized views for queries (FE memory usage and metadata copy/lock contention is reduced but risk of metadata concurrency issues can be increased). When it is set to 0, non-lock optimization is disable (the system always use the safe, copy-and-lock path). When it is set to greater than 0, non-lock optimization is applied only for tables whose number of related materialized views is less than or equal to the configured threshold. Additionally, when the value is greater than and equal to 0, the planner records query OLAP tables into the optimizer context to enable materialized view-related rewrite paths; when it is less than 0, this step is skipped. * Introduced in: v3.2.1 ##### `small_file_dir`[​](#small_file_dir "Direct link to small_file_dir") * Default: `StarRocksFE.STARROCKS_HOME_DIR` + "/small\_files" * Type: String * Unit: - * Is mutable: No * Description: The root directory of small files. * Introduced in: - ##### `task_runs_max_history_number`[​](#task_runs_max_history_number "Direct link to task_runs_max_history_number") * Default: 10000 * Type: Int * Unit: - * Is mutable: Yes * Description: Maximum number of task run records to retain in memory and to use as a default LIMIT when querying archived task-run history. When `enable_task_history_archive` is false, this value bounds in-memory history: Force GC trims older entries so only the newest `task_runs_max_history_number` remain. When archive history is queried (and no explicit LIMIT is provided), `TaskRunHistoryTable.lookup` uses `"ORDER BY create_time DESC LIMIT "` if this value is greater than 0. Note: setting this to 0 disables the query-side LIMIT (no cap) but will cause in-memory history to be truncated to zero (unless archiving is enabled). * Introduced in: v3.2.0 ##### `tmp_dir`[​](#tmp_dir "Direct link to tmp_dir") * Default: `StarRocksFE.STARROCKS_HOME_DIR` + "/temp\_dir" * Type: String * Unit: - * Is mutable: No * Description: The directory that stores temporary files such as files generated during backup and restore procedures. After these procedures finish, the generated temporary files are deleted. * Introduced in: - ##### `transform_type_prefer_string_for_varchar`[​](#transform_type_prefer_string_for_varchar "Direct link to transform_type_prefer_string_for_varchar") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to prefer string type for fixed length char/varchar columns in materialized view creation. * Introduced in: v4.0.0 --- ### FE Configuration - Statistics and Storage FE parameters are classified into dynamic parameters and static parameters. * Dynamic parameters can be configured and adjusted by running SQL commands, which is very convenient. But the configurations become invalid if you restart your FE. Therefore, we recommend that you also modify the configuration items in the **fe.conf** file to prevent the loss of modifications. * Static parameters can only be configured and adjusted in the FE configuration file **fe.conf**. **After you modify this file, you must restart your FE for the changes to take effect.** Whether a parameter is a dynamic parameter is indicated by the `IsMutable` column in the output of [ADMIN SHOW CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SHOW_CONFIG.md). `TRUE` indicates a dynamic parameter. Note that both dynamic and static FE parameters can be configured in the **fe.conf** file. #### View FE configuration items[​](#view-fe-configuration-items "Direct link to View FE configuration items") After your FE is started, you can run the ADMIN SHOW FRONTEND CONFIG command on your MySQL client to check the parameter configurations. If you want to query the configuration of a specific parameter, run the following command: ```sql ADMIN SHOW FRONTEND CONFIG [LIKE "pattern"]; ``` For detailed description of the returned fields, see [`ADMIN SHOW CONFIG`](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SHOW_CONFIG.md). note You must have administrator privileges to run cluster administration-related commands. #### Configure FE parameters[​](#configure-fe-parameters "Direct link to Configure FE parameters") ##### Configure FE dynamic parameters[​](#configure-fe-dynamic-parameters "Direct link to Configure FE dynamic parameters") You can configure or modify the settings of FE dynamic parameters using [`ADMIN SET FRONTEND CONFIG`](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md). ```sql ADMIN SET FRONTEND CONFIG ("key" = "value"); ``` note The configuration changes made with `ADMIN SET FRONTEND` will be restored to the default values in the `fe.conf` file after the FE restarts. Therefore, we recommend that you also modify the configuration items in `fe.conf` if you want the changes to be permanent. ##### Configure FE static parameters[​](#configure-fe-static-parameters "Direct link to Configure FE static parameters") note Static parameters of an FE are set by changing them in the configuration file **fe.conf** and restarting the FE to allow the changes to take effect. *** This topic introduces the following types of FE configurations: * [Statistic report](#statistic-report) * [Storage](#storage) #### Statistic report[​](#statistic-report "Direct link to Statistic report") ##### `enable_collect_warehouse_metrics`[​](#enable_collect_warehouse_metrics "Direct link to enable_collect_warehouse_metrics") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: When this item is set to `true`, the system will collect and export per-warehouse metrics. Enabling it adds warehouse-level metrics (slot/usage/availability) to the metric output and increases metric cardinality and collection overhead. Disable it to omit warehouse-specific metrics and reduce CPU/network and monitoring storage cost. * Introduced in: v3.5.0 ##### `enable_http_detail_metrics`[​](#enable_http_detail_metrics "Direct link to enable_http_detail_metrics") * Default: false * Type: boolean * Unit: - * Is mutable: Yes * Description: When true, the HTTP server computes and exposes detailed HTTP worker metrics (notably the `HTTP_WORKER_PENDING_TASKS_NUM` gauge). Enabling this causes the server to iterate over Netty worker executors and call `pendingTasks()` on each `NioEventLoop` to sum pending task counts; when disabled the gauge returns 0 to avoid that cost. This extra collection can be CPU- and latency-sensitive — enable only for debugging or detailed investigation. * Introduced in: v3.2.3 ##### `proc_profile_collect_time_s`[​](#proc_profile_collect_time_s "Direct link to proc_profile_collect_time_s") * Default: 120 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Duration in seconds for a single process profile collection. When `proc_profile_cpu_enable` or `proc_profile_mem_enable` is set to `true`, AsyncProfiler is started, the collector thread sleeps for this duration, then the profiler is stopped and the profile is written. Larger values increase sample coverage and file size but prolong profiler runtime and delay subsequent collections; smaller values reduce overhead but may produce insufficient samples. Ensure this value aligns with retention settings such as `proc_profile_file_retained_days` and `proc_profile_file_retained_size_bytes`. * Introduced in: v3.2.12 ##### `enable_external_predicate_columns_collection`[​](#enable_external_predicate_columns_collection "Direct link to enable_external_predicate_columns_collection") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to record predicate column usage (columns used in WHERE/JOIN/GROUP BY) for external (non-native) tables during query optimization. StarRocks uses this usage information to narrow down which columns ANALYZE collects statistics for on wide external tables. When disabled, external table predicate columns are not recorded, and ANALYZE falls back to collecting statistics for all columns. * Introduced in: v4.2.0 ##### `statistic_external_predicate_columns_ttl_hours`[​](#statistic_external_predicate_columns_ttl_hours "Direct link to statistic_external_predicate_columns_ttl_hours") * Default: 168 * Type: Long * Unit: Hours * Is mutable: Yes * Description: The time-to-live (TTL) of recorded external table predicate column usage. Entries whose `last_used` timestamp is older than this value are removed by the periodic vacuum job. Set to a negative value (e.g. -1) to disable vacuum. Defaults to a week because external table ANALYZE runs far less frequently than for internal tables, so a short TTL (matching the internal table's 24-hour default) would evict usage information between two collections. * Introduced in: v4.2.0 ##### `statistic_external_predicate_columns_cache_ttl_sec`[​](#statistic_external_predicate_columns_cache_ttl_sec "Direct link to statistic_external_predicate_columns_cache_ttl_sec") * Default: 300 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The TTL of the in-memory cache that serves external table predicate column queries (for example, during automatic ANALYZE column selection). A shorter value makes newly recorded usage visible sooner but increases the query load on the underlying storage table; a longer value reduces that load at the cost of staleness. * Introduced in: v4.2.0 #### Storage[​](#storage "Direct link to Storage") ##### `alter_table_timeout_second`[​](#alter_table_timeout_second "Direct link to alter_table_timeout_second") * Default: 86400 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The timeout duration for the schema change operation (ALTER TABLE). * Introduced in: - ##### `capacity_used_percent_high_water`[​](#capacity_used_percent_high_water "Direct link to capacity_used_percent_high_water") * Default: 0.75 * Type: double * Unit: Fraction (0.0–1.0) * Is mutable: Yes * Description: The high-water threshold of disk capacity used percent (fraction of total capacity) used when computing backend load scores. `BackendLoadStatistic.calcSore` uses `capacity_used_percent_high_water` to set `LoadScore.capacityCoefficient`: if a backend's used percent less than 0.5 the coefficient equal to 0.5; if used percent `>` `capacity_used_percent_high_water` the coefficient = 1.0; otherwise the coefficient transitions linearly with used percent via (2 \* usedPercent - 0.5). When the coefficient is 1.0, the load score is driven entirely by capacity proportion; lower values increase the weight of replica count. Adjusting this value changes how aggressively the balancer penalizes backends with high disk utilization. * Introduced in: v3.2.0 ##### `catalog_trash_expire_second`[​](#catalog_trash_expire_second "Direct link to catalog_trash_expire_second") * Default: 86400 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The longest duration the metadata can be retained after a database, table, or partition is dropped. If this duration expires, the data will be deleted and cannot be recovered through the RECOVER command. * Introduced in: - ##### `catalog_recycle_bin_erase_min_latency_ms`[​](#catalog_recycle_bin_erase_min_latency_ms "Direct link to catalog_recycle_bin_erase_min_latency_ms") * Default: 600000 * Type: Long * Unit: Milliseconds * Is mutable: Yes * Description: The minimum delay in milliseconds before the metadata is erased when a database, table, or partition is dropped. This avoids the erase log being written ahead of the drop log. * Introduced in: - ##### `catalog_recycle_bin_erase_max_operations_per_cycle`[​](#catalog_recycle_bin_erase_max_operations_per_cycle "Direct link to catalog_recycle_bin_erase_max_operations_per_cycle") * Default: 500 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of erase operations per cycle for actually deleting databases, tables, or partitions from the recycle bin. The erase operation holds a lock, so one batch should not be too large. * Introduced in: - ##### `catalog_recycle_bin_erase_fail_retry_interval_ms`[​](#catalog_recycle_bin_erase_fail_retry_interval_ms "Direct link to catalog_recycle_bin_erase_fail_retry_interval_ms") * Default: 60000 * Type: Long * Unit: Milliseconds * Is mutable: Yes * Description: The retry interval in milliseconds when an erase operation in the recycle bin fails. * Introduced in: - ##### `check_consistency_default_timeout_second`[​](#check_consistency_default_timeout_second "Direct link to check_consistency_default_timeout_second") * Default: 600 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The timeout duration for a replica consistency check. You can set this parameter based on the size of your tablet. * Introduced in: - ##### `consistency_check_cooldown_time_second`[​](#consistency_check_cooldown_time_second "Direct link to consistency_check_cooldown_time_second") * Default: 24 \* 3600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Controls the minimal interval (in seconds) required between consistency checks of the same tablet. During tablet selection, a tablet is considered eligible only if `tablet.getLastCheckTime()` is less than `(currentTimeMillis - consistency_check_cooldown_time_second * 1000)`. The default value (24 \* 3600) enforces roughly one check per tablet per day to reduce backend disk I/O. Lowering this value increases check frequency and resource usage; raising it reduces I/O at the cost of slower detection of inconsistencies. The value is applied globally when filtering cooldowned tablets from an index's tablet list. * Introduced in: v3.5.5 ##### `consistency_check_end_time`[​](#consistency_check_end_time "Direct link to consistency_check_end_time") * Default: "4" * Type: String * Unit: Hour of day (0-23) * Is mutable: No * Description: Specifies the end hour (hour-of-day) of the ConsistencyChecker work window. The value is parsed with SimpleDateFormat("HH") in the system time zone and accepted as 0–23 (single or two-digit). StarRocks uses it with `consistency_check_start_time` to decide when to schedule and add consistency-check jobs. When `consistency_check_start_time` is greater than `consistency_check_end_time`, the window spans midnight (for example, default is `consistency_check_start_time` = "23" to `consistency_check_end_time` = "4"). When `consistency_check_start_time` is equal to `consistency_check_end_time`, the checker never runs. Parsing failure will cause FE startup to log an error and exit, so provide a valid hour string. * Introduced in: v3.2.0 ##### `consistency_check_start_time`[​](#consistency_check_start_time "Direct link to consistency_check_start_time") * Default: "23" * Type: String * Unit: Hour of day (00-23) * Is mutable: No * Description: Specifies the start hour (hour-of-day) of the ConsistencyChecker work window. The value is parsed with SimpleDateFormat("HH") in the system time zone and accepted as 0–23 (single or two-digit). StarRocks uses it with `consistency_check_end_time` to decide when to schedule and add consistency-check jobs. When `consistency_check_start_time` is greater than `consistency_check_end_time`, the window spans midnight (for example, default is `consistency_check_start_time` = "23" to `consistency_check_end_time` = "4"). When `consistency_check_start_time` is equal to `consistency_check_end_time`, the checker never runs. Parsing failure will cause FE startup to log an error and exit, so provide a valid hour string. * Introduced in: v3.2.0 ##### `consistency_tablet_meta_check_interval_ms`[​](#consistency_tablet_meta_check_interval_ms "Direct link to consistency_tablet_meta_check_interval_ms") * Default: 2 \* 3600 \* 1000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: Interval used by the ConsistencyChecker to run a full tablet-meta consistency scan between `TabletInvertedIndex` and `LocalMetastore`. The daemon in `runAfterCatalogReady` triggers checkTabletMetaConsistency when `current time - lastTabletMetaCheckTime` exceeds this value. When an invalid tablet is first detected, its `toBeCleanedTime` is set to `now + (consistency_tablet_meta_check_interval_ms / 2)` so actual deletion is delayed until a subsequent scan. Increase this value to reduce scan frequency and load (slower cleanup); decrease it to detect and remove stale tablets faster (higher overhead). * Introduced in: v3.2.0 ##### `default_replication_num`[​](#default_replication_num "Direct link to default_replication_num") * Default: 3 * Type: Short * Unit: - * Is mutable: Yes * Description: Sets the default number of replicas for each data partition when creating a table in StarRocks. This setting can be overridden when creating a table by specifying `replication_num=x` in the CREATE TABLE DDL. * Introduced in: - ##### `enable_auto_tablet_distribution`[​](#enable_auto_tablet_distribution "Direct link to enable_auto_tablet_distribution") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to automatically set the number of buckets. * If this parameter is set to `TRUE`, you don't need to specify the number of buckets when you create a table or add a partition. StarRocks automatically determines the number of buckets. * If this parameter is set to `FALSE`, you need to manually specify the number of buckets when you create a table or add a partition. If you do not specify the bucket count when adding a new partition to a table, the new partition inherits the bucket count set at the creation of the table. However, you can also manually specify the number of buckets for the new partition. * Introduced in: v2.5.7 ##### `enable_experimental_rowstore`[​](#enable_experimental_rowstore "Direct link to enable_experimental_rowstore") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable the hybrid row-column storage feature. * Introduced in: v3.2.3 ##### `enable_fast_schema_evolution`[​](#enable_fast_schema_evolution "Direct link to enable_fast_schema_evolution") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable Fast Schema Evolution for all tables within the StarRocks cluster. Valid values are `TRUE` and `FALSE` (default). Enabling Fast Schema Evolution can increase the speed of schema changes and reduce resource usage when columns are added or dropped. * Introduced in: v3.2.0 > **NOTE** > > * StarRocks shared-data clusters supports this parameter from v3.3.0. > * If you need to configure the Fast Schema Evolution for a specific table, such as disabling Fast Schema Evolution for a specific table, you can set the table property `fast_schema_evolution` at table creation. ##### `enable_online_optimize_table`[​](#enable_online_optimize_table "Direct link to enable_online_optimize_table") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Controls whether StarRocks will use the non-blocking online optimization path when creating an optimize job. When `enable_online_optimize_table` is true and the target table meets compatibility checks (no partition/keys/sort specification, distribution is not `RandomDistributionDesc`, storage type is not `COLUMN_WITH_ROW`, replicated storage enabled, and the table is not a cloud-native table or materialized view), the planner creates an `OnlineOptimizeJobV2` to perform optimization without blocking writes. If false or any compatibility condition fails, StarRocks falls back to `OptimizeJobV2`, which may block write operations during optimization. * Introduced in: v3.3.3, v3.4.0, v3.5.0 ##### `enable_strict_storage_medium_check`[​](#enable_strict_storage_medium_check "Direct link to enable_strict_storage_medium_check") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether the FE strictly checks the storage medium of BEs when users create tables. If this parameter is set to `TRUE`, the FE checks the storage medium of BEs when users create tables and returns an error if the storage medium of the BE is different from the `storage_medium` parameter specified in the CREATE TABLE statement. For example, the storage medium specified in the CREATE TABLE statement is SSD but the actual storage medium of BEs is HDD. As a result, the table creation fails. If this parameter is `FALSE`, the FE does not check the storage medium of BEs when users create a table. * Introduced in: - ##### `max_bucket_number_per_partition`[​](#max_bucket_number_per_partition "Direct link to max_bucket_number_per_partition") * Default: 1024 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of buckets can be created in a partition. * Introduced in: v3.3.2 ##### `max_column_number_per_table`[​](#max_column_number_per_table "Direct link to max_column_number_per_table") * Default: 10000 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of columns can be created in a table. * Introduced in: v3.3.2 ##### `max_dynamic_partition_num`[​](#max_dynamic_partition_num "Direct link to max_dynamic_partition_num") * Default: 500 * Type: Int * Unit: - * Is mutable: Yes * Description: Limits the maximum number of partitions that can be created at once when analyzing or creating a dynamic-partitioned table. During dynamic partition property validation, the `systemtask_runs_max_history_number` computes expected partitions (end offset + history partition number) and throws a DDL error if that total exceeds `max_dynamic_partition_num`. Raise this value only when you expect legitimately large partition ranges; increasing it allows more partitions to be created but can increase metadata size, scheduling work, and operational complexity. * Introduced in: v3.2.0 ##### `max_partition_number_per_table`[​](#max_partition_number_per_table "Direct link to max_partition_number_per_table") * Default: 100000 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of partitions can be created in a table. * Introduced in: v3.3.2 ##### `max_task_consecutive_fail_count`[​](#max_task_consecutive_fail_count "Direct link to max_task_consecutive_fail_count") * Default: 10 * Type: Int * Unit: - * Is mutable: Yes * Description: Maximum number of consecutive failures a task may have before the scheduler automatically suspends it. When `TaskSource.MV.equals(task.getSource())` and `max_task_consecutive_fail_count` are greater than 0, if a task's consecutive failure counter reaches or exceeds `max_task_consecutive_fail_count`, the task is suspended via the TaskManager and, for materialized-view tasks, the materialized view is inactivated. An exception is thrown indicating suspension and how to reactivate (for example, `ALTER MATERIALIZED VIEW ACTIVE`). Set this item to 0 or a negative value to disable automatic suspension. * Introduced in: - ##### `partition_recycle_retention_period_secs`[​](#partition_recycle_retention_period_secs "Direct link to partition_recycle_retention_period_secs") * Default: 1800 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The metadata retention time for the partition that is dropped by INSERT OVERWRITE or materialized view refresh operations. Note that such metadata cannot be recovered by executing RECOVER. * Introduced in: v3.5.9 ##### `recover_with_empty_tablet`[​](#recover_with_empty_tablet "Direct link to recover_with_empty_tablet") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to replace a lost or corrupted tablet replica with an empty one. If a tablet replica is lost or corrupted, data queries on this tablet or other healthy tablets may fail. Replacing the lost or corrupted tablet replica with an empty tablet ensures that the query can still be executed. However, the result may be incorrect because data is lost. The default value is `FALSE`, which means lost or corrupted tablet replicas are not replaced with empty ones, and the query fails. * Introduced in: - ##### `storage_usage_hard_limit_percent`[​](#storage_usage_hard_limit_percent "Direct link to storage_usage_hard_limit_percent") * Default: 95 * Alias: `storage_flood_stage_usage_percent` * Type: Int * Unit: - * Is mutable: Yes * Description: Hard limit of the storage usage percentage in a BE directory. If the storage usage (in percentage) of the BE storage directory exceeds this value and the remaining storage space is less than `storage_usage_hard_limit_reserve_bytes`, Load and Restore jobs are rejected. You need to set this item together with the BE configuration item `storage_flood_stage_usage_percent` to allow the configurations to take effect. * Introduced in: - ##### `storage_usage_hard_limit_reserve_bytes`[​](#storage_usage_hard_limit_reserve_bytes "Direct link to storage_usage_hard_limit_reserve_bytes") * Default: 100 \* 1024 \* 1024 \* 1024 * Alias: `storage_flood_stage_left_capacity_bytes` * Type: Long * Unit: Bytes * Is mutable: Yes * Description: Hard limit of the remaining storage space in a BE directory. If the remaining storage space in the BE storage directory is less than this value and the storage usage (in percentage) exceeds `storage_usage_hard_limit_percent`, Load and Restore jobs are rejected. You need to set this item together with the BE configuration item `storage_flood_stage_left_capacity_bytes` to allow the configurations to take effect. * Introduced in: - ##### `storage_usage_soft_limit_percent`[​](#storage_usage_soft_limit_percent "Direct link to storage_usage_soft_limit_percent") * Default: 90 * Alias: `storage_high_watermark_usage_percent` * Type: Int * Unit: - * Is mutable: Yes * Description: Soft limit of the storage usage percentage in a BE directory. If the storage usage (in percentage) of the BE storage directory exceeds this value and the remaining storage space is less than `storage_usage_soft_limit_reserve_bytes`, tablets cannot be cloned into this directory. * Introduced in: - ##### `storage_usage_soft_limit_reserve_bytes`[​](#storage_usage_soft_limit_reserve_bytes "Direct link to storage_usage_soft_limit_reserve_bytes") * Default: 200 \* 1024 \* 1024 \* 1024 * Alias: `storage_min_left_capacity_bytes` * Type: Long * Unit: Bytes * Is mutable: Yes * Description: Soft limit of the remaining storage space in a BE directory. If the remaining storage space in the BE storage directory is less than this value and the storage usage (in percentage) exceeds `storage_usage_soft_limit_percent`, tablets cannot be cloned into this directory. * Introduced in: - ##### `tablet_checker_lock_time_per_cycle_ms`[​](#tablet_checker_lock_time_per_cycle_ms "Direct link to tablet_checker_lock_time_per_cycle_ms") * Default: 1000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: The maximum lock hold time per cycle for tablet checker before releasing and reacquiring the table lock. Values less than 100 will be treated as 100. * Introduced in: v3.5.9, v4.0.2 ##### `tablet_create_timeout_second`[​](#tablet_create_timeout_second "Direct link to tablet_create_timeout_second") * Default: 10 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The timeout duration for creating a tablet. The default value is changed from 1 to 10 from v3.1 onwards. * Introduced in: - ##### `tablet_delete_timeout_second`[​](#tablet_delete_timeout_second "Direct link to tablet_delete_timeout_second") * Default: 2 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The timeout duration for deleting a tablet. * Introduced in: - ##### `tablet_sched_balance_load_disk_safe_threshold`[​](#tablet_sched_balance_load_disk_safe_threshold "Direct link to tablet_sched_balance_load_disk_safe_threshold") * Default: 0.5 * Alias: `balance_load_disk_safe_threshold` * Type: Double * Unit: - * Is mutable: Yes * Description: The percentage threshold for determining whether the disk usage of BEs is balanced. If the disk usage of all BEs is lower than this value, it is considered balanced. If the disk usage is greater than this value and the difference between the highest and lowest BE disk usage is greater than 10%, the disk usage is considered unbalanced and a tablet re-balancing is triggered. * Introduced in: - ##### `tablet_sched_balance_load_score_threshold`[​](#tablet_sched_balance_load_score_threshold "Direct link to tablet_sched_balance_load_score_threshold") * Default: 0.1 * Alias: `balance_load_score_threshold` * Type: Double * Unit: - * Is mutable: Yes * Description: The percentage threshold for determining whether the load of a BE is balanced. If a BE has a lower load than the average load of all BEs and the difference is greater than this value, this BE is in a low load state. On the contrary, if a BE has a higher load than the average load and the difference is greater than this value, this BE is in a high load state. * Introduced in: - ##### `tablet_sched_be_down_tolerate_time_s`[​](#tablet_sched_be_down_tolerate_time_s "Direct link to tablet_sched_be_down_tolerate_time_s") * Default: 900 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The maximum duration the scheduler allows for a BE node to remain inactive. After the time threshold is reached, tablets on that BE node will be migrated to other active BE nodes. * Introduced in: v2.5.7 ##### `tablet_sched_disable_balance`[​](#tablet_sched_disable_balance "Direct link to tablet_sched_disable_balance") * Default: false * Alias: `disable_balance` * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to disable tablet balancing. `TRUE` indicates that tablet balancing is disabled. `FALSE` indicates that tablet balancing is enabled. * Introduced in: - ##### `tablet_sched_disable_colocate_balance`[​](#tablet_sched_disable_colocate_balance "Direct link to tablet_sched_disable_colocate_balance") * Default: false * Alias: `disable_colocate_balance` * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to disable replica balancing for Colocate Table. `TRUE` indicates replica balancing is disabled. `FALSE` indicates replica balancing is enabled. * Introduced in: - ##### `tablet_sched_max_balancing_tablets`[​](#tablet_sched_max_balancing_tablets "Direct link to tablet_sched_max_balancing_tablets") * Default: 500 * Alias: `max_balancing_tablets` * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of tablets that can be balanced at the same time. If this value is exceeded, tablet re-balancing will be skipped. * Introduced in: - ##### `tablet_sched_max_clone_task_timeout_sec`[​](#tablet_sched_max_clone_task_timeout_sec "Direct link to tablet_sched_max_clone_task_timeout_sec") * Default: 2 \* 60 \* 60 * Alias: `max_clone_task_timeout_sec` * Type: Long * Unit: Seconds * Is mutable: Yes * Description :The maximum timeout duration for cloning a tablet. * Introduced in: - ##### `tablet_sched_max_not_being_scheduled_interval_ms`[​](#tablet_sched_max_not_being_scheduled_interval_ms "Direct link to tablet_sched_max_not_being_scheduled_interval_ms") * Default: 15 \* 60 \* 1000 * Type: Long * Unit: Milliseconds * Is mutable: Yes * Description: When the tablet clone tasks are being scheduled, if a tablet has not been scheduled for the specified time in this parameter, StarRocks gives it a higher priority to schedule it as soon as possible. * Introduced in: - ##### `tablet_sched_max_scheduling_tablets`[​](#tablet_sched_max_scheduling_tablets "Direct link to tablet_sched_max_scheduling_tablets") * Default: 10000 * Alias: `max_scheduling_tablets` * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of tablets that can be scheduled at the same time. If the value is exceeded, tablet balancing and repair checks will be skipped. * Introduced in: - ##### `tablet_sched_min_clone_task_timeout_sec`[​](#tablet_sched_min_clone_task_timeout_sec "Direct link to tablet_sched_min_clone_task_timeout_sec") * Default: 3 \* 60 * Alias: `min_clone_task_timeout_sec` * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The minimum timeout duration for cloning a tablet. * Introduced in: - ##### `tablet_sched_num_based_balance_threshold_ratio`[​](#tablet_sched_num_based_balance_threshold_ratio "Direct link to tablet_sched_num_based_balance_threshold_ratio") * Default: 0.5 * Alias: - * Type: Double * Unit: - * Is mutable: Yes * Description: Doing num based balance may break the disk size balance, but the maximum gap between disks cannot exceed `tablet_sched_num_based_balance_threshold_ratio` \* `tablet_sched_balance_load_score_threshold`. If there are tablets in the cluster that are constantly balancing from A to B and B to A, reduce this value. If you want the tablet distribution to be more balanced, increase this value. * Introduced in: - 3.1 ##### `tablet_sched_repair_delay_factor_second`[​](#tablet_sched_repair_delay_factor_second "Direct link to tablet_sched_repair_delay_factor_second") * Default: 60 * Alias: `tablet_repair_delay_factor_second` * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The interval at which replicas are repaired, in seconds. * Introduced in: - ##### `tablet_sched_slot_num_per_path`[​](#tablet_sched_slot_num_per_path "Direct link to tablet_sched_slot_num_per_path") * Default: 8 * Alias: `schedule_slot_num_per_path` * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of tablet-related tasks that can run concurrently in a BE storage directory. From v2.5 onwards, the default value of this parameter is changed from `4` to `8`. * Introduced in: - ##### `tablet_sched_storage_cooldown_second`[​](#tablet_sched_storage_cooldown_second "Direct link to tablet_sched_storage_cooldown_second") * Default: -1 * Alias: `storage_cooldown_second` * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The latency of automatic cooling starting from the time of table creation. The default value `-1` specifies that automatic cooling is disabled. If you want to enable automatic cooling, set this parameter to a value greater than `-1`. * Introduced in: - ##### `tablet_stat_update_interval_second`[​](#tablet_stat_update_interval_second "Direct link to tablet_stat_update_interval_second") * Default: 300 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time interval at which the FE retrieves tablet statistics from each BE. * Introduced in: - ##### `enable_range_distribution`[​](#enable_range_distribution "Direct link to enable_range_distribution") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable the Range-based Distribution semantic for table creation. * Introduced in: v4.1.0 ##### `tablet_reshard_max_parallel_tablets`[​](#tablet_reshard_max_parallel_tablets "Direct link to tablet_reshard_max_parallel_tablets") * Default: 10240 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of tablets that can be split or merged in parallel. * Introduced in: v4.1.0 ##### `tablet_reshard_target_size`[​](#tablet_reshard_target_size "Direct link to tablet_reshard_target_size") * Default: 10737418240 (10 GB) * Type: Int * Unit: Bytes * Is mutable: Yes * Description: The target size of the tablets after the SPLIT or MERGE operation. * Introduced in: v4.1.0 ##### `tablet_reshard_max_split_count`[​](#tablet_reshard_max_split_count "Direct link to tablet_reshard_max_split_count") * Default: 1024 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of new tablets that an old tablet can be split into. * Introduced in: v4.1.0 ##### `tablet_reshard_min_split_size`[​](#tablet_reshard_min_split_size "Direct link to tablet_reshard_min_split_size") * Default: 2147483648 (2 GB) * Type: Long * Unit: Bytes * Is mutable: Yes * Description: The minimum size of a tablet produced by tablet pre-split. It bounds compute-node alignment during pre-split so that a small load on a large cluster is not split into many tiny tablets. Should be no larger than `tablet_reshard_target_size`. * Introduced in: v4.1.0 ##### `tablet_reshard_history_job_max_keep_ms`[​](#tablet_reshard_history_job_max_keep_ms "Direct link to tablet_reshard_history_job_max_keep_ms") * Default: 259200000 (72 hours) * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: The maximum retention time of historical tablet SPLIT/MERGE jobs. * Introduced in: v4.1.0 ##### `enable_tablet_pre_split_for_insert_from_files`[​](#enable_tablet_pre_split_for_insert_from_files "Direct link to enable_tablet_pre_split_for_insert_from_files") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable Sample-Based Tablet Pre-Split for `INSERT INTO ... SELECT FROM FILES()` loads. On by default as of v4.1.0. Set to `false` to disable cluster-wide. The session variable `enable_tablet_pre_split` must also be `true` for pre-split to run. * Introduced in: v4.1.0 ##### `enable_tablet_pre_split_for_broker_load`[​](#enable_tablet_pre_split_for_broker_load "Direct link to enable_tablet_pre_split_for_broker_load") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable Sample-Based Tablet Pre-Split for Broker Load. On by default as of v4.1.0. Set to `false` to disable cluster-wide. The session variable `enable_tablet_pre_split` must also be `true` for pre-split to run. * Introduced in: v4.1.0 ##### `enable_tablet_pre_split_for_insert_from_table`[​](#enable_tablet_pre_split_for_insert_from_table "Direct link to enable_tablet_pre_split_for_insert_from_table") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable Sample-Based Tablet Pre-Split for `INSERT INTO ... SELECT FROM ` loads (INSERT-from-OLAP-table). On by default as of v4.1.0. Set to `false` to disable cluster-wide. The session variable `enable_tablet_pre_split` must also be `true` for pre-split to run. To roll back, set to `false`; new INSERT-from-table loads will skip pre-split immediately. * Introduced in: v4.1.0 ##### `tablet_pre_split_pre_submit_timeout_seconds`[​](#tablet_pre_split_pre_submit_timeout_seconds "Direct link to tablet_pre_split_pre_submit_timeout_seconds") * Default: 300 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: Wall-clock budget for the pre-submit phase of Sample-Based Tablet Pre-Split (sample + plan boundaries + build reshard job). On expiry the coordinator skips pre-split and the load proceeds against the original single tablet. Default 300s: the data-tier sampler can take tens of seconds on large datasets / slow object storage (a ~40GB many-file Parquet load sampled in ~78s in testing), and this budget mainly bites large loads — exactly the ones pre-split benefits; small loads sample in well under a second regardless. The load stays `PENDING` for at most this long during sampling, so keep it below the load's own timeout. * Introduced in: v4.1.0 ##### `tablet_pre_split_post_submit_wait_seconds`[​](#tablet_pre_split_post_submit_wait_seconds "Direct link to tablet_pre_split_post_submit_wait_seconds") * Default: 300 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: Maximum time the coordinator will wait for an admitted Sample-Based Tablet Pre-Split reshard job to reach `FINISHED`. Both INSERT-from-FILES and Broker Load synchronously wait and on expiry **proceed without abort** — the load then plans against the currently visible tablet layout (still the original layout if the daemon hasn't transitioned, or partially / fully post-split if the daemon raced past the wait); the `tablet_pre_split_post_submit_hard_cap` counter records the timeout. The strict `runPreSplit` wrapper used by tests aborts the calling load via `PreSplitPostSubmitTimeoutException`. For Broker Load the wait runs after the broker pending task resolves file statuses but before `beginTxn` opens `T_load` — it occupies a `pending_load_task_scheduler` thread for at most this many seconds per table, so size `max_broker_load_job_concurrency` accordingly when many concurrent Broker Loads target a pre-splittable layout. **Operator note:** the Broker Load remains `PENDING` in `SHOW LOAD` during the wait and is still subject to its own `timeoutSecond` — set this well below the smallest Broker Load timeout in normal use. * Introduced in: v4.1.0 ##### `tablet_pre_split_sample_byte_limit`[​](#tablet_pre_split_sample_byte_limit "Direct link to tablet_pre_split_sample_byte_limit") * Default: 16777216 (16 MiB) * Type: Long * Unit: Bytes * Is mutable: Yes * Description: Soft byte cap on the FE-side accumulation buffer of the data-tier reservoir sampler used by Sample-Based Tablet Pre-Split. The sampler stops reading once accumulated values exceed this limit. The first row is always admitted so an oversize row still produces a non-empty sample. * Introduced in: v4.1.0 ##### `tablet_pre_split_meta_tier_overlap_threshold`[​](#tablet_pre_split_meta_tier_overlap_threshold "Direct link to tablet_pre_split_meta_tier_overlap_threshold") * Default: 0.3 * Type: Double * Unit: - * Is mutable: Yes * Description: Maximum overlap fraction tolerated when Sample-Based Tablet Pre-Split's meta tier (Parquet/ORC row-group metadata) computes boundaries. Above this threshold the cumulative-row count stops being monotone in sorted-min order so meta tier falls back to data tier (row sampling). * Introduced in: v4.1.0 ##### `tablet_pre_split_max_partitions_per_load`[​](#tablet_pre_split_max_partitions_per_load "Direct link to tablet_pre_split_max_partitions_per_load") * Default: 32 * Type: Int * Unit: - * Is mutable: Yes * Description: Maximum number of predicted target partitions a single Sample-Based Tablet Pre-Split invocation will operate on. Excess predicted partitions (those with the lowest sample count) are dropped and fall back to runtime auto-create with no pre-split. Bounds hook latency on pathological multi-partition loads. Set to zero or a negative value to disable the cap. * Introduced in: v4.1.0 ###### Rolling back Sample-Based Tablet Pre-Split[​](#rolling-back-sample-based-tablet-pre-split "Direct link to Rolling back Sample-Based Tablet Pre-Split") To disable the feature safely before a downgrade or during a production rollback: 1. Set all three pre-split flags to `false`: `enable_tablet_pre_split_for_insert_from_files`, `enable_tablet_pre_split_for_broker_load`, and `enable_tablet_pre_split_for_insert_from_table`. New loads will skip pre-split immediately. 2. Wait for in-flight reshard jobs created by pre-split to drain. Monitor with `SHOW TABLET RESHARD JOB`; the rollback is complete once no `RUNNING` or `PENDING` rows remain. 3. Proceed with the downgrade. The substrate (External-Boundaries Tablet Split) remains available regardless of the pre-split feature flag. ###### Behavioral notes for multi-partition Sample-Based Tablet Pre-Split (P2-a)[​](#behavioral-notes-for-multi-partition-sample-based-tablet-pre-split-p2-a "Direct link to Behavioral notes for multi-partition Sample-Based Tablet Pre-Split (P2-a)") The multi-partition path extends Sample-Based Tablet Pre-Split to loads that target many partitions in one statement. Two operational caveats apply: * **Broker Load triggering-load asymmetry.** The multi-partition pre-split hook fires from `BrokerLoadJob.createLoadingTask` **after** `task.prepare()` has built the load's sink plan against the catalog as it existed at that moment. For Broker Load, pre-created partitions and the post-reshard tablet layout are therefore only visible to **subsequent** loads on the same table — the triggering Broker Load itself runs against the original layout and uses BE runtime auto-create for any partitions it touches. INSERT-from-FILES (where the hook fires before `StatementPlanner.plan()`) is unaffected and benefits in the same load. * **Pre-created partition leak on subsequent INSERT failure.** When pre-create succeeds and the triggering INSERT later fails for unrelated reasons (FILES schema mismatch, BE crash, load timeout, etc.), the empty pre-created partitions remain in the catalog. This matches the semantics of `ALTER TABLE ADD PARTITION`, which also leaves a partition behind on subsequent failure. Operators who care can drop the empty partitions manually with `ALTER TABLE ... DROP PARTITION`; in practice empty partitions are cheap and the next retry of the load will reuse them. ###### Production deployment guidance[​](#production-deployment-guidance "Direct link to Production deployment guidance") Set `enable_execute_script_on_frontend = false` in production. Sample-Based Tablet Pre-Split exposes no SQL surface that depends on FE-side script execution; the production code paths sample through the connector + planner directly. Leaving `enable_execute_script_on_frontend = true` widens the FE attack surface without enabling any pre-split functionality, so the safe default for production clusters is to keep it off. --- ### FE Configuration - Authentication, Query, and Loading FE parameters are classified into dynamic parameters and static parameters. * Dynamic parameters can be configured and adjusted by running SQL commands, which is very convenient. But the configurations become invalid if you restart your FE. Therefore, we recommend that you also modify the configuration items in the **fe.conf** file to prevent the loss of modifications. * Static parameters can only be configured and adjusted in the FE configuration file **fe.conf**. **After you modify this file, you must restart your FE for the changes to take effect.** Whether a parameter is a dynamic parameter is indicated by the `IsMutable` column in the output of [ADMIN SHOW CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SHOW_CONFIG.md). `TRUE` indicates a dynamic parameter. Note that both dynamic and static FE parameters can be configured in the **fe.conf** file. #### View FE configuration items[​](#view-fe-configuration-items "Direct link to View FE configuration items") After your FE is started, you can run the ADMIN SHOW FRONTEND CONFIG command on your MySQL client to check the parameter configurations. If you want to query the configuration of a specific parameter, run the following command: ```sql ADMIN SHOW FRONTEND CONFIG [LIKE "pattern"]; ``` For detailed description of the returned fields, see [`ADMIN SHOW CONFIG`](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SHOW_CONFIG.md). note You must have administrator privileges to run cluster administration-related commands. #### Configure FE parameters[​](#configure-fe-parameters "Direct link to Configure FE parameters") ##### Configure FE dynamic parameters[​](#configure-fe-dynamic-parameters "Direct link to Configure FE dynamic parameters") You can configure or modify the settings of FE dynamic parameters using [`ADMIN SET FRONTEND CONFIG`](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md). ```sql ADMIN SET FRONTEND CONFIG ("key" = "value"); ``` note The configuration changes made with `ADMIN SET FRONTEND` will be restored to the default values in the `fe.conf` file after the FE restarts. Therefore, we recommend that you also modify the configuration items in `fe.conf` if you want the changes to be permanent. ##### Configure FE static parameters[​](#configure-fe-static-parameters "Direct link to Configure FE static parameters") note Static parameters of an FE are set by changing them in the configuration file **fe.conf** and restarting the FE to allow the changes to take effect. *** This topic introduces the following types of FE configurations: * [User, role, and privilege](#user-role-and-privilege) * [Query](#query-engine) * [Loading and unloading](#loading-and-unloading) #### User, role, and privilege[​](#user-role-and-privilege "Direct link to User, role, and privilege") ##### `enable_task_info_mask_credential`[​](#enable_task_info_mask_credential "Direct link to enable_task_info_mask_credential") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: When true, StarRocks redacts credentials from task SQL definitions before returning them in `information_schema.tasks` and `information_schema.task_runs` by applying SqlCredentialRedactor.redact to the DEFINITION column. In `information_schema.task_runs` the same redaction is applied whether the definition comes from the task run status or, when empty, from the task definition lookup. When false, raw task definitions are returned (may expose credentials). Masking is CPU/string-processing work and can be time-consuming when the number of tasks or `task_runs` is large; disable only if you need unredacted definitions and accept the security risk. * Introduced in: v3.5.6 ##### `privilege_max_role_depth`[​](#privilege_max_role_depth "Direct link to privilege_max_role_depth") * Default: 16 * Type: Int * Unit: * Is mutable: Yes * Description: The maximum role depth (level of inheritance) of a role. * Introduced in: v3.0.0 ##### `privilege_max_total_roles_per_user`[​](#privilege_max_total_roles_per_user "Direct link to privilege_max_total_roles_per_user") * Default: 64 * Type: Int * Unit: * Is mutable: Yes * Description: The maximum number of roles a user can have. * Introduced in: v3.0.0 #### Query engine[​](#query-engine "Direct link to Query engine") ##### `brpc_send_plan_fragment_timeout_ms`[​](#brpc_send_plan_fragment_timeout_ms "Direct link to brpc_send_plan_fragment_timeout_ms") * Default: 60000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: Timeout in milliseconds applied to the BRPC TalkTimeoutController before sending a plan fragment. `BackendServiceClient.sendPlanFragmentAsync` sets this value prior to calling the backend `execPlanFragmentAsync`. It governs how long BRPC will wait when borrowing an idle connection from the connection pool and while performing the send; if exceeded, the RPC will fail and may trigger the method's retry logic. Set this lower to fail fast under contention, or raise it to tolerate transient pool exhaustion or slow networks. Be cautious: very large values can delay failure detection and block request threads. * Introduced in: v3.3.11, v3.4.1, v3.5.0 ##### `connector_row_size_estimate_bytes`[​](#connector_row_size_estimate_bytes "Direct link to connector_row_size_estimate_bytes") * Default: 256 * Type: Long * Unit: Bytes * Is mutable: Yes * Description: The estimated average row size in bytes used by the optimizer to estimate row counts for external file tables (FILES() and ENGINE=file tables) when the storage format is unknown or the column schema is not available. The row count is estimated as `total_file_bytes / connector_row_size_estimate_bytes`. A smaller value produces a higher row count estimate and may affect join ordering decisions. * Introduced in: v3.4 ##### `connector_table_analyze_scan_bytes_cap`[​](#connector_table_analyze_scan_bytes_cap "Direct link to connector_table_analyze_scan_bytes_cap") * Default: 2147483648 (2 GB) * Type: Long * Unit: Bytes * Is mutable: Yes * Description: The primary per-scan byte budget for external-table statistics collection (Iceberg). Each per-(partition, column) statistics scan accumulates the byte size of the splits it opens and stops early once this budget is reached (a soft cap: the last split may overshoot), so an oversized single partition or unpartitioned table is collected as a bounded, degraded sample instead of failing or timing out. A value of `0` or less means this dimension is unlimited. Calibrate against the number of columns collected, because a partition runs one independent scan per column. Set this and `connector_table_analyze_scan_files_cap` and `connector_table_analyze_scan_rows_cap` all to `0` or less to disable bounded-cost collection entirely and fall back to full scans. Can be overridden per statement with `ANALYZE TABLE ... PROPERTIES("scan_bytes_cap" = "...")`. * Introduced in: v4.1 ##### `connector_table_analyze_scan_files_cap`[​](#connector_table_analyze_scan_files_cap "Direct link to connector_table_analyze_scan_files_cap") * Default: 1000 * Type: Long * Unit: - * Is mutable: Yes * Description: The secondary per-scan file-count budget for external-table statistics collection (Iceberg). A statistics scan stops early once it has opened this many files, which caps the cost of a partition made up of a very large number of small files. A value of `0` or less means this dimension is unlimited. Can be overridden per statement with `ANALYZE TABLE ... PROPERTIES("scan_files_cap" = "...")`. * Introduced in: v4.1 ##### `connector_table_analyze_scan_rows_cap`[​](#connector_table_analyze_scan_rows_cap "Direct link to connector_table_analyze_scan_rows_cap") * Default: 10000000 * Type: Long * Unit: - * Is mutable: Yes * Description: The auxiliary per-scan estimated-row budget for external-table statistics collection (Iceberg). A statistics scan stops early once the estimated number of rows it has scanned reaches this budget. Because per-split row counts can only be estimated (the record count is recorded per file, not per split), this is an auxiliary soft budget rather than the primary control. The default aligns with `connector_table_query_trigger_analyze_small_table_rows`. A value of `0` or less means this dimension is unlimited. Can be overridden per statement with `ANALYZE TABLE ... PROPERTIES("scan_rows_cap" = "...")`. * Introduced in: v4.1 ##### `connector_table_query_trigger_analyze_large_table_interval`[​](#connector_table_query_trigger_analyze_large_table_interval "Direct link to connector_table_query_trigger_analyze_large_table_interval") * Default: 12 \* 3600 * Type: Int * Unit: Second * Is mutable: Yes * Description: The interval for query-trigger ANALYZE tasks of large tables. * Introduced in: v3.4.0 ##### `connector_table_query_trigger_analyze_max_pending_task_num`[​](#connector_table_query_trigger_analyze_max_pending_task_num "Direct link to connector_table_query_trigger_analyze_max_pending_task_num") * Default: 100 * Type: Int * Unit: - * Is mutable: Yes * Description: Maximum number of query-trigger ANALYZE tasks that are in Pending state on the FE. * Introduced in: v3.4.0 ##### `connector_table_query_trigger_analyze_max_running_task_num`[​](#connector_table_query_trigger_analyze_max_running_task_num "Direct link to connector_table_query_trigger_analyze_max_running_task_num") * Default: 2 * Type: Int * Unit: - * Is mutable: Yes * Description: Maximum number of query-trigger ANALYZE tasks that are in Running state on the FE. * Introduced in: v3.4.0 ##### `connector_table_query_trigger_analyze_small_table_interval`[​](#connector_table_query_trigger_analyze_small_table_interval "Direct link to connector_table_query_trigger_analyze_small_table_interval") * Default: 2 \* 3600 * Type: Int * Unit: Second * Is mutable: Yes * Description: The interval for query-trigger ANALYZE tasks of small tables. * Introduced in: v3.4.0 ##### `connector_table_query_trigger_analyze_small_table_rows`[​](#connector_table_query_trigger_analyze_small_table_rows "Direct link to connector_table_query_trigger_analyze_small_table_rows") * Default: 10000000 * Type: Int * Unit: - * Is mutable: Yes * Description: The threshold for determining whether a table is a small table for query-trigger ANALYZE tasks. * Introduced in: v3.4.0 ##### `connector_table_query_trigger_task_schedule_interval`[​](#connector_table_query_trigger_task_schedule_interval "Direct link to connector_table_query_trigger_task_schedule_interval") * Default: 30 * Type: Int * Unit: Second * Is mutable: Yes * Description: The interval at which the Scheduler thread schedules the query-trigger background tasks. This item is to replace `connector_table_query_trigger_analyze_schedule_interval` introduced in v3.4.0. Here, the background tasks refer `ANALYZE` tasks in v3.4，and the collection task of low-cardinality columns' dictionary in versions later than v3.4. * Introduced in: v3.4.2 ##### `create_table_max_serial_replicas`[​](#create_table_max_serial_replicas "Direct link to create_table_max_serial_replicas") * Default: 128 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of replicas to create serially. If actual replica count exceeds this value, replicas will be created concurrently. Try to reduce this value if table creation is taking a long time to complete. * Introduced in: - ##### `default_mv_partition_refresh_number`[​](#default_mv_partition_refresh_number "Direct link to default_mv_partition_refresh_number") * Default: 1 * Type: Int * Unit: - * Is mutable: Yes * Description: When a materialized view refresh involves multiple partitions, this parameter controls how many partitions are refreshed in a single batch by default. Starting from version 3.3.0, the system defaults to refreshing one partition at a time to avoid potential out-of-memory (OOM) issues. In earlier versions, all partitions were refreshed at once by default, which could lead to memory exhaustion and task failure. However, note that when a materialized view refresh involves a large number of partitions, refreshing only one partition at a time may lead to excessive scheduling overhead, longer overall refresh time, and a large number of refresh records. In such cases, it is recommended to adjust this parameter appropriately to improve refresh efficiency and reduce scheduling costs. * Introduced in: v3.3.0 ##### `default_mv_refresh_immediate`[​](#default_mv_refresh_immediate "Direct link to default_mv_refresh_immediate") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to refresh an asynchronous materialized view immediately after creation. When this item is set to `true`, newly created materialized view will be refreshed immediately. * Introduced in: v3.2.3 ##### `dynamic_partition_check_interval_seconds`[​](#dynamic_partition_check_interval_seconds "Direct link to dynamic_partition_check_interval_seconds") * Default: 600 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The interval at which new data is checked. If new data is detected, StarRocks automatically creates partitions for the data. * Introduced in: - ##### `dynamic_partition_enable`[​](#dynamic_partition_enable "Direct link to dynamic_partition_enable") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable the dynamic partitioning feature. When this feature is enabled, StarRocks dynamically creates partitions for new data and automatically deletes expired partitions to ensure the freshness of data. * Introduced in: - ##### `enable_active_materialized_view_schema_strict_check`[​](#enable_active_materialized_view_schema_strict_check "Direct link to enable_active_materialized_view_schema_strict_check") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to strictly check the length consistency of data types when activating an inactive materialized view. When this item is set to `false`, the activation of the materialized view is not affected if the length of the data types has changed in the base table. * Introduced in: v3.3.4 ##### `mv_fast_schema_change_mode`[​](#mv_fast_schema_change_mode "Direct link to mv_fast_schema_change_mode") * Default: strict * Type: String * Unit: - * Is mutable: Yes * Description: Controls the behavior of Materialized View (MV) Fast Schema Evolution (FSE). Valid values are: `strict` (default) - only allow FSE when `isSupportFastSchemaEvolutionInDanger` is true and clear affected partition entries from the version map; `force` - allow FSE even when `isSupportFastSchemaEvolutionInDanger` is false and clear affected partition entries to trigger recomputation on refresh; `force_no_clear` - allow FSE even when `isSupportFastSchemaEvolutionInDanger` is false but do not clear partition entries. * Introduced in: v4.1.0 ##### `enable_auto_collect_array_ndv`[​](#enable_auto_collect_array_ndv "Direct link to enable_auto_collect_array_ndv") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable automatic collection for the NDV information of the ARRAY type. * Introduced in: v4.1 ##### `enable_backup_materialized_view`[​](#enable_backup_materialized_view "Direct link to enable_backup_materialized_view") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable the BACKUP and RESTORE of asynchronous materialized views when backing up or restoring a specific database. If this item is set to `false`, StarRocks will skip backing up asynchronous materialized views. * Introduced in: v3.2.0 ##### `enable_collect_full_statistic`[​](#enable_collect_full_statistic "Direct link to enable_collect_full_statistic") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable automatic full statistics collection. This feature is enabled by default. * Introduced in: - ##### `enable_colocate_mv_index`[​](#enable_colocate_mv_index "Direct link to enable_colocate_mv_index") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to support colocating the synchronous materialized view index with the base table when creating a synchronous materialized view. If this item is set to `true`, tablet sink will speed up the write performance of synchronous materialized views. * Introduced in: v3.2.0 ##### `enable_decimal_v3`[​](#enable_decimal_v3 "Direct link to enable_decimal_v3") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to support the DECIMAL V3 data type. * Introduced in: - ##### `enable_experimental_mv`[​](#enable_experimental_mv "Direct link to enable_experimental_mv") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable the asynchronous materialized view feature. TRUE indicates this feature is enabled. From v2.5.2 onwards, this feature is enabled by default. For versions earlier than v2.5.2, this feature is disabled by default. * Introduced in: v2.4 ##### `enable_local_replica_selection`[​](#enable_local_replica_selection "Direct link to enable_local_replica_selection") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to select local replicas for queries. Local replicas reduce the network transmission cost. If this parameter is set to TRUE, the CBO preferentially selects tablet replicas on BEs that have the same IP address as the current FE. If this parameter is set to `FALSE`, both local replicas and non-local replicas can be selected. * Introduced in: - ##### `enable_manual_collect_array_ndv`[​](#enable_manual_collect_array_ndv "Direct link to enable_manual_collect_array_ndv") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable manual collection for the NDV information of the ARRAY type. * Introduced in: v4.1 ##### `enable_materialized_view`[​](#enable_materialized_view "Direct link to enable_materialized_view") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable the creation of materialized views. * Introduced in: - ##### `enable_materialized_view_external_table_precise_refresh`[​](#enable_materialized_view_external_table_precise_refresh "Direct link to enable_materialized_view_external_table_precise_refresh") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Set this item to `true` to enable an internal optimization for materialized view refresh when a base table is an external (non-cloud-native) table. When enabled, the materialized view refresh processor computes candidate partitions and refreshes only the affected base-table partitions instead of all partitions, reducing I/O and refresh cost. Set it to `false` to force full-partition refresh of external tables. * Introduced in: v3.2.9 ##### `enable_materialized_view_metrics_collect`[​](#enable_materialized_view_metrics_collect "Direct link to enable_materialized_view_metrics_collect") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to collect monitoring metrics for asynchronous materialized views by default. * Introduced in: v3.1.11, v3.2.5 ##### `enable_materialized_view_spill`[​](#enable_materialized_view_spill "Direct link to enable_materialized_view_spill") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable Intermediate Result Spilling for materialized view refresh tasks. * Introduced in: v3.1.1 ##### `enable_materialized_view_text_based_rewrite`[​](#enable_materialized_view_text_based_rewrite "Direct link to enable_materialized_view_text_based_rewrite") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable text-based query rewrite by default. If this item is set to `true`, the system builds the abstract syntax tree while creating an asynchronous materialized view. * Introduced in: v3.2.5 ##### `enable_mv_automatic_active_check`[​](#enable_mv_automatic_active_check "Direct link to enable_mv_automatic_active_check") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable the system to automatically check and re-activate the asynchronous materialized views that are set inactive because their base tables (views) had undergone Schema Change or had been dropped and re-created. Please note that this feature will not re-activate the materialized views that are manually set inactive by users. * Introduced in: v3.1.6 ##### `enable_mv_automatic_repairing_for_broken_base_tables`[​](#enable_mv_automatic_repairing_for_broken_base_tables "Direct link to enable_mv_automatic_repairing_for_broken_base_tables") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: When this item is set to `true`, StarRocks will attempt to automatically repair materialized view base-table metadata when a base external table is dropped and recreated or its table identifier changes. The repair flow can update the materialized view's base table information, collect partition-level repair information for external table partitions, and drive partition refresh decisions for async auto-refresh materialized views while honoring `autoRefreshPartitionsLimit`. Currently the automated repair supports Hive external tables; unsupported table types will cause the materialized view to be set inactive and a repair exception. Partition information collection is non-blocking and failures are logged. * Introduced in: v3.3.19, v3.4.8, v3.5.6 ##### `enable_predicate_columns_collection`[​](#enable_predicate_columns_collection "Direct link to enable_predicate_columns_collection") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable predicate columns collection. If disabled, predicate columns will not be recorded during query optimization. * Introduced in: - ##### `push_down_non_grouped_aggregate_below_union`[​](#push_down_non_grouped_aggregate_below_union "Direct link to push_down_non_grouped_aggregate_below_union") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to push down non-grouped aggregations below Union in the physical plan. * Introduced in: - ##### `enable_query_queue_v2`[​](#enable_query_queue_v2 "Direct link to enable_query_queue_v2") * Default: true * Type: boolean * Unit: - * Is mutable: No * Description: When true, switches the FE slot-based query scheduler to Query Queue V2. The flag is read by the slot manager and trackers (for example, `BaseSlotManager.isEnableQueryQueueV2` and `SlotTracker#createSlotSelectionStrategy`) to choose `SlotSelectionStrategyV2` instead of the legacy strategy. `query_queue_v2_xxx` configuration options and `QueryQueueOptions` take effect only when this flag is enabled. From v4.1 onwards, the default value is changed from `false` to `true`. * Introduced in: v3.3.4, v3.4.0, v3.5.0 ##### `enable_sql_blacklist`[​](#enable_sql_blacklist "Direct link to enable_sql_blacklist") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable blacklist check for SQL queries. When this feature is enabled, queries in the blacklist cannot be executed. * Introduced in: - ##### `enable_statistic_collect`[​](#enable_statistic_collect "Direct link to enable_statistic_collect") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to collect statistics for the CBO. This feature is enabled by default. * Introduced in: - ##### `enable_statistic_collect_on_first_load`[​](#enable_statistic_collect_on_first_load "Direct link to enable_statistic_collect_on_first_load") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Controls automatic statistics collection and maintenance triggered by data loading operations. This includes: * Statistics collection when data is first loaded into a partition (partition version equals 2). * Statistics collection when data is loaded into empty partitions of multi-partition tables. * Statistics copying and updating for INSERT OVERWRITE operations. **Decision Policy for Statistics Collection Type:** * For INSERT OVERWRITE: `deltaRatio = |targetRows - sourceRows| / (sourceRows + 1)` * If `deltaRatio < statistic_sample_collect_ratio_threshold_of_first_load` (Default: 0.1), statistics collection will not be performed. Only the existing statistics will be copied. * Else, if `targetRows > statistic_sample_collect_rows` (Default: 200000), SAMPLE statistics collection is used. * Else, FULL statistics collection is used. * For First Load: `deltaRatio = loadRows / (totalRows + 1)` * If `deltaRatio < statistic_sample_collect_ratio_threshold_of_first_load` (Default: 0.1), statistics collection will not be performed. * Else, if `loadRows > statistic_sample_collect_rows` (Default: 200000), SAMPLE statistics collection is used. * Else, FULL statistics collection is used. **Synchronization Behavior:** * For DML statements (INSERT INTO/INSERT OVERWRITE): Synchronous mode with table lock. The load operation waits for statistics collection to complete (up to `semi_sync_collect_statistic_await_seconds`). * For Stream Load and Broker Load: Asynchronous mode without lock. Statistics collection runs in background without blocking the load operation. note Disabling this configuration will prevent all loading-triggered statistics operations, including statistics maintenance for INSERT OVERWRITE, which may result in tables lacking statistics. If new tables are frequently created and data is frequently loaded, enabling this feature will increase memory and CPU overhead. * Introduced in: v3.1 ##### `enable_statistic_collect_on_update`[​](#enable_statistic_collect_on_update "Direct link to enable_statistic_collect_on_update") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Controls whether UPDATE statements can trigger automatic statistics collection. When enabled, UPDATE operations that modify table data may schedule statistics collection through the same ingestion-based statistics framework controlled by `enable_statistic_collect_on_first_load`. Disabling this configuration skips statistics collection for UPDATE statements while keeping load-triggered statistics collection behavior unchanged. * Introduced in: v3.5.11, v4.0.4 ##### `enable_udf`[​](#enable_udf "Direct link to enable_udf") * Default: false * Type: Boolean * Unit: - * Is mutable: No * Description: Whether to enable UDF. * Introduced in: - ##### `expr_children_limit`[​](#expr_children_limit "Direct link to expr_children_limit") * Default: 10000 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of child expressions allowed in an expression. * Introduced in: - ##### `histogram_buckets_size`[​](#histogram_buckets_size "Direct link to histogram_buckets_size") * Default: 64 * Type: Long * Unit: - * Is mutable: Yes * Description: The default bucket number for a histogram. * Introduced in: - ##### `histogram_max_sample_row_count`[​](#histogram_max_sample_row_count "Direct link to histogram_max_sample_row_count") * Default: 10000000 * Type: Long * Unit: - * Is mutable: Yes * Description: The maximum number of rows to collect for a histogram. * Introduced in: - ##### `histogram_mcv_size`[​](#histogram_mcv_size "Direct link to histogram_mcv_size") * Default: 100 * Type: Long * Unit: - * Is mutable: Yes * Description: The number of most common values (MCV) for a histogram. * Introduced in: - ##### `histogram_sample_ratio`[​](#histogram_sample_ratio "Direct link to histogram_sample_ratio") * Default: 0.1 * Type: Double * Unit: - * Is mutable: Yes * Description: The sampling ratio for a histogram. * Introduced in: - ##### `http_slow_request_threshold_ms`[​](#http_slow_request_threshold_ms "Direct link to http_slow_request_threshold_ms") * Default: 5000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: If the response time for an HTTP request exceeds the value specified by this parameter, a log is generated to track this request. * Introduced in: v2.5.15, v3.1.5 ##### `lock_checker_enable_deadlock_check`[​](#lock_checker_enable_deadlock_check "Direct link to lock_checker_enable_deadlock_check") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: When enabled, the LockChecker thread performs JVM-level deadlock detection using ThreadMXBean.findDeadlockedThreads() and logs the offending threads' stack traces. The check runs inside the LockChecker daemon (whose frequency is controlled by `lock_checker_interval_second`) and writes detailed stack information to the log, which may be CPU- and I/O-intensive. Enable this option only for troubleshooting live or reproducible deadlock issues; leaving it enabled in normal operation can increase overhead and log volume. * Introduced in: v3.2.0 ##### `low_cardinality_threshold`[​](#low_cardinality_threshold "Direct link to low_cardinality_threshold") * Default: 255 * Type: Int * Unit: - * Is mutable: No * Description: Threshold of low cardinality dictionary. * Introduced in: v3.5.0 ##### `materialized_view_min_refresh_interval`[​](#materialized_view_min_refresh_interval "Direct link to materialized_view_min_refresh_interval") * Default: 60 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The minimum allowed refresh interval (in seconds) for ASYNC materialized view schedules. When a materialized view is created with a time-based interval, the interval is converted to seconds and must not be less tham this value; otherwise the CREATE/ALTER operation fails with a DDL error. If this value is greater than 0, the check is enforced; set it to 0 or a negative value to disable the limit, which prevents excessive TaskManager scheduling and high FE memory/CPU usage from overly frequent refreshes. This item does not apply to `EVENT_TRIGGERED` refreshes. * Introduced in: v3.3.0, v3.4.0, v3.5.0 ##### `materialized_view_refresh_ascending`[​](#materialized_view_refresh_ascending "Direct link to materialized_view_refresh_ascending") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: When this item is set to `true`, materialized view partition refresh will iterate partitions in ascending partition-key order (oldest to newest). When it is set to `false` (default), the system iterates in descending order (newest to oldest). StarRocks uses this item in both list- and range-partitioned materialized view refresh logic to choose which partitions to process when partition refresh limits apply and to compute the next start/end partition boundaries for subsequent TaskRun executions. Changing this item alters which partitions are refreshed first and how the next partition range is derived; for range-partitioned materialized views, the scheduler validates new start/end and will raise an error if a change would create a repeated boundary (dead-loop), so set this item with care. * Introduced in: v3.3.1, v3.4.0, v3.5.0 ##### `max_allowed_in_element_num_of_delete`[​](#max_allowed_in_element_num_of_delete "Direct link to max_allowed_in_element_num_of_delete") * Default: 10000 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of elements allowed for the IN predicate in a DELETE statement. * Introduced in: - ##### `max_create_table_timeout_second`[​](#max_create_table_timeout_second "Direct link to max_create_table_timeout_second") * Default: 600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The maximum timeout duration for creating a table. * Introduced in: - ##### `max_distribution_pruner_recursion_depth`[​](#max_distribution_pruner_recursion_depth "Direct link to max_distribution_pruner_recursion_depth") * Default: 100 * Type: Int * Unit: - * Is mutable: Yes * Description:: The maximum recursion depth allowed by the partition pruner. Increasing the recursion depth can prune more elements but also increases CPU consumption. * Introduced in: - ##### `max_partitions_in_one_batch`[​](#max_partitions_in_one_batch "Direct link to max_partitions_in_one_batch") * Default: 4096 * Type: Long * Unit: - * Is mutable: Yes * Description: The maximum number of partitions that can be created when you bulk create partitions. * Introduced in: - ##### `max_planner_scalar_rewrite_num`[​](#max_planner_scalar_rewrite_num "Direct link to max_planner_scalar_rewrite_num") * Default: 100000 * Type: Long * Unit: - * Is mutable: Yes * Description: The maximum number of times that the optimizer can rewrite a scalar operator. * Introduced in: - ##### `max_query_queue_history_slots_number`[​](#max_query_queue_history_slots_number "Direct link to max_query_queue_history_slots_number") * Default: 0 * Type: Int * Unit: Slots * Is mutable: Yes * Description: Controls how many recently released (history) allocated slots are retained per query queue for monitoring and observability. When `max_query_queue_history_slots_number` is set to a value `> 0`, BaseSlotTracker keeps up to that many most-recently released LogicalSlot entries in an in-memory queue, evicting the oldest when the limit is exceeded. Enabling this causes getSlots() to include these history entries (newest first), allows BaseSlotTracker to attempt registering slots with the ConnectContext for richer ExtraMessage data, and lets LogicalSlot.ConnectContextListener attach query finish metadata to history slots. When `max_query_queue_history_slots_number` `<= 0` the history mechanism is disabled (no extra memory used). Use a reasonable value to balance observability and memory overhead. * Introduced in: v3.5.0 ##### `max_query_retry_time`[​](#max_query_retry_time "Direct link to max_query_retry_time") * Default: 2 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of query retries on an FE. * Introduced in: - ##### `max_running_rollup_job_num_per_table`[​](#max_running_rollup_job_num_per_table "Direct link to max_running_rollup_job_num_per_table") * Default: 1 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of rollup jobs can run in parallel for a table. * Introduced in: - ##### `max_scalar_operator_flat_children`[​](#max_scalar_operator_flat_children "Direct link to max_scalar_operator_flat_children") * Default: 10000 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of flat children for ScalarOperator. You can set this limit to prevent the optimizer from using too much memory. * Introduced in: - ##### `max_scalar_operator_optimize_depth`[​](#max_scalar_operator_optimize_depth "Direct link to max_scalar_operator_optimize_depth") * Default: 256 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum depth that ScalarOperator optimization can be applied. * Introduced in: - ##### `mv_active_checker_interval_seconds`[​](#mv_active_checker_interval_seconds "Direct link to mv_active_checker_interval_seconds") * Default: 60 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: When the background `active_checker` thread is enabled, the system will periodically detect and automatically reactivate materialized views that became Inactive due to schema changes or rebuilds of their base tables (or views). This parameter controls the scheduling interval of the checker thread, in seconds. The default value is system-defined. * Introduced in: v3.1.6 ##### `mv_rewrite_consider_data_layout_mode`[​](#mv_rewrite_consider_data_layout_mode "Direct link to mv_rewrite_consider_data_layout_mode") * Default: `enable` * Type: String * Unit: - * Is mutable: Yes * Description: Controls whether materialized view rewrite should take the base table data layout into account when selecting the best materialized view. Valid values: * `disable`: Never use data-layout criteria when choosing between candidate materialized views. * `enable`: Use data-layout criteria only when the query is recognized as layout-sensitive. * `force`: Always apply data-layout criteria when selecting the best materialized view. Changing this item affects `BestMvSelector` behavior and can improve or broaden rewrite applicability depending on whether physical layout matters for plan correctness or performance. * Introduced in: - ##### `publish_version_interval_ms`[​](#publish_version_interval_ms "Direct link to publish_version_interval_ms") * Default: 10 * Type: Int * Unit: Milliseconds * Is mutable: No * Description: The time interval at which release validation tasks are issued. * Introduced in: - ##### `query_queue_slots_estimator_strategy`[​](#query_queue_slots_estimator_strategy "Direct link to query_queue_slots_estimator_strategy") * Default: PBE * Type: String * Unit: - * Is mutable: Yes * Description: Selects the slot estimation strategy used for queue-based queries when `enable_query_queue_v2` is true. Valid values: `PBE` (parallelism-based, the default), `MBE` (memory-cost-based), and `CBE` (CPU-cost-based). PBE estimates a query's slots from scan parallelism, capped by the worker count: for OLAP tables it uses the number of scan ranges left after pruning, so only very small queries fall below the worker count; a connector/external scan is treated as a full-parallelism scan (the worker count) rather than a single-slot query. MBE estimates slots from the query's memory cost divided by `query_queue_v2_mem_bytes_per_slot`. CBE estimates slots from the plan CPU cost divided by `query_queue_v2_cpu_costs_per_slot`. MBE and CBE per-query slots are additionally capped by `number_of_workers * max(1, pipeline_dop / 2)`. The legacy values `MAX` and `MIN` are still accepted for forward compatibility and are treated as the default estimator; any other value is rejected by configuration validation. * Introduced in: v3.5.0 ##### `query_queue_v2_concurrency_level`[​](#query_queue_v2_concurrency_level "Direct link to query_queue_v2_concurrency_level") * Default: 4 * Type: Int * Unit: - * Is mutable: Yes * Description: Interpreted as a capacity level relative to the default level `4`. For the default (PBE) and CPU-cost-based (CBE) estimators, the system's total query slots are computed as `number_of_workers * cores_per_worker * (query_queue_v2_concurrency_level / 4)` (derived from BackendResourceStat). For the memory-cost-based estimator (MBE), the total slots are instead derived from the warehouse memory budget. If the configured value is non-positive it is treated as `4`. total\_slots is clamped to at least `number_of_workers`. Increasing this value raises totalSlots (and therefore concurrent query capacity); at the default `4` the total capacity equals `number_of_workers * cores_per_worker`. Set it proportional to the concurrency you want for the cluster. * Introduced in: v3.3.4, v3.4.0, v3.5.0 ##### `query_queue_v2_cpu_costs_per_slot`[​](#query_queue_v2_cpu_costs_per_slot "Direct link to query_queue_v2_cpu_costs_per_slot") * Default: 1000000000 * Type: Long * Unit: planner CPU cost units * Is mutable: Yes * Description: Per-slot CPU cost threshold used by the CPU-cost-based estimator (CBE) to estimate how many slots a query needs from its plan CPU cost. The scheduler computes slots as `ceil(plan_cpu_costs / query_queue_v2_cpu_costs_per_slot)` and clamps the result to the range `[1, min(totalSlots, number_of_workers * max(1, pipeline_dop / 2))]`. A non-positive value is normalized to `1`. Increasing this value reduces slots allocated per query (favoring fewer, larger-slot queries); decreasing it increases slots per query. * Introduced in: v3.3.4, v3.4.0, v3.5.0 ##### `query_queue_v2_mem_bytes_per_slot`[​](#query_queue_v2_mem_bytes_per_slot "Direct link to query_queue_v2_mem_bytes_per_slot") * Default: 0 * Type: Long * Unit: Bytes * Is mutable: Yes * Description: Per-slot memory target used by the memory-cost-based estimator (MBE). When `query_queue_slots_estimator_strategy` is `MBE`, the total slots are derived from the warehouse memory budget, and a query's slots are estimated from its total memory cost divided by this value, capped by `number_of_workers * max(1, pipeline_dop / 2)`. If it is non-positive, Query Queue V2 uses the average worker memory per core. * Introduced in: - ##### `query_queue_v2_num_rows_per_slot`[​](#query_queue_v2_num_rows_per_slot "Direct link to query_queue_v2_num_rows_per_slot") * Default: 4096 * Type: Int * Unit: Rows * Is mutable: Yes * Description: Retained for backward compatibility with existing Query Queue V2 serialized and debug output. It is no longer used by the PBE, MBE, or CBE slot estimators. * Introduced in: v3.3.4, v3.4.0, v3.5.0 ##### `query_queue_v2_schedule_strategy`[​](#query_queue_v2_schedule_strategy "Direct link to query_queue_v2_schedule_strategy") * Default: SWRR * Type: String * Unit: - * Is mutable: Yes * Description: Selects the scheduling policy used by Query Queue V2 to order pending queries. Supported values (case-insensitive) are `SWRR` (Smooth Weighted Round Robin) — the default, suitable for mixed/hybrid workloads that need fair weighted sharing — and `SJF` (Short Job First + Aging) — prioritizes short jobs while using aging to avoid starvation. The value is parsed with case-insensitive enum lookup; an unrecognized value is logged as an error and the default policy is used. This configuration only affects behavior when Query Queue V2 is enabled and interacts with V2 sizing settings such as `query_queue_v2_concurrency_level`. * Introduced in: v3.3.12, v3.4.2, v3.5.0 ##### `semi_sync_collect_statistic_await_seconds`[​](#semi_sync_collect_statistic_await_seconds "Direct link to semi_sync_collect_statistic_await_seconds") * Default: 30 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Maximum wait time for semi-synchronous statistics collection during DML operations (INSERT INTO and INSERT OVERWRITE statements). Stream Load and Broker Load use asynchronous mode and are not affected by this configuration. If statistics collection time exceeds this value, the load operation continues without waiting for collection to complete. This configuration works in conjunction with `enable_statistic_collect_on_first_load`. * Introduced in: v3.1 ##### `slow_query_analyze_threshold`[​](#slow_query_analyze_threshold "Direct link to slow_query_analyze_threshold") * Default: 5000 * Type: Int * Unit: Seconds * Is mutable: Yes * Description:: The execution time threshold for queries to trigger the analysis of Query Feedback. * Introduced in: v3.4.0 ##### `statistic_analyze_status_keep_second`[​](#statistic_analyze_status_keep_second "Direct link to statistic_analyze_status_keep_second") * Default: 3 \* 24 \* 3600 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The duration to retain the history of collection tasks. The default value is 3 days. * Introduced in: - ##### `statistic_auto_analyze_end_time`[​](#statistic_auto_analyze_end_time "Direct link to statistic_auto_analyze_end_time") * Default: 23:59:59 * Type: String * Unit: - * Is mutable: Yes * Description: The end time of automatic collection. Value range: `00:00:00` - `23:59:59`. * Introduced in: - ##### `statistic_auto_analyze_start_time`[​](#statistic_auto_analyze_start_time "Direct link to statistic_auto_analyze_start_time") * Default: 00:00:00 * Type: String * Unit: - * Is mutable: Yes * Description: The start time of automatic collection. Value range: `00:00:00` - `23:59:59`. * Introduced in: - ##### `statistic_auto_collect_ratio`[​](#statistic_auto_collect_ratio "Direct link to statistic_auto_collect_ratio") * Default: 0.8 * Type: Double * Unit: - * Is mutable: Yes * Description: The threshold for determining whether the statistics for automatic collection are healthy. If statistics health is below this threshold, automatic collection is triggered. * Introduced in: - ##### `statistic_auto_collect_small_table_rows`[​](#statistic_auto_collect_small_table_rows "Direct link to statistic_auto_collect_small_table_rows") * Default: 10000000 * Type: Long * Unit: - * Is mutable: Yes * Description: Threshold to determine whether a table in an external data source (Hive, Iceberg, Hudi) is a small table during automatic collection. If the table has rows less than this value, the table is considered a small table. * Introduced in: v3.2 ##### `statistic_cache_columns`[​](#statistic_cache_columns "Direct link to statistic_cache_columns") * Default: 100000 * Type: Long * Unit: - * Is mutable: No * Description: The number of rows that can be cached for the statistics table. * Introduced in: - ##### `statistic_cache_thread_pool_size`[​](#statistic_cache_thread_pool_size "Direct link to statistic_cache_thread_pool_size") * Default: 5 * Type: Int * Unit: - * Is mutable: No * Description: The size of the thread-pool which will be used to refresh statistic caches. * Introduced in: - ##### `statistic_collect_interval_sec`[​](#statistic_collect_interval_sec "Direct link to statistic_collect_interval_sec") * Default: 10 \* 60 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The interval for checking data updates during automatic collection. * Introduced in: - ##### `statistic_max_full_collect_data_size`[​](#statistic_max_full_collect_data_size "Direct link to statistic_max_full_collect_data_size") * Default: 100 \* 1024 \* 1024 \* 1024 * Type: Long * Unit: bytes * Is mutable: Yes * Description: The data size threshold for the automatic collection of statistics. If the total size exceeds this value, then sampled collection is performed instead of full. * Introduced in: - ##### `statistic_sample_collect_rows`[​](#statistic_sample_collect_rows "Direct link to statistic_sample_collect_rows") * Default: 200000 * Type: Long * Unit: - * Is mutable: Yes * Description: The row count threshold for deciding between SAMPLE and FULL statistics collection during loading-triggered statistics operations. If the number of loaded or changed rows exceeds this threshold (default 200,000), SAMPLE statistics collection is used; otherwise, FULL statistics collection is used. This setting works in conjunction with `enable_statistic_collect_on_first_load` and `statistic_sample_collect_ratio_threshold_of_first_load`. * Introduced in: - ##### `statistic_update_interval_sec`[​](#statistic_update_interval_sec "Direct link to statistic_update_interval_sec") * Default: 24 \* 60 \* 60 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The interval at which the cache of statistical information is updated. * Introduced in: - ##### `enable_external_stats_lazy_refresh_on_replay`[​](#enable_external_stats_lazy_refresh_on_replay "Direct link to enable_external_stats_lazy_refresh_on_replay") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Controls how followers (and restart recovery) refresh the connector (external table) statistics cache when replaying statistics journals. When set to `true`, the cache is invalidated by the table UUID persisted in the journal and reloaded lazily on the next query, which avoids resolving external table metadata (`MetadataMgr.getTable`) during replay — such resolution may block the journal replayer on the Hive Metastore or object storage. When set to `false` (default), the legacy eager refresh is used, preserving existing behavior. Statistics journals written before this UUID was persisted always fall back to eager refresh regardless of this setting. ##### `statistics_large_string_column_merge_threshold`[​](#statistics_large_string_column_merge_threshold "Direct link to statistics_large_string_column_merge_threshold") * Default: 0 * Type: Long * Unit: Bytes * Is mutable: Yes * Description: Disabled by default (`0`). When set to a positive value, a dedicated SQL is generated during statistics collection to collect the statistics of string columns (`VARCHAR` / `CHAR`) whose declared length exceeds this threshold, instead of merging them with other columns. Both sampled and full statistics collection follow this strategy. The purpose is to bound the Exchange-stage memory peak of a single statistics SQL and prevent long string columns from further amplifying the aggregate operator state when merged with other columns. Keep it at `0` to collect all columns through the original merged-batch path. Note that `STRING` is represented internally as a maximum-length `VARCHAR`, so enabling this option with a positive threshold may also isolate `STRING` columns. * Introduced in: - ##### `task_check_interval_second`[​](#task_check_interval_second "Direct link to task_check_interval_second") * Default: 60 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Interval between executions of task background jobs. GlobalStateMgr uses this value to schedule the TaskCleaner FrontendDaemon which invokes `doTaskBackgroundJob()`; the value is multiplied by 1000 to set the daemon interval in milliseconds. Decreasing the value makes background maintenance (task cleanup, checks) run more frequently and react faster but increases CPU/IO overhead; increasing it reduces overhead but delays cleanup and detection of stale tasks. Tune this value to balance maintenance responsiveness and resource usage. * Introduced in: v3.2.0 ##### `task_min_schedule_interval_s`[​](#task_min_schedule_interval_s "Direct link to task_min_schedule_interval_s") * Default: 10 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Minimum allowed schedule interval (in seconds) for task schedules checked by the SQL layer. When a task is submitted, TaskAnalyzer converts the schedule period to seconds and rejects the submission with `ERR_INVALID_PARAMETER` if the period is smaller than `task_min_schedule_interval_s`. This prevents creating tasks that run too frequently and protects the scheduler from high-frequency tasks. If a schedule has no explicit start time, TaskAnalyzer sets the start time to the current epoch seconds. * Introduced in: v3.3.0, v3.4.0, v3.5.0 ##### `task_runs_timeout_second`[​](#task_runs_timeout_second "Direct link to task_runs_timeout_second") * Default: 4 \* 3600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Default execution timeout (in seconds) for a TaskRun. This item is used by TaskRun execution as the baseline timeout. If the task run's properties include session variables `query_timeout` or `insert_timeout` with a positive integer value, the runtime uses the larger value between that session timeout and `task_runs_timeout_second`. The effective timeout is then bounded to not exceed the configured `task_runs_ttl_second` and `task_ttl_second`. Set this item to limit how long a task run may execute. Very large values may be clipped by the task/task-run TTL settings. * Introduced in: - #### Loading and unloading[​](#loading-and-unloading "Direct link to Loading and unloading") ##### `broker_load_default_timeout_second`[​](#broker_load_default_timeout_second "Direct link to broker_load_default_timeout_second") * Default: 14400 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The timeout duration for a Broker Load job. * Introduced in: - ##### `desired_max_waiting_jobs`[​](#desired_max_waiting_jobs "Direct link to desired_max_waiting_jobs") * Default: 1024 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of pending jobs in an FE. The number refers to all jobs, such as table creation, loading, and schema change jobs. If the number of pending jobs in an FE reaches this value, the FE will reject new load requests. This parameter takes effect only for asynchronous loading. From v2.5 onwards, the default value is changed from 100 to 1024. * Introduced in: - ##### `disable_load_job`[​](#disable_load_job "Direct link to disable_load_job") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to disable loading when the cluster encounters an error. This prevents any loss caused by cluster errors. The default value is `FALSE`, indicating that loading is not disabled. `TRUE` indicates loading is disabled and the cluster is in read-only state. * Introduced in: - ##### `empty_load_as_error`[​](#empty_load_as_error "Direct link to empty_load_as_error") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to return an error message "all partitions have no load data" if no data is loaded. Valid values: * `true`: If no data is loaded, the system displays a failure message and returns an error "all partitions have no load data". * `false`: If no data is loaded, the system displays a success message and returns OK, instead of an error. * Introduced in: - ##### `enable_file_bundling`[​](#enable_file_bundling "Direct link to enable_file_bundling") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to enable the File Bundling optimization for the cloud-native table. When this feature is enabled (set to `true`), the system automatically bundles the data files generated by loading, Compaction, or Publish operations, thereby reducing the API cost caused by high-frequency access to the external storage system. You can also control this behavior on the table level using the CREATE TABLE property `file_bundling`. For detailed instructions, see CREATE TABLE. * Introduced in: v4.1 ##### `enable_routine_load_lag_metrics`[​](#enable_routine_load_lag_metrics "Direct link to enable_routine_load_lag_metrics") * Default: false * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to collect Routine Load Kafka partition offset lag metrics. Please note that set this item to `true` will call the Kafka API to get the partition's latest offset. * Introduced in: - ##### `enable_sync_publish`[​](#enable_sync_publish "Direct link to enable_sync_publish") * Default: true * Type: Boolean * Unit: - * Is mutable: Yes * Description: Whether to synchronously execute the apply task at the publish phase of a load transaction. This parameter is applicable only to Primary Key tables. Valid values: * `TRUE` (default): The apply task is synchronously executed at the publish phase of a load transaction. It means that the load transaction is reported as successful only after the apply task is completed, and the loaded data can truly be queried. When a task loads a large volume of data at a time or loads data frequently, setting this parameter to `true` can improve query performance and stability, but may increase load latency. * `FALSE`: The apply task is asynchronously executed at the publish phase of a load transaction. It means that the load transaction is reported as successful after the apply task is submitted, but the loaded data cannot be immediately queried. In this case, concurrent queries need to wait for the apply task to complete or time out before they can continue. When a task loads a large volume of data at a time or loads data frequently, setting this parameter to `false` may affect query performance and stability. * Introduced in: v3.2.0 ##### `export_checker_interval_second`[​](#export_checker_interval_second "Direct link to export_checker_interval_second") * Default: 5 * Type: Int * Unit: Seconds * Is mutable: No * Description: The time interval at which load jobs are scheduled. * Introduced in: - ##### `export_max_bytes_per_be_per_task`[​](#export_max_bytes_per_be_per_task "Direct link to export_max_bytes_per_be_per_task") * Default: 268435456 * Type: Long * Unit: Bytes * Is mutable: Yes * Description: The maximum amount of data that can be exported from a single BE by a single data unload task. * Introduced in: - ##### `export_running_job_num_limit`[​](#export_running_job_num_limit "Direct link to export_running_job_num_limit") * Default: 5 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of data exporting tasks that can run in parallel. * Introduced in: - ##### `export_task_default_timeout_second`[​](#export_task_default_timeout_second "Direct link to export_task_default_timeout_second") * Default: 2 \* 3600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The timeout duration for a data exporting task. * Introduced in: - ##### `export_task_pool_size`[​](#export_task_pool_size "Direct link to export_task_pool_size") * Default: 5 * Type: Int * Unit: - * Is mutable: No * Description: The size of the unload task thread pool. * Introduced in: - ##### `external_table_commit_timeout_ms`[​](#external_table_commit_timeout_ms "Direct link to external_table_commit_timeout_ms") * Default: 10000 * Type: Int * Unit: Milliseconds * Is mutable: Yes * Description: The timeout duration for committing (publishing) a write transaction to a StarRocks external table. The default value `10000` indicates a 10-second timeout duration. * Introduced in: - ##### `finish_transaction_default_lock_timeout_ms`[​](#finish_transaction_default_lock_timeout_ms "Direct link to finish_transaction_default_lock_timeout_ms") * Default: 1000 * Type: Int * Unit: MilliSeconds * Is mutable: Yes * Description: The default timeout for acquiring the db and table lock during finishing transaction. * Introduced in: v4.0.0, v3.5.8 ##### `history_job_keep_max_second`[​](#history_job_keep_max_second "Direct link to history_job_keep_max_second") * Default: 7 \* 24 \* 3600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The maximum duration a historical job can be retained, such as schema change jobs. * Introduced in: - ##### `insert_load_default_timeout_second`[​](#insert_load_default_timeout_second "Direct link to insert_load_default_timeout_second") * Default: 3600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The timeout duration for the INSERT INTO statement that is used to load data. * Introduced in: - ##### `label_clean_interval_second`[​](#label_clean_interval_second "Direct link to label_clean_interval_second") * Default: 4 \* 3600 * Type: Int * Unit: Seconds * Is mutable: No * Description: The time interval at which labels are cleaned up. Unit: second. We recommend that you specify a short time interval to ensure that historical labels can be cleaned up in a timely manner. * Introduced in: - ##### `label_keep_max_num`[​](#label_keep_max_num "Direct link to label_keep_max_num") * Default: 1000 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of load jobs that can be retained within a period of time. If this number is exceeded, the information of historical jobs will be deleted. * Introduced in: - ##### `label_keep_max_second`[​](#label_keep_max_second "Direct link to label_keep_max_second") * Default: 3 \* 24 \* 3600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The maximum duration in seconds to keep the labels of load jobs that have been completed and are in the FINISHED or CANCELLED state. The default value is 3 days. After this duration expires, the labels will be deleted. This parameter applies to all types of load jobs. A value too large consumes a lot of memory. * Introduced in: - ##### `load_checker_interval_second`[​](#load_checker_interval_second "Direct link to load_checker_interval_second") * Default: 5 * Type: Int * Unit: Seconds * Is mutable: No * Description: The time interval at which load jobs are processed on a rolling basis. * Introduced in: - ##### `load_parallel_instance_num`[​](#load_parallel_instance_num "Direct link to load_parallel_instance_num") * Default: 1 * Type: Int * Unit: - * Is mutable: Yes * Description: Controls the number of parallel load fragment instances created on a single host for broker and stream loads. LoadPlanner uses this value as the per-host degree of parallelism unless the session enables adaptive sink DOP; if the session variable `enable_adaptive_sink_dop` is true, the session`s `sink\_degree\_of\_parallelism\` overrides this configuration. When shuffle is required, this value is applied to fragment parallel execution (scan fragment and sink fragment parallel exec instances). When no shuffle is needed, it is used as the sink pipeline DOP. Note: loads from local files are forced to a single instance (pipeline DOP = 1, parallel exec = 1) to avoid local disk contention. Increasing this number raises per-host concurrency and throughput but may increase CPU, memory and I/O contention. * Introduced in: v3.2.0 ##### `load_straggler_wait_second`[​](#load_straggler_wait_second "Direct link to load_straggler_wait_second") * Default: 300 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The maximum loading lag that can be tolerated by a BE replica. If this value is exceeded, cloning is performed to clone data from other replicas. * Introduced in: - ##### `loads_history_retained_days`[​](#loads_history_retained_days "Direct link to loads_history_retained_days") * Default: 30 * Type: Int * Unit: Days * Is mutable: Yes * Description: Number of days to retain load history in the internal `_statistics_.loads_history` table. This value is used for table creation to set the table property `partition_live_number` and is passed to `TableKeeper` (clamped to a minimum of 1) to determine how many daily partitions to keep. Increasing or decreasing this value adjusts how long completed load jobs are retained in daily partitions; it affects new table creation and the keeper's pruning behavior but does not automatically recreate past partitions. The `LoadsHistorySyncer` relies on this retention when managing the loads history lifecycle; its sync cadence is controlled by `loads_history_sync_interval_second`. * Introduced in: v3.3.6, v3.4.0, v3.5.0 ##### `loads_history_sync_interval_second`[​](#loads_history_sync_interval_second "Direct link to loads_history_sync_interval_second") * Default: 60 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Interval (in seconds) used by LoadsHistorySyncer to schedule periodic syncs of finished load jobs from `information_schema.loads` into the internal `_statistics_.loads_history` table. The value is multiplied by 1000 in the constructor to set the FrontendDaemon interval. The syncer skips the first run (to allow table creation) and only imports loads that finished more than one minute ago; small values increase DML and executor load, while larger values delay availability of historical load records. See `loads_history_retained_days` for retention/partitioning behavior of the target table. * Introduced in: v3.3.6, v3.4.0, v3.5.0 ##### `max_broker_load_job_concurrency`[​](#max_broker_load_job_concurrency "Direct link to max_broker_load_job_concurrency") * Default: 5 * Alias: `async_load_task_pool_size` * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of concurrent Broker Load jobs allowed within the StarRocks cluster. This parameter is valid only for Broker Load. The value of this parameter must be less than the value of `max_running_txn_num_per_db`. From v2.5 onwards, the default value is changed from `10` to `5`. * Introduced in: - ##### `max_load_initial_open_partition_number`[​](#max_load_initial_open_partition_number "Direct link to max_load_initial_open_partition_number") * Default: 4096 * Type: Long * Unit: - * Is mutable: Yes * Description: The upper bound on how many partitions a load can open up front. The value is used as a cap in two scenarios: (1) for LIST-partitioned tables (which open all partitions by default) and (2) for RANGE-partitioned tables loaded via INSERT / Broker Load / Spark Load (which also open all partitions by default). Stream Load and Routine Load on RANGE-partitioned tables ignore this cap and keep the conservative latest-32 default. The per-table property `load_initial_open_partition_number` overrides this value, bypasses this cap, and is the highest-priority setting. From v4.0 onwards, the default value is increased from 32 to 4096. * Introduced in: - ##### `max_load_timeout_second`[​](#max_load_timeout_second "Direct link to max_load_timeout_second") * Default: 259200 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The maximum timeout duration allowed for a load job. The load job fails if this limit is exceeded. This limit applies to all types of load jobs. * Introduced in: - ##### `max_routine_load_batch_size`[​](#max_routine_load_batch_size "Direct link to max_routine_load_batch_size") * Default: 4294967296 * Type: Long * Unit: Bytes * Is mutable: Yes * Description: The maximum amount of data that can be loaded by a Routine Load task. * Introduced in: - ##### `max_routine_load_task_concurrent_num`[​](#max_routine_load_task_concurrent_num "Direct link to max_routine_load_task_concurrent_num") * Default: 5 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of concurrent tasks for each Routine Load job. * Introduced in: - ##### `max_routine_load_task_num_per_be`[​](#max_routine_load_task_num_per_be "Direct link to max_routine_load_task_num_per_be") * Default: 16 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of concurrent Routine Load tasks on each BE. Since v3.1.0, the default value for this parameter is increased to 16 from 5, and no longer needs to be less than or equal to the value of BE static parameter `routine_load_thread_pool_size` (deprecated). * Introduced in: - ##### `max_running_txn_num_per_db`[​](#max_running_txn_num_per_db "Direct link to max_running_txn_num_per_db") * Default: 1000 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of load transactions allowed to be running for each database within a StarRocks cluster. The default value is `1000`. From v3.1 onwards, the default value is changed to `1000` from `100`. When the actual number of load transactions running for a database exceeds the value of this parameter, new load requests will not be processed. New requests for synchronous load jobs will be denied, and new requests for asynchronous load jobs will be placed in queue. We do not recommend you increase the value of this parameter because this will increase system load. * Introduced in: - ##### `max_stream_load_timeout_second`[​](#max_stream_load_timeout_second "Direct link to max_stream_load_timeout_second") * Default: 259200 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The maximum allowed timeout duration for a Stream Load job. * Introduced in: - ##### `max_tolerable_backend_down_num`[​](#max_tolerable_backend_down_num "Direct link to max_tolerable_backend_down_num") * Default: 0 * Type: Int * Unit: - * Is mutable: Yes * Description: The maximum number of faulty BE nodes allowed. If this number is exceeded, Routine Load jobs cannot be automatically recovered. * Introduced in: - ##### `min_bytes_per_broker_scanner`[​](#min_bytes_per_broker_scanner "Direct link to min_bytes_per_broker_scanner") * Default: 67108864 * Type: Long * Unit: Bytes * Is mutable: Yes * Description: The minimum allowed amount of data that can be processed by a Broker Load instance. * Introduced in: - ##### `min_load_timeout_second`[​](#min_load_timeout_second "Direct link to min_load_timeout_second") * Default: 1 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The minimum timeout duration allowed for a load job. This limit applies to all types of load jobs. * Introduced in: - ##### `min_routine_load_lag_for_metrics`[​](#min_routine_load_lag_for_metrics "Direct link to min_routine_load_lag_for_metrics") * Default: 10000 * Type: INT * Unit: - * Is mutable: Yes * Description: The minimum offset lag of Routine Load jobs to be shown in monitoring metrics. Routine Load jobs whose offset lags are greater than this value will be displayed in the metrics. * Introduced in: - ##### `period_of_auto_resume_min`[​](#period_of_auto_resume_min "Direct link to period_of_auto_resume_min") * Default: 5 * Type: Int * Unit: Minutes * Is mutable: Yes * Description: The interval at which Routine Load jobs are automatically recovered. * Introduced in: - ##### `prepared_transaction_default_timeout_second`[​](#prepared_transaction_default_timeout_second "Direct link to prepared_transaction_default_timeout_second") * Default: 86400 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The default timeout duration for a prepared transaction. * Introduced in: - ##### `routine_load_task_consume_second`[​](#routine_load_task_consume_second "Direct link to routine_load_task_consume_second") * Default: 15 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The maximum time for each Routine Load task within the cluster to consume data. Since v3.1.0, Routine Load job supports a new parameter `task_consume_second` in `job_properties`. This parameter applies to individual load tasks within a Routine Load job, which is more flexible. * Introduced in: - ##### `routine_load_task_timeout_second`[​](#routine_load_task_timeout_second "Direct link to routine_load_task_timeout_second") * Default: 60 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: The timeout duration for each Routine Load task within the cluster. Since v3.1.0, Routine Load job supports a new parameter `task_timeout_second` in `job_properties`. This parameter applies to individual load tasks within a Routine Load job, which is more flexible. * Introduced in: - ##### `routine_load_unstable_threshold_second`[​](#routine_load_unstable_threshold_second "Direct link to routine_load_unstable_threshold_second") * Default: 3600 * Type: Long * Unit: Seconds * Is mutable: Yes * Description: Routine Load job is set to the UNSTABLE state if any task within the Routine Load job lags. To be specific, the difference between the timestamp of the message being consumed and the current time exceeds this threshold, and unconsumed messages exist in the data source. * Introduced in: - ##### `spark_dpp_version`[​](#spark_dpp_version "Direct link to spark_dpp_version") * Default: 1.0.0 * Type: String * Unit: - * Is mutable: No * Description: The version of Spark Dynamic Partition Pruning (DPP) used. * Introduced in: - ##### `spark_home_default_dir`[​](#spark_home_default_dir "Direct link to spark_home_default_dir") * Default: `StarRocksFE.STARROCKS_HOME_DIR` + "/lib/spark2x" * Type: String * Unit: - * Is mutable: No * Description: The root directory of a Spark client. * Introduced in: - ##### `spark_launcher_log_dir`[​](#spark_launcher_log_dir "Direct link to spark_launcher_log_dir") * Default: `sys_log_dir` + "/spark\_launcher\_log" * Type: String * Unit: - * Is mutable: No * Description: The directory that stores Spark log files. * Introduced in: - ##### `spark_load_default_timeout_second`[​](#spark_load_default_timeout_second "Direct link to spark_load_default_timeout_second") * Default: 86400 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The timeout duration for each Spark Load job. * Introduced in: - ##### `spark_load_submit_timeout_second`[​](#spark_load_submit_timeout_second "Direct link to spark_load_submit_timeout_second") * Default: 300 * Type: long * Unit: Seconds * Is mutable: No * Description: Maximum time in seconds to wait for a YARN response after submitting a Spark application. `SparkLauncherMonitor.LogMonitor` converts this value to milliseconds and will stop monitoring and forcibly kill the spark launcher process if the job remains in UNKNOWN/CONNECTED/SUBMITTED longer than this timeout. `SparkLoadJob` reads this configuration as the default and allows a per-load override via the `LoadStmt.SPARK_LOAD_SUBMIT_TIMEOUT` property. Set it high enough to accommodate YARN queueing delays; setting it too low may abort legitimately queued jobs, while setting it too high may delay failure handling and resource cleanup. * Introduced in: v3.2.0 ##### `spark_resource_path`[​](#spark_resource_path "Direct link to spark_resource_path") * Default: Empty string * Type: String * Unit: - * Is mutable: No * Description: The root directory of the Spark dependency package. * Introduced in: - ##### `stream_load_default_timeout_second`[​](#stream_load_default_timeout_second "Direct link to stream_load_default_timeout_second") * Default: 600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The default timeout duration for each Stream Load job. * Introduced in: - ##### `stream_load_max_txn_num_per_be`[​](#stream_load_max_txn_num_per_be "Direct link to stream_load_max_txn_num_per_be") * Default: -1 * Type: Int * Unit: Transactions * Is mutable: Yes * Description: Limits the number of concurrent stream-load transactions accepted from a single BE (backend) host. When set to a non-negative integer, FrontendServiceImpl checks the current transaction count for the BE (by client IP) and rejects new stream-load begin requests if the count `>=` this limit. A value of `< 0` disables the limit (unlimited). This check occurs during stream load begin and may cause a `streamload txn num per be exceeds limit` error when exceeded. Related runtime behavior uses `stream_load_default_timeout_second` for request timeout fallback. * Introduced in: v3.3.0, v3.4.0, v3.5.0 ##### `stream_load_task_keep_max_num`[​](#stream_load_task_keep_max_num "Direct link to stream_load_task_keep_max_num") * Default: 1000 * Type: Int * Unit: - * Is mutable: Yes * Description: Maximum number of Stream Load tasks that StreamLoadMgr keeps in memory (global across all databases). When the number of tracked tasks (`idToStreamLoadTask`) exceeds this threshold, StreamLoadMgr first calls `cleanSyncStreamLoadTasks()` to remove completed synchronous stream-load tasks; if the size still remains greater than half of this threshold, it invokes `cleanOldStreamLoadTasks(true)` to force removal of older or finished tasks. Increase this value to retain more task history in memory; decrease it to reduce memory usage and make cleanup more aggressive. This value controls in-memory retention only and does not affect persisted/replayed tasks. * Introduced in: v3.2.0 ##### `stream_load_task_keep_max_second`[​](#stream_load_task_keep_max_second "Direct link to stream_load_task_keep_max_second") * Default: 3 \* 24 \* 3600 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: Retention window for finished or cancelled Stream Load tasks. After a task reaches a final state and its end timestamp is ealier than this threshold (`currentMs - endTimeMs > stream_load_task_keep_max_second * 1000`), it becomes eligible for removal by `StreamLoadMgr.cleanOldStreamLoadTasks` and is discarded when loading persisted state. Applies to both `StreamLoadTask` and `StreamLoadMultiStmtTask`. If total task count exceeds `stream_load_task_keep_max_num`, cleanup may be triggered earlier (synchronous tasks are prioritized by `cleanSyncStreamLoadTasks`). Set this to balance history/debugability against memory usage. * Introduced in: v3.2.0 ##### `transaction_clean_interval_second`[​](#transaction_clean_interval_second "Direct link to transaction_clean_interval_second") * Default: 30 * Type: Int * Unit: Seconds * Is mutable: No * Description: The time interval at which finished transactions are cleaned up. Unit: second. We recommend that you specify a short time interval to ensure that finished transactions can be cleaned up in a timely manner. * Introduced in: - ##### `transaction_stream_load_coordinator_cache_capacity`[​](#transaction_stream_load_coordinator_cache_capacity "Direct link to transaction_stream_load_coordinator_cache_capacity") * Default: 4096 * Type: Int * Unit: - * Is mutable: Yes * Description: The capacity of the cache that stores the mapping from transaction label to coordinator node. * Introduced in: - ##### `transaction_stream_load_coordinator_cache_expire_seconds`[​](#transaction_stream_load_coordinator_cache_expire_seconds "Direct link to transaction_stream_load_coordinator_cache_expire_seconds") * Default: 900 * Type: Int * Unit: Seconds * Is mutable: Yes * Description: The time to keep the coordinator mapping in the cache before it's evicted(TTL). * Introduced in: - ##### `yarn_client_path`[​](#yarn_client_path "Direct link to yarn_client_path") * Default: `StarRocksFE.STARROCKS_HOME_DIR` + "/lib/yarn-client/hadoop/bin/yarn" * Type: String * Unit: - * Is mutable: No * Description: The root directory of the Yarn client package. * Introduced in: - ##### `yarn_config_dir`[​](#yarn_config_dir "Direct link to yarn_config_dir") * Default: `StarRocksFE.STARROCKS_HOME_DIR` + "/lib/yarn-config" * Type: String * Unit: - * Is mutable: No * Description: The directory that stores the Yarn configuration file. * Introduced in: - --- ### Graceful Exit From v3.3 onwards, StarRocks supports Graceful Exit. #### Overview[​](#overview "Direct link to Overview") Graceful Exit is a mechanism designed to support **non-disruptive upgrades and restarts** of StarRocks FE, BE, and CN nodes. Its primary objective is to minimize the impact on running queries and data ingestion tasks during maintenance operations such as node restart, rolling upgrade, or cluster scaling. Graceful Exit ensures that: * The node **stops accepting new tasks** once exit begins; * Existing queries and load jobs are **allowed to complete** within a controlled time window; * System components (FE/BE/CN) **coordinate status changes** so that the cluster correctly reroutes traffic. Graceful Exit mechanisms differ between FE and BE/CN nodes, as described below. ##### FE Graceful Exit Mechanism[​](#fe-graceful-exit-mechanism "Direct link to FE Graceful Exit Mechanism") ###### Trigger Signal[​](#trigger-signal "Direct link to Trigger Signal") FE Graceful Exit is initiated via: ```bash stop_fe.sh -g ``` This sends a `SIGUSR1` signal, while the default exit (without the `-g` option) sends `SIGTERM` signal. ###### Load Balancer Awareness[​](#load-balancer-awareness "Direct link to Load Balancer Awareness") Upon receiving the signal: * FE immediately returns **HTTP 500** on the `/api/health` endpoint. * Load balancers detect the degraded state within ~15 seconds and stop routing new connections to this FE. ###### Connection Drain and Shutdown Logic[​](#connection-drain-and-shutdown-logic "Direct link to Connection Drain and Shutdown Logic") **Follower FE** * Handles read-only queries. * If the FE node has no active sessions, the connection is closed immediately. * If SQL is running, the FE node waits for execution to finish before closing the session. **Leader FE** * Read request handling is identical to the of Followers. * Write request handling requires: * Closing BDBJE. * Allowing a new Leader election to complete. * Redirecting subsequent writes to the newly elected Leader. ###### Timeout Control[​](#timeout-control "Direct link to Timeout Control") If a query runs for too long, FE forcibly exits after **60 seconds** (configurable via the `--timeout` option). ##### BE/CN Graceful Exit Mechanism[​](#becn-graceful-exit-mechanism "Direct link to BE/CN Graceful Exit Mechanism") ###### Trigger Signal[​](#trigger-signal-1 "Direct link to Trigger Signal") BE Graceful Exit is initiated via: ```bash stop_be.sh -g ``` CN Graceful Exit is initiated via: ```bash stop_cn.sh -g ``` This sends a `SIGTERM` signal, while the default exit (without the `-g` option) sends `SIGKILL` signal. ###### State Transition[​](#state-transition "Direct link to State Transition") After receiving the signal: * The BE/CN node marks itself as **exiting**. * It rejects **new query fragments** by returning `INTERNAL_ERROR`. * It continues processing existing fragments. ###### Wait Loop for In-Flight Queries[​](#wait-loop-for-in-flight-queries "Direct link to Wait Loop for In-Flight Queries") The behavior that BE/CN waits for existing fragments to finish is controlled by the BE/CN configuration `loop_count_wait_fragments_finish` (Default: 2). The actual wait duration equals `loop_count_wait_fragments_finish × 10 seconds` (that is, 20 seconds by default). If fragments remain after timeout, BE/CN proceeds with normal shutdown (closing threads, network, and other processes). ###### Improved FE Awareness[​](#improved-fe-awareness "Direct link to Improved FE Awareness") From v3.4 onwards, FE no longer marks BE/CN as `DEAD` based on heartbeat failures. It correctly recognizes the BE/CN “exiting” state, allowing significantly longer graceful-exit windows for fragments to be completed. #### Configurations[​](#configurations "Direct link to Configurations") ##### FE Configurations[​](#fe-configurations "Direct link to FE Configurations") ###### `stop_fe.sh -g --timeout`[​](#stop_fesh--g---timeout "Direct link to stop_fesh--g---timeout") * Description: Maximum waiting time before FE is force-killed. * Default: 60 (seconds) * How to apply: Specify it in the script command, for example, `--timeout 120`. ###### *Minimum LB detection time*[​](#minimum-lb-detection-time "Direct link to minimum-lb-detection-time") * Description: LB requires at least 15 seconds to detect degraded health. * Default: 15 (seconds) * How to apply: Fixed value ##### BE/CN Configurations[​](#becn-configurations "Direct link to BE/CN Configurations") ###### `loop_count_wait_fragments_finish`[​](#loop_count_wait_fragments_finish "Direct link to loop_count_wait_fragments_finish") * Description: BE/CN wait duration for existing fragments. Multiply the value with 10 seconds. * Default: 2 * How to apply: Modify it in the BE/CN configuration file or update it dynamically. ###### `graceful_exit_wait_for_frontend_heartbeat`[​](#graceful_exit_wait_for_frontend_heartbeat "Direct link to graceful_exit_wait_for_frontend_heartbeat") * Description: Whether BE/CN waits for FE to confirm **SHUTDOWN** via heartbeat. From v3.4.5 onwards. * Default: false * How to apply: Modify it in the BE/CN configuration file or update it dynamically. ###### `stop_be.sh -g --timeout`, `stop_cn.sh -g --timeout`[​](#stop_besh--g---timeout-stop_cnsh--g---timeout "Direct link to stop_besh--g---timeout-stop_cnsh--g---timeout") * Description: Maximum waiting time before BE/CN is force-killed. Set it to a value larger than `loop_count_wait_fragments_finish` \* 10 to prevent termination before the BE/CN wait duration is reached. * Default: false * How to apply: Specify it in the script command, for example, `--timeout 30`. ##### Global Switches[​](#global-switches "Direct link to Global Switches") Graceful Exit is **enabled by default** from v3.4 onwards. To disable it temporarily, set the BE/CN configuration `loop_count_wait_fragments_finish` to `0`. #### Expected Behavior During Graceful Exit[​](#expected-behavior-during-graceful-exit "Direct link to Expected Behavior During Graceful Exit") ##### Query Workloads[​](#query-workloads "Direct link to Query Workloads") | Query Type | Expected Behavior | | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | | **Short (less than 20 seconds)** | BE/CN waits long enough, so queries complete normally. | | **Medium (20 to 60 seconds)** | Queries completed within BE/CN wait window are returned with success; else queries are cancelled and require manual retry. | | **Long (more than 60 seconds)** | Queries are likely terminated by FE/BE/CN due to timeout and requires manual retry. | ##### Data Ingestion Tasks[​](#data-ingestion-tasks "Direct link to Data Ingestion Tasks") * **Loading tasks via Flink or Kafka Connectors** are automatically retried with no user-visible interruption. * **Stream Load (non-framework), Broker Load, and Routine Load tasks** may fail if connection breaks. Manual retry is required. * **Background tasks** are automatically re-scheduled and executed by the FE retry mechanism. ##### Upgrade and Restart Operations[​](#upgrade-and-restart-operations "Direct link to Upgrade and Restart Operations") Graceful Exit ensures: * No cluster-wide downtime; * Safe rolling upgrade by draining nodes one at a time. #### Limitations and Version Differences[​](#limitations-and-version-differences "Direct link to Limitations and Version Differences") ##### Version Behavior Differences[​](#version-behavior-differences "Direct link to Version Behavior Differences") | Version | Behavior | | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | **v3.3** | BE Graceful Exit flawed: FE may prematurely mark BE/CN as `DEAD`, causing queries to get cancelled. The effective wait is limited (15 seconds by default). | | **v3.4+** | Fully supports extended wait duration; FE correctly identifies BE/CN “exiting” state. Recommended for production. | ##### Operational Limitations[​](#operational-limitations "Direct link to Operational Limitations") * In extreme cases (for example, BE/CN hangs), Graceful Exit may fail. Terminating the process requires `kill -9`, risking partial data persistence (recoverable via snapshot). #### Usage[​](#usage "Direct link to Usage") ##### Prerequisites[​](#prerequisites "Direct link to Prerequisites") **StarRocks version**: * **v3.3+**: Basic Graceful Exit support. * **v3.4+**: Enhanced status management, longer wait windows (up to several minutes). **Configuration**: * Make sure `loop_count_wait_fragments_finish` is set to a positive integer. * Set `graceful_exit_wait_for_frontend_heartbeat` to `true` allow FE to detect BE's "EXITING" state. ##### Perform FE Graceful Exit[​](#perform-fe-graceful-exit "Direct link to Perform FE Graceful Exit") ```bash ./bin/stop_fe.sh -g --timeout 60 ``` Parameters: * `--timeout`: The maximum time to wait before the FE node is force-killed. Behavior: * The system sends the `SIGUSR1` signal first. * After timeout, it falls back to `SIGKILL`. ###### Validate FE State[​](#validate-fe-state "Direct link to Validate FE State") You can check the FE health via the following API: ```text http://:8030/api/health ``` LB removes the node after receiving consecutive non-200 responses. ##### Perform BE/CN Graceful Exit[​](#perform-becn-graceful-exit "Direct link to Perform BE/CN Graceful Exit") * **For v3.3:** * BE: ```bash ./be/bin/stop_be.sh -g ``` * CN: ```bash ./be/bin/stop_cn.sh -g ``` * **For v3.4+:** * BE: ```bash ./bin/stop_be.sh -g --timeout 600 ``` * CN: ```bash ./bin/stop_cn.sh -g --timeout 600 ``` BE/CN exits immediately if no fragments remain. ###### Validate BE/CN Status[​](#validate-becn-status "Direct link to Validate BE/CN Status") Run on FE: ```sql SHOW BACKENDS; ``` `StatusCode`: * `SHUTDOWN`: BE/CN Graceful Exit in progress. * `DISCONNECTED`: BE/CN Node has fully exited. #### Rolling Upgrade Workflow[​](#rolling-upgrade-workflow "Direct link to Rolling Upgrade Workflow") ##### Procedure[​](#procedure "Direct link to Procedure") 1. Perform Graceful Exit on the node `A`. 2. Confirm the node `A` is shown as `DISCONNECTED` from the FE side. 3. Upgrade and restart the node `A`. 4. Repeat the above for remaining nodes. ##### Monitor Graceful Exit[​](#monitor-graceful-exit "Direct link to Monitor Graceful Exit") Check FE logs `fe.log`, BE logs `be.log`, or CN logs `cn.log` to make sure whether there were tasks during the exit. #### Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") ##### BE/CN exits by timeout[​](#becn-exits-by-timeout "Direct link to BE/CN exits by timeout") If tasks fail to complete within the Graceful Exit period, BE/CN will trigger forced termination (`SIGKILL`). Verify whether this is caused by excessively long task duration or improper configurations (for example, an overly small `--timeout` value). ##### Node status is not SHUTDOWN[​](#node-status-is-not-shutdown "Direct link to Node status is not SHUTDOWN") If the node status is not `SHUTDOWN`, verify whether `loop_count_wait_fragments_finish` is set to a positive integer, or if BE/CN reported a heartbeat before exiting (if not, set `graceful_exit_wait_for_frontend_heartbeat` to `true`). --- ### logs When deploying and operating StarRocks, understanding and properly using the logging system is critical for troubleshooting, performance analysis, and system tuning. This article provides a detailed overview of the log file types, typical content, configuration methods, and log rolling and retention strategies for both the Frontend (FE) and Backend (BE or CN) components of StarRocks. The information in this document is based on StarRocks Version 3.5.x. #### FE Logging in detail[​](#fe-logging-in-detail "Direct link to FE Logging in detail") ##### `fe.log`[​](#felog "Direct link to felog") The main FE logs include the startup process, cluster state changes, DML/DQL requests, and scheduling-related information. These logs primarily record the behavior of the FE during its runtime. ###### Configuration[​](#configuration "Direct link to Configuration") * `sys_log_dir`: Log storage directory. Default is `${STARROCKS_HOME}/log` * `sys_log_level`: Log level. Default is `INFO` * `sys_log_roll_num`: Controls the number of retained log files to prevent unlimited growth from consuming too much disk space. Default is 10 * `sys_log_roll_interval`: Specifies the rotation frequency. Default is `DAY`, meaning logs are rotated daily * `sys_log_delete_age`: Controls how long to keep old log files before deletion. Default is 7 day * `sys_log_roll_mode`: Log rotation mode. Default is `SIZE-MB-1024`, meaning a new log file will be created when the current one reaches 1024 MB. Together with sys\_log\_roll\_interval, this indicates that FE logs can be rotated either daily or based on file size * `sys_log_enable_compress`: Controls whether log compression is enabled. Default is false, meaning compression is disabled ##### `fe.warn.log`[​](#fewarnlog "Direct link to fewarnlog") `fe.warn.log` is an important log file for system monitoring and troubleshooting: * Operations monitoring–monitors system health status * Fault diagnosis–quickly locates critical issues * Performance analysis–identifies system bottlenecks and anomalies * Security auditing–records permission and access errors Compared to `fe.log`, which records logs of all levels, `fe.warn.log` focuses on warnings and errors that require attention, helping operations personnel quickly identify and address system issues. ##### `fe.gc.log`[​](#fegclog "Direct link to fegclog") `fe.gc.log` is the Java garbage collection log of the StarRocks FE, used to monitor and analyze JVM garbage collection behavior. It’s important to note that this log file uses the native JVM log rotation mechanism. For example, you can enable automatic rotation based on file size and file count with the following configuration: ```bash -Xlog:gc*:${LOG_DIR}/fe.gc.log:time,tags:filecount=7,filesize=100M ``` ##### `fe.out`[​](#feout "Direct link to feout") `fe.out` is the standard output log file of StarRocks FE. It records the content printed to standard output (stdout) and standard error (stderr) during the runtime of the FE process. The main content includes: * Console output during FE startup * JVM startup information (e.g., heap settings, GC parameters) * FE module initialization order (Catalog, Scheduler, RPC, HTTP Server, etc.) * Error stack traces from stderr * Java exceptions (Exception/StackTrace) * Uncaught errors like ClassNotFound, NullPointerException, etc. * Output not captured by other logging systems * Some third-party libraries that use `System.out.println()` or `e.printStackTrace()` You should check `fe.out` in the following scenarios: * FE fails to start: Check `fe.out` for Java exceptions or invalid parameter messages. * FE crashes unexpectedly: Look for uncaught exception stack traces. * Logging system uncertainty: If `fe.out` is the only log file with output in the FE log directory, it’s likely that `log4j` failed to initialize, or the configuration is incorrect. By default, `fe.out` does not support automatic log rotation. ##### `fe.profile.log`[​](#feprofilelog "Direct link to feprofilelog") The purpose of `fe.profile.log` is to record detailed query execution information for performance analysis. Its main functions include: * Query performance analysis: Logs detailed execution data for each query, including: * Query ID, user, database, SQL statement * Execution timing (startTime, endTime, latency) * Resource usage (CPU, memory, number/size of scanned rows) * Execution status (RUNNING, FINISHED, FAILED, CANCELLED) * Runtime metrics tracking: Captures key indicators via the QueryDetail class: * `scanRows / scanBytes`: Amount of data scanned * `returnRows`: Number of result rows returned * `cpuCostNs`: CPU time consumed (in nanoseconds) * `memCostBytes`: Memory usage * `spillBytes`: Amount of data spilled to disk * Error diagnosis: Records error messages and stack traces for failed queries * Resource group monitoring: Tracks query execution metrics across different resource groups `fe.profile.log` is stored in JSON format. ###### Configuration[​](#configuration-1 "Direct link to Configuration") * `enable_profile_log`: Whether to enable profile logging * `profile_log_dir`: Directory for storing profile logs * `profile_log_roll_size_mb`: Log rotation size (in MB) * `profile_log_roll_num`: Controls the number of retained profile log files to prevent unlimited growth and excessive disk usage. Default is 5 * `profile_log_roll_interval`: Specifies the rotation frequency. Default is DAY, meaning daily rotation. When rotation conditions are met, the latest 5 files are retained, and older files are deleted * `profile_log_delete_age`: Controls how long old files are kept before deletion. Default is 1 day ##### `fe.internal.log`[​](#feinternallog "Direct link to feinternallog") The purpose of `fe.internal.log` is to record logs dedicated to internal operations of the FE (Frontend), primarily for system-level auditing and debugging. Its main functions include: 1. Internal operation auditing: Logs system-initiated internal SQL executions separately from user queries 2. Statistics tracking: Specifically records operations related to statistics collection 3. Debugging support: Provides detailed logs for troubleshooting internal operation issues The log records include entries such as: * Statistics module (internal.statistic) * Core system module (internal.base) This log is especially useful for analyzing StarRocks’ internal statistics collection process and diagnosing issues related to internal operations. ###### Configuration[​](#configuration-2 "Direct link to Configuration") * `internal_log_dir`: Controls the storage directory for this log * `internal_log_modules`: An array configuring internal log modules, defining which internal operation modules need to be recorded in the `fe.internal.log` file. Default is `{"base", "statistic"}` * `internal_log_roll_num`: Number of files to retain. Default is 90 * `internal_log_roll_interval`: Specifies the rotation frequency. Default is DAY, meaning daily rotation. When rotation conditions are met, the latest 90 files are retained, and older files are deleted * `internal_log_delete_age`: Controls how long old files are kept before deletion. Default is 7 days ##### `fe.audit.log`[​](#feauditlog "Direct link to feauditlog") This is StarRocks’ query audit log, which records detailed information about all user queries and connections. It is used for monitoring, analysis, and auditing. Its main purposes include: 1. Query monitoring: Logs the execution status and performance metrics of all SQL queries 2. User auditing: Tracks user behavior and database access 3. Performance analysis: Provides metrics such as query execution time and resource consumption 4. Issue diagnosis: Records error statuses and error codes to facilitate troubleshooting ###### Configuration[​](#configuration-3 "Direct link to Configuration") * `audit_log_dir`: Controls the storage directory for this log * `audit_log_roll_num`: Number of files to retain. Default is 90 * `audit_log_roll_interval`: Specifies the rotation frequency. Default is DAY, meaning daily rotation. When rotation conditions are met, the latest 90 files are retained, and older files are deleted * `audit_log_delete_age`: Controls how long old files are kept before deletion. Default is 7 days * `audit_log_json_format`: Whether to log in JSON format. Default is false * `audit_log_enable_compress`: Whether compression is enabled ##### `fe.big_query.log`[​](#febig_querylog "Direct link to febig_querylog") This is StarRocks’ dedicated Big Query log file, used to monitor and analyze queries with high resource consumption. Its structure is similar to the audit log, but it includes three additional fields: * `bigQueryLogCPUSecondThreshold`: CPU time threshold * `bigQueryLogScanBytesThreshold`: Scan size (in bytes) threshold * `bigQueryLogScanRowsThreshold`: Scan row count threshold ###### Configuration[​](#configuration-4 "Direct link to Configuration") * `big_query_log_dir`: Controls the storage directory for this log * `big_query_log_roll_num`: Number of files to retain. Default is 10 * `big_query_log_modules`: Types of internal log modules. Default is query * `big_query_log_roll_interval`: Specifies the rotation frequency. Default is DAY, meaning daily rotation. When rotation conditions are met, the latest 10 files are retained, and older files are deleted * `big_query_log_delete_age`: Controls how long old files are kept before deletion. Default is 7 days ##### `fe.dump.log`[​](#fedumplog "Direct link to fedumplog") This is StarRocks’ query dump log, specifically used for detailed query debugging and issue diagnosis. Its main purposes include: * Exception debugging: Automatically records the complete query context when query execution encounters exceptions * Issue reproduction: Provides sufficiently detailed information to reproduce query problems * In-depth diagnosis: Contains debugging information such as metadata, statistics, execution plans, and more * Technical support: Provides comprehensive data for the technical support team to analyze issues It can be enabled using the following command: ```bash SET enable_query_dump = true; ``` ###### Configuration[​](#configuration-5 "Direct link to Configuration") * `dump_log_dir`: Controls the storage directory for this log * `dump_log_roll_num`: Number of files to retain. Default is 10 * `dump_log_modules`: Types of internal log modules. Default is query * `dump_log_roll_interval`: Specifies the rotation frequency. Default is DAY, meaning daily rotation. When rotation conditions are met, the latest 10 files are retained, and older files are deleted * `dump_log_delete_age`: Controls how long old files are kept before deletion. Default is 7 days ##### `fe.features.log`[​](#fefeatureslog "Direct link to fefeatureslog") This is StarRocks’ query plan feature log, used to collect and record feature information of query execution plans. It mainly serves machine learning and query optimization analysis. Key purposes include: 1. Query plan feature collection: Extracting various characteristics from query execution plans 2. Machine learning data source: Providing training data for query cost prediction models 3. Query pattern analysis: Analyzing execution patterns and feature distributions 4. Optimizer improvement: Supplying data to support enhancements in the cost-based optimizer (CBO) It can be enabled via configuration. ```bash // Enable plan feature collection enable_plan_feature_collection = false // Disabled by default // Enable query cost prediction enable_query_cost_prediction = false // Disabled by default ``` ###### Configuration[​](#configuration-6 "Direct link to Configuration") * `feature_log_dir`: Controls the storage directory for this log * `feature_log_roll_num`: Number of files to retain. Default is 5 * `feature_log_roll_interval`: Specifies the rotation frequency. Default is DAY, meaning daily rotation. When rotation conditions are met, the latest 5 files are retained, and older files are deleted * `feature_log_delete_age`: Controls how long old files are kept before deletion. Default is 3 days * `feature_log_roll_size_mb`: Log rotation size. Default is 1024 MB, meaning a new file is created every 1 GB #### BE/CN Logging in detail[​](#becn-logging-in-detail "Direct link to BE/CN Logging in detail") ##### `{be or cn}.INFO.log`[​](#be-or-cninfolog "Direct link to be-or-cninfolog") It primarily records various runtime behavior logs generated by BE/CN nodes, and these logs are at the `INFO` level. For example: * System startup information: * BE process startup and initialization * Hardware resource detection (CPU, memory, disk) * Configuration parameter loading * Query execution information: * Query reception and dispatch * Fragment execution status * Storage-related information: * Tablet loading and unloading * Compaction execution process * Data import status * Storage space management ###### Configuration[​](#configuration-7 "Direct link to Configuration") * `sys_log_level`: Log level, default is `INFO` * `sys_log_dir`: Log storage directory, default is `${STARROCKS_HOME}/log` * `sys_log_roll_mode`: Log rotation mode, default is `SIZE-MB-1024`, meaning a new log file is created when the current one reaches 1024 MB * `sys_log_roll_num`: Number of retained log files, default is 10 ##### `{be or cn}.WARN.log`[​](#be-or-cnwarnlog "Direct link to be-or-cnwarnlog") `be.WARN.log` stores log entries at WARNING level and above. Examples include Query execution warnings: * Query execution time too long * Memory allocation failure warning * Operator execution exception Storage-related warnings: * Abnormal Tablet status * Slow compaction execution * Data file corruption warning * Storage I/O exception ###### Configuration[​](#configuration-8 "Direct link to Configuration") * `sys_log_level`: Log level, default is INFO * `sys_log_dir`: Log storage directory, default is `${STARROCKS_HOME}/log` * `sys_log_roll_mode`: Log rotation mode, default is `SIZE-MB-1024`, meaning a new log file is created when the current one reaches 1024 MB * `sys_log_roll_num`: Number of retained log files, default is 10 ##### `{be or cn}.ERROR.log`[​](#be-or-cnerrorlog "Direct link to be-or-cnerrorlog") `be.ERROR.log` stores log entries at `ERROR` level and above. Typical error log contents: Query Execution Errors * Query timeout or cancellation * Query failure due to insufficient memory Data Processing Errors * Data load failures (e.g., format errors, constraint violations) * Data write failures * Data read errors (e.g., file corruption, I/O errors) Storage System Errors * Tablet load failures * Compaction execution failures * Corrupted data files * Disk I/O errors ###### Configuration[​](#configuration-9 "Direct link to Configuration") * `sys_log_level`: Log level, default is `INFO` * `sys_log_dir`: Log storage directory, default is `${STARROCKS_HOME}/log` * `sys_log_roll_mode`: Log rotation mode, default is `SIZE-MB-1024`, meaning a new log file is created when the current one reaches 1024 MB * `sys_log_roll_num`: Number of retained log files, default is 10 ##### `{be or cn}.FATAL.log`[​](#be-or-cnfatallog "Direct link to be-or-cnfatallog") It primarily records various runtime behavior logs generated by BE/CN nodes, and these logs are at the `FATAL` level. Once such a log is generated, the BE/CN node process will exit. ###### Configuration[​](#configuration-10 "Direct link to Configuration") * `sys_log_level`: Log level, default is `INFO` * `sys_log_dir`: Log storage directory, default is `${STARROCKS_HOME}/log` * `sys_log_roll_mode`: Log rotation mode, default is `SIZE-MB-1024`, meaning a new log file is created when the current one reaches 1024 MB * `sys_log_roll_num`: Number of retained log files, default is 10 ##### `error_log`[​](#error_log "Direct link to error_log") This log primarily records various errors, rejected records, and ETL issues encountered by BE/CN nodes during data import. Users can obtain the main reasons for import errors via `http://be_ip:be_port/api/get_log_file`. The log files are stored in the `${STARROCKS_HOME}/storage/error_log` directory. ###### Configuration[​](#configuration-11 "Direct link to Configuration") * `load_error_log_reserve_hours`: How long error log files are retained. The default is 48 hours, meaning the log files will be deleted 48 hours after they are created. --- ### Monitor and manage big queries This topic describes how to monitor and manage big queries in your StarRocks cluster. Big queries include queries that scan too many rows or occupy too many CPU and memory resources. They can easily exhaust cluster resources and cause system overload if no restrictions are imposed on them. To tackle this issue, StarRocks provides a series of measures to monitor and manage big queries, preventing queries from monopolizing cluster resources. The overall idea of handling big queries in StarRocks is as follows: 1. Set automatic precautions against big queries with resource groups and query queues. 2. Monitor big queries in real-time, and terminate those who bypass the precautions. 3. Analyze audit logs and Big Query Logs to study the patterns of big queries, and fine-tune the precaution mechanisms you set earlier. This feature is supported from v3.0. #### Set precautions against big queries[​](#set-precautions-against-big-queries "Direct link to Set precautions against big queries") StarRocks provides two precautionary instruments for dealing with big queries - resource groups and query queues. You can use resource groups to stop big queries from being executed. Query queues, on the other hand, can help you queue the incoming queries when the concurrency threshold or resource limit is reached, preventing system overload. ##### Filter out big queries via resource groups[​](#filter-out-big-queries-via-resource-groups "Direct link to Filter out big queries via resource groups") Resource groups can automatically identify and terminate big queries. When creating a resource group, you can specify the upper limit of CPU time, memory usage, or scan row count that a query is entitled to. Among all queries that hit the resource group, any queries that require more resources are rejected and returned with an error. For more information and instructions on resource groups, see [Resource Isolation](https://docs.starrocks.io/docs/administration/management/resource_management/resource_group.md). Before creating resource groups, you must execute the following statement to enable Pipeline Engine, on which the Resource Group feature depends: ```sql SET GLOBAL enable_pipeline_engine = true; ``` The following example creates a resource group `bigQuery` that limits the CPU time upper limit to `100` seconds, scan row count upper limit to `100000`, and memory usage upper limit to `1073741824` bytes (1 GB): ```sql CREATE RESOURCE GROUP bigQuery TO (db='sr_hub') WITH ( 'cpu_weight' = '10', 'mem_limit' = '20%', 'big_query_cpu_second_limit' = '100', 'big_query_scan_rows_limit' = '100000', 'big_query_mem_limit' = '1073741824' ); ``` If the required resources of a query exceed any of the limits, the query will not be executed and is returned with an error. The following example shows the error message returned when a query demands too many scan rows: ```plain ERROR 1064 (HY000): exceed big query scan_rows limit: current is 4 but limit is 1 ``` If it is your first time setting up resource groups, we recommend you set relatively higher limits so that they will not hinder regular queries. You can fine-tune these limits after you have a better knowledge of the big query patterns. ##### Ease system overload via query queues[​](#ease-system-overload-via-query-queues "Direct link to Ease system overload via query queues") Query queues are designed to cushion the system overload deterioration when the cluster resource occupation exceeds the prespecified thresholds. You can set thresholds for maximum concurrency, memory usage, and CPU usage. StarRocks automatically queues the incoming queries when any of these thresholds is reached. Pending queries either wait in the queue for execution or get cancelled when the prespecified resource threshold is reached. For more information, see [Query Queues](https://docs.starrocks.io/docs/administration/management/resource_management/query_queues.md). Execute the following statements to enable query queues for the SELECT queries: ```sql SET GLOBAL enable_query_queue_select = true; ``` After the query queue feature is enabled, you can then define the rules to trigger query queues. * Specify the concurrency threshold for triggering the query queue. The following example sets the concurrency threshold to `100`: ```sql SET GLOBAL query_queue_concurrency_limit = 100; ``` * Specify the memory usage ratio threshold for triggering the query queue. The following example sets the memory usage ratio threshold to `0.9`: ```sql SET GLOBAL query_queue_mem_used_pct_limit = 0.9; ``` * Specify the CPU usage ratio threshold for triggering the query queue. The following example sets the CPU usage permille (CPU usage \* 1000) threshold to `800`: ```sql SET GLOBAL query_queue_cpu_used_permille_limit = 800; ``` You can also decide how to deal with these queued queries by configuring the maximum queue length and the timeout for each pending query in the queue. * Specify the maximum query queue length. When this threshold is reached, incoming queries are rejected. The following example sets the query queue length to `100`: ```sql SET GLOBAL query_queue_max_queued_queries = 100; ``` * Specify the maximum timeout of a pending query in a queue. When this threshold is reached, the corresponding query is rejected. The following example sets the maximum timeout to `480` seconds: ```sql SET GLOBAL query_queue_pending_timeout_second = 480; ``` You can check whether a query is pending using [SHOW PROCESSLIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROCESSLIST.md). ```plain mysql> SHOW PROCESSLIST; +------+------+---------------------+-------+---------+---------------------+------+-------+-------------------+-----------+ | Id | User | Host | Db | Command | ConnectionStartTime | Time | State | Info | IsPending | +------+------+---------------------+-------+---------+---------------------+------+-------+-------------------+-----------+ | 2 | root | xxx.xx.xxx.xx:xxxxx | | Query | 2022-11-24 18:08:29 | 0 | OK | SHOW PROCESSLIST | false | +------+------+---------------------+-------+---------+---------------------+------+-------+-------------------+-----------+ ``` If `IsPending` is `true`, the corresponding query is pending in the query queue. #### Monitor big queries in real-time[​](#monitor-big-queries-in-real-time "Direct link to Monitor big queries in real-time") From v3.0 onwards, StarRocks supports viewing the queries that are currently processed in the cluster and the resources they occupy. This allows you to monitor the cluster in case any big queries bypass the precautions and cause unexpected system overload. ##### Monitor via MySQL client[​](#monitor-via-mysql-client "Direct link to Monitor via MySQL client") 1. You can view the queries that are currently processed (`current_queries`) using [SHOW PROC](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md). ```sql SHOW PROC '/current_queries'; ``` StarRocks returns the query ID (`QueryId`), connection ID (`ConnectionId`), and the resource consumption of each query, including the scanned data size (`ScanBytes`), processed row count (`ProcessRows`), CPU time (`CPUCostSeconds`), memory usage (`MemoryUsageBytes`), and execution time (`ExecTime`). ```plain mysql> SHOW PROC '/current_queries'; +--------------------------------------+--------------+------------+------+-----------+----------------+----------------+------------------+----------+ | QueryId | ConnectionId | Database | User | ScanBytes | ProcessRows | CPUCostSeconds | MemoryUsageBytes | ExecTime | +--------------------------------------+--------------+------------+------+-----------+----------------+----------------+------------------+----------+ | 7c56495f-ae8b-11ed-8ebf-00163e00accc | 4 | tpcds_100g | root | 37.88 MB | 1075769 Rows | 11.13 Seconds | 146.70 MB | 3804 | | 7d543160-ae8b-11ed-8ebf-00163e00accc | 6 | tpcds_100g | root | 13.02 GB | 487873176 Rows | 81.23 Seconds | 6.37 GB | 2090 | +--------------------------------------+--------------+------------+------+-----------+----------------+----------------+------------------+----------+ 2 rows in set (0.01 sec) ``` 2. You can further examine a query's resource consumption on each BE node by specifying the query ID. ```sql SHOW PROC '/current_queries//hosts'; ``` StarRocks returns the query's scanned data size (`ScanBytes`), scanned row count (`ScanRows`), CPU time (`CPUCostSeconds`), and memory usage (`MemUsageBytes`) on each BE node. ```plain mysql> show proc '/current_queries/7c56495f-ae8b-11ed-8ebf-00163e00accc/hosts'; +--------------------+-----------+-------------+----------------+---------------+ | Host | ScanBytes | ScanRows | CpuCostSeconds | MemUsageBytes | +--------------------+-----------+-------------+----------------+---------------+ | 172.26.34.185:8060 | 11.61 MB | 356252 Rows | 52.93 Seconds | 51.14 MB | | 172.26.34.186:8060 | 14.66 MB | 362646 Rows | 52.89 Seconds | 50.44 MB | | 172.26.34.187:8060 | 11.60 MB | 356871 Rows | 52.91 Seconds | 48.95 MB | +--------------------+-----------+-------------+----------------+---------------+ 3 rows in set (0.00 sec) ``` ##### Monitor via FE console[​](#monitor-via-fe-console "Direct link to Monitor via FE console") In addition to MySQL client, you can use the FE console for visualized, interactive monitoring. 1. Navigate to the FE console in your browser using the following URL: ```bash http://:/system?path=//current_queries ``` ![FE console 1](/assets/images/console_1-cbd35b88e205f18566c1a3ecdb888f4c.png) You can view the queries that are currently processed and their resource consumption on the **System Info** page. 2. Click the **QueryID** of the query. ![FE console 2](/assets/images/console_2-b772487ce01c6612ba48da1e08fd7609.png) You can view the detailed, node-specific resource consumption information on the page that appears. ##### Manually terminate big queries[​](#manually-terminate-big-queries "Direct link to Manually terminate big queries") If any big queries bypass the precautions you have set and threaten the system availability, you can terminate them manually using the corresponding connection ID in the [KILL](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/KILL.md) statement: ```sql KILL QUERY ; ``` #### Analyze Big Query Logs[​](#analyze-big-query-logs "Direct link to Analyze Big Query Logs") From v3.0 onwards, StarRocks supports Big Query Logs, which are stored in the file **fe/log/fe.big\_query.log**. Compared to the StarRocks audit logs, Big Query Logs print additional three fields: * `bigQueryLogCPUSecondThreshold` * `bigQueryLogScanBytesThreshold` * `bigQueryLogScanRowsThreshold` These three fields correspond to the resource consumption thresholds you defined to determine whether a query is a big query. To enable Big Query Logs, execute the following statement: ```sql SET GLOBAL enable_big_query_log = true; ``` After Big Query Logs are enabled, you can then define the rules to trigger Big Query Logs. * Specify the CPU time threshold for triggering Big Query Logs. The following example sets the CPU time threshold to `600` seconds: ```sql SET GLOBAL big_query_log_cpu_second_threshold = 600; ``` * Specify the scan data size threshold for triggering Big Query Logs. The following example sets the scan data size threshold to `10737418240` bytes (10 GB): ```sql SET GLOBAL big_query_log_scan_bytes_threshold = 10737418240; ``` * Specify the scan row count threshold for triggering Big Query Logs. The following example sets the scan row count threshold to `1500000000`: ```sql SET GLOBAL big_query_log_scan_rows_threshold = 1500000000; ``` #### Fine-tune precautions[​](#fine-tune-precautions "Direct link to Fine-tune precautions") From the statistics obtained from real-time monitoring and Big Query Logs, you can study the pattern of the omitted big queries (or regular queries that are mistakenly diagnosed as big queries) in your cluster, and then optimize the settings for resource groups and the query queue. If a notable proportion of big queries conform to a certain SQL pattern, and you want to permanently forbid this SQL pattern, you can add this pattern to SQL Blacklist. StarRocks rejects all queries that match any patterns specified in SQL Blacklist, and returns an error. For more information, see [Manage SQL Blacklist](https://docs.starrocks.io/docs/administration/management/resource_management/Blacklist.md). To enable SQL Blacklist, execute the following statement: ```sql ADMIN SET FRONTEND CONFIG ("enable_sql_blacklist" = "true"); ``` Then you can add the regular expression that represents the SQL pattern to SQL Blacklist using [ADD SQLBLACKLIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/sql_blacklist/ADD_SQLBLACKLIST.md). The following example adds `COUNT(DISTINCT)` to SQL Blacklist: ```sql ADD SQLBLACKLIST "SELECT COUNT(DISTINCT .+) FROM .+"; ``` --- ### Manage Alerts This topic introduces various alert items from different dimensions, including business continuity, cluster availability, and machine load, and provides corresponding resolutions. note In the following examples, all variables are prefixed with `$`. They should be replaced according to your business environment. For example, `$job_name` should be replaced with the corresponding Job Name in the Prometheus configuration, and `$fe_leader` should be replaced with the IP address of the Leader FE. #### Service Suspension Alerts[​](#service-suspension-alerts "Direct link to Service Suspension Alerts") ##### FE Service Suspension[​](#fe-service-suspension "Direct link to FE Service Suspension") **PromSQL** ```plain count(up{group="fe", job="$job_name"}) >= 3 ``` **Alert Description** An alert is triggered when the number of active FE nodes falls below a specified value. You can adjust this value based on the actual number of FE nodes. **Resolution** Try to restart the suspended FE node. ##### BE Service Suspension[​](#be-service-suspension "Direct link to BE Service Suspension") **PromSQL** ```plain node_info{type="be_node_num", job="$job_name",state="dead"} > 1 ``` **Alert Description** An alert is triggered when more than one BE node is suspended. **Resolution** Try to restart the suspended BE node. #### Machine Load Alerts[​](#machine-load-alerts "Direct link to Machine Load Alerts") ##### BE CPU Alert[​](#be-cpu-alert "Direct link to BE CPU Alert") **PromSQL** ```plain (1-(sum(rate(starrocks_be_cpu{mode="idle", job="$job_name",instance=~".*"}[5m])) by (job, instance)) / (sum(rate(starrocks_be_cpu{job="$job_name",host=~".*"}[5m])) by (job, instance))) * 100 > 90 ``` **Alert Description** An alert is triggered when BE CPU Utilization exceeds 90%. **Resolution** Check whether there are large queries or large-scale data loading and forward the details to the support team for further investigation. 1. Use the `top` command to check resource usage by processes. ```bash top -Hp $be_pid ``` 2. Use the `perf` command to collect and analyze performance data. ```bash # Execute the command for 1-2 minutes, and terminate it by pressing CTRL+C. sudo perf top -p $be_pid -g >/tmp/perf.txt ``` note In emergencies, to quickly restore service, you can try to restart the corresponding BE node after preserving the stack. An emergency here refers to a situation where the BE node's CPU utilization remains abnormally high, and no effective means are available to reduce CPU usage. ##### Memory Alert[​](#memory-alert "Direct link to Memory Alert") **PromSQL** ```plain (1-node_memory_MemAvailable_bytes{instance=~".*"}/node_memory_MemTotal_bytes{instance=~".*"})*100 > 90 ``` **Alert Description** An alert is triggered when memory usage exceeds 90%. **Resolution** Refer to the [Get Heap Profile](https://github.com/StarRocks/starrocks/pull/35322) for troubleshooting. note * In emergencies, you can try to restart the corresponding BE service to restore the service. An emergency here refers to a situation where the BE node's memory usage remains abnormally high, and no effective means are available to reduce memory usage. * If other mixed-deployed services are affecting the system, you may consider terminating those services in emergencies. ##### Disk Alerts[​](#disk-alerts "Direct link to Disk Alerts") ###### Disk Load Alert[​](#disk-load-alert "Direct link to Disk Load Alert") **PromSQL** ```sql rate(node_disk_io_time_seconds_total{instance=~".*"}[1m]) * 100 > 90 ``` **Alert Description** An alert is triggered when disk load exceeds 90%. **Resolution** If the cluster triggers a `node_disk_io_time_seconds_total` alert, first check if there are any business changes. If so, consider rolling back the changes to maintain the previous resource balance. If no changes are identified or rollback is not possible, consider whether normal business growth is driving the need for resource expansion. You can use the `iotop` tool to analyze disk I/O usage. `iotop` has a UI similar to `top` and includes information such as `pid`, `user`, and I/O. You can also use the following SQL query to identify the tablets consuming significant I/O and trace them back to specific tasks and tables. ```sql -- "all" indicates all services. 10 indicates the collection lasts 10 seconds. 3 indicates fetching the top 3 results. ADMIN EXECUTE ON $backend_id 'System.print(ExecEnv.io_profile_and_get_topn_stats("all", 10, 3))'; ``` ###### Root Path Capacity Alert[​](#root-path-capacity-alert "Direct link to Root Path Capacity Alert") **PromSQL** ```sql node_filesystem_free_bytes{mountpoint="/"} /1024/1024/1024 < 5 ``` **Alert Description** An alert is triggered when the available space in the root directory is less than 5GB. **Resolution** Common directories that may occupy significant space include **/var**, \*\*/\*\***opt**, and **/tmp**. Use the following command to check for large files and clear unnecessary files. ```bash du -sh / --max-depth=1 ``` ###### Data Disk Capacity Alert[​](#data-disk-capacity-alert "Direct link to Data Disk Capacity Alert") **PromSQL** ```bash (SUM(starrocks_be_disks_total_capacity{job="$job"}) by (host, path) - SUM(starrocks_be_disks_avail_capacity{job="$job"}) by (host, path)) / SUM(starrocks_be_disks_total_capacity{job="$job"}) by (host, path) * 100 > 90 ``` **Alert Description** An alert is triggered when disk capacity utilization exceeds 90%. **Resolution** 1. Check if there have been changes in the loaded data volume. Monitor the `load_bytes` metric in Grafana. If there has been a significant increase in data loading volume, you may need to scale the system resources. 2. Check for any DROP operations. If data loading volume has not changed much, run `SHOW BACKENDS`. If the reported disk usage does not match the actual usage, check the FE Audit Log for recent DROP DATABASE, TABLE, or PARTITION operations. Metadata for these operations remains in FE memory for one day, allowing you to restore data using the RECOVER statement within 24 hours to avoid misoperations. After recovery, the actual disk usage may exceed what is shown in `SHOW BACKENDS`. The retention period of deleted data in memory can be adjusted using the FE dynamic parameter `catalog_trash_expire_second` (default value: 86400). ```bash ADMIN SET FRONTEND CONFIG ("catalog_trash_expire_second"="86400"); ``` To persist this change, add the configuration item to the FE configuration file **fe.conf**. After that, deleted data will be moved to the **trash** directory on BE nodes (`$storage_root_path/trash`). By default, deleted data is kept in the **trash** directory for one day, which may also result in the actual disk usage exceeding what is shown in `SHOW BACKENDS`. The retention time of deleted data in the **trash** directory can be adjusted using the BE dynamic parameter `trash_file_expire_time_sec` (default value: 86400). ```bash curl http://$be_ip:$be_http_port/api/update_config?trash_file_expire_time_sec=86400 ``` ###### FE Metadata Disk Capacity Alert[​](#fe-metadata-disk-capacity-alert "Direct link to FE Metadata Disk Capacity Alert") **PromSQL** ```bash node_filesystem_free_bytes{mountpoint="${meta_path}"} /1024/1024/1024 < 10 ``` **Alert Description** An alert is triggered when the available disk space for FE metadata is less than 10GB. **Resolution** Use the following commands to check for directories occupying large amounts of space and clear unnecessary files. The metadata path is specified by the `meta_dir` configuration in **fe.conf**. ```bash du -sh /${meta_dir} --max-depth=1 ``` If the metadata directory occupies a lot of space, it is usually because the **bdb** directory is large, possibly due to CheckPoint failure. Refer to the [CheckPoint Failure Alert](#checkpoint-failure-alert) for troubleshooting. If this method does not solve the issue, contact the technical support team. #### Cluster Service Exception Alerts[​](#cluster-service-exception-alerts "Direct link to Cluster Service Exception Alerts") ##### Compaction Failure Alerts[​](#compaction-failure-alerts "Direct link to Compaction Failure Alerts") ###### Cumulative Compaction Failure Alert[​](#cumulative-compaction-failure-alert "Direct link to Cumulative Compaction Failure Alert") **PromSQL** ```bash increase(starrocks_be_engine_requests_total{job="$job_name" ,status="failed",type="cumulative_compaction"}[1m]) > 3 increase(starrocks_be_engine_requests_total{job="$job_name" ,status="failed",type="base_compaction"}[1m]) > 3 ``` **Alert Description** An alert is triggered when there are three failures in Cumulative Compaction or Base Compaction within the last minute. **Resolution** Search the log of the corresponding BE node for the following keywords to identify the involved tablet. ```bash grep -E 'compaction' be.INFO | grep failed ``` A log record like the following indicates a Compaction failure. ```plain W0924 17:52:56:537041 123639 comaction_task_cpp:193] compaction task:8482. tablet:8423674 failed. ``` You can check the context of the log to analyze the failure. Typically, the failure may have been caused by a DROP TABLE or PARTITION operation during the Compaction process. The system has an internal retry mechanism for Compaction, and you can also manually set the tablet's status to BAD and trigger a Clone task to repair it. note Before performing the following operation, ensure that the table has at least three complete replicas. ```bash ADMIN SET REPLICA STATUS PROPERTIES("tablet_id" = "$tablet_id", "backend_id" = "$backend_id", "status" = "bad"); ``` ###### High Compaction Pressure Alert[​](#high-compaction-pressure-alert "Direct link to High Compaction Pressure Alert") **PromSQL** ```bash starrocks_fe_max_tablet_compaction_score{job="$job_name",instance="$fe_leader"} > 100 ``` **Alert Description** An alert is triggered when the highest Compaction Score exceeds 100, indicating high Compaction pressure. **Resolution** This alert is typically caused by frequent loading, `INSERT INTO VALUES`, or `DELETE` operations (at a rate of 1 per second). It is recommended to set the interval between loading or DELETE tasks to more than 5 seconds and avoid submitting high concurrency DELETE tasks. ###### Exceeding Version Count Alert[​](#exceeding-version-count-alert "Direct link to Exceeding Version Count Alert") **PromSQL** ```bash starrocks_be_max_tablet_rowset_num{job="$job_name"} > 700 ``` **Alert Description** An alert is triggered when a tablet on a BE node has more than 700 data versions. **Resolution** Use the following command to check the tablet with excessive versions: ```sql SELECT BE_ID,TABLET_ID FROM information_schema.be_tablets WHERE NUM_ROWSET>700; ``` Example for Tablet with ID `2889156`: ```sql SHOW TABLET 2889156; ``` Execute the command returned in the `DetailCmd` field: ```sql SHOW PROC '/dbs/2601148/2889154/partitions/2889153/2889155/2889156'; ``` ![show proc replica](/assets/images/alert_show_proc_3-b5ec100163dd7cf0d4b942da44532d5e.png) Under normal circumstances, as shown, all three replicas should be in `NORMAL` status, and other metrics like `RowCount` and `DataSize` should remain consistent. If only one replica exceeds the version limit of 700, you can trigger a Clone task based on other replicas using the following command: ```sql ADMIN SET REPLICA STATUS PROPERTIES("tablet_id" = "$tablet_id", "backend_id" = "$backend_id", "status" = "bad"); ``` If two or more replicas exceed the version limit, you can temporarily increase the version count limit: ```bash # Replace be_ip with the IP of the BE node which stores the tablet that exceeds the version limit. # The default be_http_port is 8040. # The default value of tablet_max_versions is 1000. curl -XPOST http://$be_ip:$be_http_port/api/update_config?tablet_max_versions=2000 ``` ##### CheckPoint Failure Alert[​](#checkpoint-failure-alert "Direct link to CheckPoint Failure Alert") **PromSQL** ```bash starrocks_fe_meta_log_count{job="$job_name",instance="$fe_master"} > 100000 ``` **Alert Description** An alert is triggered when the FE node's BDB log count exceeds 100,000. By default, the system performs a CheckPoint when the BDB log count exceeds 50,000, and then resets the count to 0. **Resolution** This alert indicates that a CheckPoint was not performed. You need to investigate the FE logs to analyze the CheckPoint process and resolve the issue: In the **fe.log** of the Leader FE node, search for records like `begin to generate new image: image.xxxx`. If found, it means the system has started generating a new image. Continue checking the logs for records like `checkpoint finished save image.xxxx` to confirm successful image creation. If you find `Exception when generate new image file`, the image generation failed. You should carefully handle the metadata based on the specific error. It is recommended to contact the support team for further analysis. ##### Excessive FE Thread Count Alert[​](#excessive-fe-thread-count-alert "Direct link to Excessive FE Thread Count Alert") **PromSQL** ```bash starrocks_fe_thread_pool{job="$job_name", type!="completed_task_count"} > 3000 ``` **Alert Description** An alert is triggered when the number of threads on the FE exceeds 3000. **Resolution** The default thread count limit for FE and BE nodes is 4096. A large number of UNION ALL queries typically lead to an excessive thread count. It is recommended to reduce the concurrency of UNION ALL queries and adjust the system variable `pipeline_dop`. If it is not possible to adjust SQL query granularity, you can globally adjust `pipeline_dop`: ```sql SET GLOBAL pipeline_dop=8; ``` note In emergencies, to restore services quickly, you can increase the FE dynamic parameter `thrift_server_max_worker_threads` (default value: 4096). ```sql ADMIN SET FRONTEND CONFIG ("thrift_server_max_worker_threads"="8192"); ``` ##### High FE JVM Usage Alert[​](#high-fe-jvm-usage-alert "Direct link to High FE JVM Usage Alert") **PromSQL** ```sql sum(jvm_heap_size_bytes{job="$job_name", type="used"}) * 100 / sum(jvm_heap_size_bytes{job="$job_name", type="max"}) > 90 ``` **Alert Description** An alert is triggered when the JVM usage on an FE node exceeds 90%. **Resolution** This alert indicates that JVM usage is too high. You can use the `jmap` command to analyze the situation. Since detailed monitoring information for this metric is still under development, direct insights are limited. Perform the following actions and send the results to the support team for analysis: ```bash # Note that specifying `live` in the command may cause FE to restart. jmap -histo[:live] $fe_pid > jmap.dump ``` note In emergencies, to quickly restore services, you can restart the corresponding FE node or increase the JVM (Xmx) size and then restart the FE service. #### Service Availability Alerts[​](#service-availability-alerts "Direct link to Service Availability Alerts") ##### Loading Exception Alerts[​](#loading-exception-alerts "Direct link to Loading Exception Alerts") ###### Loading Failure Alert[​](#loading-failure-alert "Direct link to Loading Failure Alert") **PromSQL** ```sql rate(starrocks_fe_txn_failed{job="$job_name",instance="$fe_master"}[5m]) * 100 > 5 ``` **Alert Description** An alert is triggered when the number of failed loading transactions exceeds 5% of the total. **Resolution** Check the logs of the Leader FE node to find information about the loading errors. Search for the keyword `status: ABORTED` to identify failed loading tasks. ```plain 2024-04-09 18:34:02.363+08:00 INFO (thrift-server-pool-8845163|12111749) [DatabaseTransactionMgr.abortTransaction():1279] transaction:[TransactionState. txn_id: 7398864, label: 967009-2f20a55e-368d-48cf-833a-762cf1fe07c5, db id: 10139, table id list: 155532, callback id: 967009, coordinator: FE: 192.168.2.1, transaction status: ABORTED, error replicas num: 0, replica ids: , prepare time: 1712658795053, commit time: -1, finish time: 1712658842360, total cost: 47307ms, reason: [E1008]Reached timeout=30000ms @192.168.1.1:8060 attachment: RLTaskTxnCommitAttachment [filteredRows=0, loadedRows=0, unselectedRows=0, receivedBytes=1033110486, taskExecutionTimeMs=0, taskId=TUniqueId(hi:3395895943098091727, lo:-8990743770681178171), jobId=967009, progress=KafkaProgress [partitionIdToOffset=2_1211970882|7_1211893755]]] successfully rollback ``` ###### Routine Load Consumption Delay Alert[​](#routine-load-consumption-delay-alert "Direct link to Routine Load Consumption Delay Alert") **PromSQL** ```sql (sum by (job_name)(starrocks_fe_routine_load_max_lag_of_partition{job="$job_name",instance="$fe_mater"})) > 300000 starrocks_fe_routine_load_jobs{job="$job_name",host="$fe_mater",state="NEED_SCHEDULE"} > 3 starrocks_fe_routine_load_jobs{job="$job_name",host="$fe_mater",state="PAUSED"} > 0 starrocks_fe_routine_load_jobs{job="$job_name",host="$fe_mater",state="UNSTABLE"} > 0 ``` **Alert Description** * An alert is triggered when over 300,000 entries are delayed in consumption. * An alert is triggered when the number of pending Routine Load tasks exceeds 3. * An alert is triggered when there are tasks in the `PAUSED` state. * An alert is triggered when there are tasks in the `UNSTABLE` state. **Resolution** 1. First, check if the Routine Load task status is `RUNNING`. ```sql SHOW ROUTINE LOAD FROM $db; ``` Pay attention to the `State` field in the returned data. 2. If any Routine Load task is in the `PAUSED` state, examine the `ReasonOfStateChanged`, `ErrorLogUrls`, and `TrackingSQL` fields. Typically, executing the SQL query in `TrackingSQL` can reveal the specific error. Example: ![Tracking SQL](/assets/images/alert_routine_load_tracking-ea00801a160616a69c2e799c541fd285.png) 3. If the Routine Load task status is `RUNNING`, you can try to increase the task’s concurrency. The concurrency of individual Routine Load jobs is determined by the minimum value of the following four parameters: * `kafka_partition_num`: Number of partitions in the Kafka Topic. * `desired_concurrent_number`: The set concurrency for the task. * `alive_be_num`: Number of live BE nodes. * `max_routine_load_task_concurrent_num`: FE configuration parameter, with a default value of 5. In most cases, you may need to adjust the task’s concurrency or the number of Kafka Topic partitions (contact Kafka support if necessary). The following example shows how to set concurrency for the task. ```sql ALTER ROUTINE LOAD FOR ${routine_load_jobname} PROPERTIES ( "desired_concurrent_number" = "5" ); ``` ###### Loading Transaction Limit Alert for a Single Database[​](#loading-transaction-limit-alert-for-a-single-database "Direct link to Loading Transaction Limit Alert for a Single Database") **PromSQL** ```sql sum(starrocks_fe_txn_running{job="$job_name"}) by(db) > 900 ``` **Alert Description** An alert is triggered when the number of loading transactions for a single database exceeds 900 (100 in versions prior to v3.1). **Resolution** This alert is typically triggered by a large number of newly added loading tasks. You can temporarily increase the limit on loading transactions for a single database. ```sql ADMIN SET FRONTEND CONFIG ("max_running_txn_num_per_db" = "2000"); ``` ##### Query Exception Alerts[​](#query-exception-alerts "Direct link to Query Exception Alerts") ###### Query Latency Alert[​](#query-latency-alert "Direct link to Query Latency Alert") **PromSQL** ```sql starrocks_fe_query_latency_ms{job="$job_name", quantile="0.95"} > 5000 ``` **Alert Description** An alert is triggered when the P95 query latency exceeds 5 seconds. **Resolution** 1. Investigate whether there are any big queries. Check whether large queries have consumed significant machine resources during the exception, leading to other queries timing out or failing. * Execute `show proc '/current_queries';` to view the `QueryId` of big queries. If you need to quickly restore service, you can use the `KILL` command to terminate the long-running queries. ```sql mysql> SHOW PROC '/current_queries'; +--------------------------------------+--------------+------------+------+-----------+----------------+----------------+------------------+----------+ | QueryId | ConnectionId | Database | User | ScanBytes | ProcessRows | CPUCostSeconds | MemoryUsageBytes | ExecTime | +--------------------------------------+--------------+------------+------+-----------+----------------+----------------+------------------+----------+ | 7c56495f-ae8b-11ed-8ebf-00163e00accc | 4 | tpcds_100g | root | 37.88 MB | 1075769 Rows | 11.13 Seconds | 146.70 MB | 3804 | | 7d543160-ae8b-11ed-8ebf-00163e00accc | 6 | tpcds_100g | root | 13.02 GB | 487873176 Rows | 81.23 Seconds | 6.37 GB | 2090 | +--------------------------------------+--------------+------------+------+-----------+----------------+----------------+------------------+----------+ 2 rows in set (0.01 sec) ``` * You can also restart the BE nodes with high CPU utilization to resolve the issue. 2. Check if the machine resources are sufficient. Verify whether CPU, memory, Disk I/O, and network traffic during the exception are normal. If anomalies are detected, investigate the root cause by examining peak traffic variations and cluster resource usage. If the issue persists, consider restarting the affected node. note In emergencies, you can resolve the issue by: * Reducing business traffic and restarting the affected BE node if a sudden traffic spike caused resource overuse and query failure. * Expanding node capacity if high resource usage is due to normal operations. ###### Query Failure Alert[​](#query-failure-alert "Direct link to Query Failure Alert") **PromSQL** ```plain sum by (job,instance)(starrocks_fe_query_err_rate{job="$job_name"}) * 100 > 10 # This PromSQL is supported from v3.1.15, v3.2.11, and v3.3.3 onwards. increase(starrocks_fe_query_internal_err{job="$job_name"})[1m] >10 ``` **Alert Description** An alert is triggered when the query failure rate exceeds 0.1/second or 10 failed queries occur within one minute. **Resolution** When this alert is triggered, check the logs to identify the queries that failed. ```bash grep 'State=ERR' fe.audit.log ``` If you have the AuditLoader plugin installed, you can locate the corresponding queries using the following query. ```bash SELECT stmt FROM starrocks_audit_db__.starrocks_audit_tbl__ WHERE state='ERR'; ``` Note that queries that fail due to syntax errors or timeouts are also recorded in `starrocks_fe_query_err_rate`. For query failures caused by kernel issues, search the `fe.log` for the error and obtain the complete stack trace and [Query Dump](https://docs.starrocks.io/docs/faq/Dump_query.md), and contact the support team for troubleshooting. ###### Query Overload Alert[​](#query-overload-alert "Direct link to Query Overload Alert") **PromSQL** ```bash abs((sum by (exported_job)(rate(starrocks_fe_query_total{process="FE",job="$job_name"}[3m]))-sum by (exported_job)(rate(starrocks_fe_query_total{process="FE",job="$job_name"}[3m] offset 1m)))/sum by (exported_job)(rate(starrocks_fe_query_total{process="FE",job="$job_name"}[3m]))) * 100 > 100 abs((sum(starrocks_fe_connection_total{job="$job_name"})-sum(starrocks_fe_connection_total{job="$job_name"} offset 3m))/sum(starrocks_fe_connection_total{job="$job_name"})) * 100 > 100 ``` **Alert Description** An alert is triggered when the QPS or the number of connections increases by 100% within the last minute. **Resolution** Check whether the high-frequency queries in the `fe.audit.log` are expected. If there are legitimate changes in business behavior (for example, new services going live or increased data volumes), monitor machine load and scale BE nodes as needed. ###### User Connection Limit Exceeded Alert[​](#user-connection-limit-exceeded-alert "Direct link to User Connection Limit Exceeded Alert") **PromSQL** ```bash sum(starrocks_fe_connection_total{job="$job_name"}) by(user) > 90 ``` **Alert Description** An alert is triggered when the number of user connections exceeds 90. (User connection limits are supported from versions v3.1.16, v3.2.12, and v3.3.4 onward.) **Resolution** Use the SQL command `SHOW PROCESSLIST` to check if the number of current connections is as expected. You can terminate unexpected connections using the `KILL` command. Additionally, ensure that frontend services are not holding connections open for too long, and consider adjusting the system variable `wait_timeout` (Unit: Seconds) to accelerate the system's automatic termination of idle connections. ```bash SET wait_timeout = 3600; ``` note In emergencies, you can increase the user connection limit temporarily to restore service: * For v3.1.16, v3.2.12, and v3.3.4 or later: ```bash ALTER USER 'jack' SET PROPERTIES ("max_user_connections" = "1000"); ``` * For v2.5 and earlier: ```bash SET PROPERTY FOR 'jack' 'max_user_connections' = '1000'; ``` ##### Schema Change Exception Alert[​](#schema-change-exception-alert "Direct link to Schema Change Exception Alert") **PromSQL** ```bash increase(starrocks_be_engine_requests_total{job="$job_name",type="schema_change", status="failed"}[1m]) > 1 ``` **Alert Description** An alert is triggered when more than one Schema Change task fails in the last minute. **Resolution** Run the following statement to check if the `Msg` field contains any error messages: ```bash SHOW ALTER COLUMN FROM $db; ``` If no message is found, search for the JobId from the previous step in the Leader FE logs to retrieve the context. * Schema Change Out of Memory If the Schema Change fails due to insufficient memory, search the **be.WARNING** logs for `failed to process the version`, `failed to process the schema change from tablet`, or `Memory of schema change task exceeded limit` to identify log records shown in the following: ```bash fail to execute schema change: Memory of schema change task exceed limit. DirectSchemaChange Used: 2149621304, Limit: 2147483648. You can change the limit by modify BE config [memory_limitation_per_thread_for_schema_change] ``` The memory limit error is typically caused by exceeding the 2GB memory limit for a single Schema Change, controlled by the BE dynamic parameter `memory_limitation_per_thread_for_schema_change`. You can modify this parameter to resolve the issue. ```bash curl -XPOST http://be_host:http_port/api/update_config?memory_limitation_per_thread_for_schema_change=8 ``` * Schema Change Timeout Except for adding columns, which is a lightweight implementation, most Schema Changes involve creating a large number of new tablets, rewriting the original data, and implementing the operation via SWAP. ```plain Create replicas failed. Error: Error replicas:21539953=99583471, 21539953=99583467, 21539953=99599851 ``` You can address this by: * Increasing the timeout for creating tablets (Default: 10 seconds). ```bash ADMIN SET FRONTEND CONFIG ("tablet_create_timeout_second"="60"); ``` * Increasing the number of threads for creating tablets (default: 3). ```bash curl -XPOST http://be_host:http_port/api/update_config?alter_tablet_worker_count=6 ``` * Non-Normal Tablet State 1. If a tablet is in a non-normal state, search the **be.WARNING** logs for `tablet is not normal` and execute `SHOW PROC '/statistic'` to check the cluster-level `UnhealthyTabletNum`. ![show statistic](/assets/images/alert_show_statistic-0b98665c37c4ad9d02dd67870099f871.png) 2. Execute `SHOW PROC '/statistic/$DbId'` to check the unhealthy tablet number in the specified database. ![show statistic db](/assets/images/alert_show_statistic_db-d7769f340e126de5cd9161e4a3920cbc.png) 3. Execute `SHOW TABLET $tablet_id` to view the table information of the corresponding tablet. ![show tablet](/assets/images/alert_show_tablet-24f35ed634afe110d11f2edce74be565.png) 4. Execute the command returned in the `DetailCmd` field to identify the cause of the unhealthy tablets. ![show proc](/assets/images/alert_show_proc-a71de5e2f30f015ef8e3483c0ff69abe.png) Typically, unhealthy as well as inconsistent replicas are usually caused by high-frequency loading, where the progress of writes to different replicas is not synchronized. You can check if the table has a large number of real-time writes and reduce the number of abnormal replicas by reducing the frequency of loading or temporarily suspending the service and retrying the task thereafter. note In emergencies, to restore the service, you can set the non-Normal replicas as Bad to trigger a Clone task. ```bash ADMIN SET REPLICA STATUS PROPERTIES("tablet_id" = "$tablet_id", "backend_id" = "$backend_id", "status" = "bad"); ``` Before performing this operation, ensure the table has at least three complete replicas with only one non-normal replica. ##### Materialized View Refresh Exception Alert[​](#materialized-view-refresh-exception-alert "Direct link to Materialized View Refresh Exception Alert") **PromSQL** ```bash increase(starrocks_fe_mv_refresh_total_failed_jobs[5m]) > 0 ``` **Alert Description** An alert is triggered when more than one materialized view refresh fails in the last five minutes. **Resolution** 1. Check the materialized views that failed to refresh. ```sql SELECT TABLE_NAME,IS_ACTIVE,INACTIVE_REASON,TASK_NAME FROM information_schema.materialized_views WHERE LAST_REFRESH_STATE !=" SUCCESS"; ``` 2. Try manually refreshing the materialized view. ```sql REFRESH MATERIALIZED VIEW $mv_name; ``` 3. If the materialized view is in the `INACTIVE` state, try to manually activate it. ```sql ALTER MATERIALIZED VIEW $mv_name ACTIVE; ``` 4. Investigate the cause of the refresh failure. ```sql SELECT * FROM information_schema.task_runs WHERE task_name ='mv-112517' \G ``` --- ### Metrics a through c note Metrics for materialized views and shared-data clusters are detailed in the corresponding sections: * [Metrics for asynchronous materialized view metrics](https://docs.starrocks.io/docs/administration/management/monitoring/metrics-materialized_view.md) * [Metrics for Shared-data Dashboard metrics, and Starlet Dashboard metrics](https://docs.starrocks.io/docs/administration/management/monitoring/metrics-shared-data.md) For more information on how to build a monitoring service for your StarRocks cluster, see [Monitor and Alert](https://docs.starrocks.io/docs/administration/management/monitoring/Monitor_and_Alert.md). #### `active_scan_context_count`[​](#active_scan_context_count "Direct link to active_scan_context_count") * Unit: Count * Description: Total number of scan tasks created by Flink/Spark SQL. #### `async_delta_writer_queue_count`[​](#async_delta_writer_queue_count "Direct link to async_delta_writer_queue_count") * Unit: Count * Description: Queued task count in the tablet delta writer thread pool. #### `base_compaction_task_byte_per_second`[​](#base_compaction_task_byte_per_second "Direct link to base_compaction_task_byte_per_second") * Unit: Bytes/s * Description: Estimated rate of base compactions. #### `base_compaction_task_cost_time_ms`[​](#base_compaction_task_cost_time_ms "Direct link to base_compaction_task_cost_time_ms") * Unit: ms * Description: Total time spent on base compactions. #### `be_base_compaction_bytes_per_second`[​](#be_base_compaction_bytes_per_second "Direct link to be_base_compaction_bytes_per_second") * Unit: Bytes/s * Type: Average * Description: Base compaction speed of BE. #### `be_base_compaction_failed`[​](#be_base_compaction_failed "Direct link to be_base_compaction_failed") * Unit: Count/s * Type: Average * Description: Base compaction failure of BE. #### `be_base_compaction_requests`[​](#be_base_compaction_requests "Direct link to be_base_compaction_requests") * Unit: Count/s * Type: Average * Description: Base compaction request of BE. #### `be_base_compaction_rowsets_per_second`[​](#be_base_compaction_rowsets_per_second "Direct link to be_base_compaction_rowsets_per_second") * Unit: Count * Type: Average * Description: Base compaction speed of BE rowsets. #### `be_broker_count`[​](#be_broker_count "Direct link to be_broker_count") * Unit: Count * Type: Average * Description: Number of brokers. #### `be_brpc_endpoint_count`[​](#be_brpc_endpoint_count "Direct link to be_brpc_endpoint_count") * Unit: Count * Type: Average * Description: Number of StubCache in bRPC. #### `be_bytes_read_per_second`[​](#be_bytes_read_per_second "Direct link to be_bytes_read_per_second") * Unit: Bytes/s * Type: Average * Description: Read speed of BE. #### `be_bytes_written_per_second`[​](#be_bytes_written_per_second "Direct link to be_bytes_written_per_second") * Unit: Bytes/s * Type: Average * Description: Write speed of BE. #### `be_clone_failed`[​](#be_clone_failed "Direct link to be_clone_failed") * Unit: Count/s * Type: Average * Description: BE clone failure. #### `be_clone_total_requests`[​](#be_clone_total_requests "Direct link to be_clone_total_requests") * Unit: Count/s * Type: Average * Description: Clone request of BE. #### `be_create_rollup_failed`[​](#be_create_rollup_failed "Direct link to be_create_rollup_failed") * Unit: Count/s * Type: Average * Description: Materialized view creation failure of BE. #### `be_create_rollup_requests`[​](#be_create_rollup_requests "Direct link to be_create_rollup_requests") * Unit: Count/s * Type: Average * Description: Materialized view creation request of BE. #### `be_create_tablet_failed`[​](#be_create_tablet_failed "Direct link to be_create_tablet_failed") * Unit: Count/s * Type: Average * Description: Tablet creation failure of BE. #### `be_create_tablet_requests`[​](#be_create_tablet_requests "Direct link to be_create_tablet_requests") * Unit: Count/s * Type: Average * Description: Tablet creation request of BE. #### `be_cumulative_compaction_bytes_per_second`[​](#be_cumulative_compaction_bytes_per_second "Direct link to be_cumulative_compaction_bytes_per_second") * Unit: Bytes/s * Type: Average * Description: Cumulative compaction speed of BE. #### `be_cumulative_compaction_failed`[​](#be_cumulative_compaction_failed "Direct link to be_cumulative_compaction_failed") * Unit: Count/s * Type: Average * Description: Cumulative compaction failure of BE. #### `be_cumulative_compaction_requests`[​](#be_cumulative_compaction_requests "Direct link to be_cumulative_compaction_requests") * Unit: Count/s * Type: Average * Description: Cumulative compaction request of BE. #### `be_cumulative_compaction_rowsets_per_second`[​](#be_cumulative_compaction_rowsets_per_second "Direct link to be_cumulative_compaction_rowsets_per_second") * Unit: Count * Type: Average * Description: Cumulative compaction speed of BE rowsets. #### `be_delete_failed`[​](#be_delete_failed "Direct link to be_delete_failed") * Unit: Count/s * Type: Average * Description: Delete failure of BE. #### `be_delete_requests`[​](#be_delete_requests "Direct link to be_delete_requests") * Unit: Count/s * Type: Average * Description: Delete request of BE. #### `be_finish_task_failed`[​](#be_finish_task_failed "Direct link to be_finish_task_failed") * Unit: Count/s * Type: Average * Description: Task failure of BE. #### `be_finish_task_requests`[​](#be_finish_task_requests "Direct link to be_finish_task_requests") * Unit: Count/s * Type: Average * Description: Task finish request of BE. #### `be_fragment_endpoint_count`[​](#be_fragment_endpoint_count "Direct link to be_fragment_endpoint_count") * Unit: Count * Type: Average * Description: Number of BE DataStream. #### `be_fragment_request_latency_avg`[​](#be_fragment_request_latency_avg "Direct link to be_fragment_request_latency_avg") * Unit: ms * Type: Average * Description: Latency of fragment requests. #### `be_fragment_requests_per_second`[​](#be_fragment_requests_per_second "Direct link to be_fragment_requests_per_second") * Unit: Count/s * Type: Average * Description: Number of fragment requests. #### `be_http_request_latency_avg`[​](#be_http_request_latency_avg "Direct link to be_http_request_latency_avg") * Unit: ms * Type: Average * Description: Latency of HTTP requests. #### `be_http_request_send_bytes_per_second`[​](#be_http_request_send_bytes_per_second "Direct link to be_http_request_send_bytes_per_second") * Unit: Bytes/s * Type: Average * Description: Number of bytes sent for HTTP requests. #### `be_http_requests_per_second`[​](#be_http_requests_per_second "Direct link to be_http_requests_per_second") * Unit: Count/s * Type: Average * Description: Number of HTTP requests. #### `be_publish_failed`[​](#be_publish_failed "Direct link to be_publish_failed") * Unit: Count/s * Type: Average * Description: Version release failure of BE. #### `be_publish_requests`[​](#be_publish_requests "Direct link to be_publish_requests") * Unit: Count/s * Type: Average * Description: Version publish request of BE. #### `be_report_disk_failed`[​](#be_report_disk_failed "Direct link to be_report_disk_failed") * Unit: Count/s * Type: Average * Description: Disk report failure of BE. #### `be_report_disk_requests`[​](#be_report_disk_requests "Direct link to be_report_disk_requests") * Unit: Count/s * Type: Average * Description: Disk report request of BE. #### `be_report_tables_failed`[​](#be_report_tables_failed "Direct link to be_report_tables_failed") * Unit: Count/s * Type: Average * Description: Table report failure of BE. #### `be_report_tablet_failed`[​](#be_report_tablet_failed "Direct link to be_report_tablet_failed") * Unit: Count/s * Type: Average * Description: Tablet report failure of BE. #### `be_report_tablet_requests`[​](#be_report_tablet_requests "Direct link to be_report_tablet_requests") * Unit: Count/s * Type: Average * Description: Tablet report request of BE. #### `be_report_tablets_requests`[​](#be_report_tablets_requests "Direct link to be_report_tablets_requests") * Unit: Count/s * Type: Average * Description: Tablet report request of BE. #### `be_report_task_failed`[​](#be_report_task_failed "Direct link to be_report_task_failed") * Unit: Count/s * Type: Average * Description: Task report failure of BE. #### `be_report_task_requests`[​](#be_report_task_requests "Direct link to be_report_task_requests") * Unit: Count/s * Type: Average * Description: Task report request of BE. #### `be_schema_change_failed`[​](#be_schema_change_failed "Direct link to be_schema_change_failed") * Unit: Count/s * Type: Average * Description: Schema change failure of BE. #### `be_schema_change_requests`[​](#be_schema_change_requests "Direct link to be_schema_change_requests") * Unit: Count/s * Type: Average * Description: Schema change report request of BE. #### `be_storage_migrate_requests`[​](#be_storage_migrate_requests "Direct link to be_storage_migrate_requests") * Unit: Count/s * Type: Average * Description: Migration request of BE. #### `binary_column_pool_bytes`[​](#binary_column_pool_bytes "Direct link to binary_column_pool_bytes") * Unit: Bytes * Description: Memory used by the BINARY column pool. #### `bitmap_index_mem_bytes`[​](#bitmap_index_mem_bytes "Direct link to bitmap_index_mem_bytes") * Unit: Bytes * Description: Memory used by bitmap indexes. #### `block_cache_hit_bytes`[​](#block_cache_hit_bytes "Direct link to block_cache_hit_bytes") * Unit: Bytes * Type: Counter * Description: Cumulative bytes of block cache hits. For now, only the cache hit bytes for external table is being counted. #### `block_cache_miss_bytes`[​](#block_cache_miss_bytes "Direct link to block_cache_miss_bytes") * Unit: Bytes * Type: Counter * Description: Cumulative bytes of block cache misses. For now, only the cache miss bytes for external table is being counted. #### `block_cache_hit_count`[​](#block_cache_hit_count "Direct link to block_cache_hit_count") * Unit: Count * Type: Counter * Description: Cumulative count of block cache hits. #### `block_cache_miss_count`[​](#block_cache_miss_count "Direct link to block_cache_miss_count") * Unit: Count * Type: Counter * Description: Cumulative count of block cache misses. #### `blocks_created_total (Deprecated)`[​](#blocks_created_total-deprecated "Direct link to blocks_created_total-deprecated") #### `blocks_deleted_total (Deprecated)`[​](#blocks_deleted_total-deprecated "Direct link to blocks_deleted_total-deprecated") #### `blocks_open_reading (Deprecated)`[​](#blocks_open_reading-deprecated "Direct link to blocks_open_reading-deprecated") #### `blocks_open_writing (Deprecated)`[​](#blocks_open_writing-deprecated "Direct link to blocks_open_writing-deprecated") #### `bloom_filter_index_mem_bytes`[​](#bloom_filter_index_mem_bytes "Direct link to bloom_filter_index_mem_bytes") * Unit: Bytes * Description: Memory used by Bloomfilter indexes. #### `broker_count`[​](#broker_count "Direct link to broker_count") * Unit: Count * Description: Total number of filesystem brokers (by host address) created. #### `brpc_endpoint_stub_count`[​](#brpc_endpoint_stub_count "Direct link to brpc_endpoint_stub_count") * Unit: Count * Description: Total number of bRPC stubs (by address). #### `builtin_inverted_index_mem_bytes`[​](#builtin_inverted_index_mem_bytes "Direct link to builtin_inverted_index_mem_bytes") * Unit: Bytes * Description: Memory used by builtin inverted indexes. #### `bytes_read_total (Deprecated)`[​](#bytes_read_total-deprecated "Direct link to bytes_read_total-deprecated") #### `bytes_written_total`[​](#bytes_written_total "Direct link to bytes_written_total") * Unit: Bytes * Description: Total bytes written (sectors write \* 512). #### `central_column_pool_bytes (Deprecated)`[​](#central_column_pool_bytes-deprecated "Direct link to central_column_pool_bytes-deprecated") #### `chunk_allocator_mem_bytes`[​](#chunk_allocator_mem_bytes "Direct link to chunk_allocator_mem_bytes") * Unit: Bytes * Description: Memory used by the chunk allocator. #### `chunk_pool_local_core_alloc_count`[​](#chunk_pool_local_core_alloc_count "Direct link to chunk_pool_local_core_alloc_count") * Unit: Count * Description: Memory chunk allocation/cache metrics. #### `chunk_pool_other_core_alloc_count`[​](#chunk_pool_other_core_alloc_count "Direct link to chunk_pool_other_core_alloc_count") * Unit: Count * Description: Memory chunk allocation/cache metrics. #### `chunk_pool_system_alloc_cost_ns`[​](#chunk_pool_system_alloc_cost_ns "Direct link to chunk_pool_system_alloc_cost_ns") * Unit: ns * Description: Memory chunk allocation/cache metrics. #### `chunk_pool_system_alloc_count`[​](#chunk_pool_system_alloc_count "Direct link to chunk_pool_system_alloc_count") * Unit: Count * Description: Memory chunk allocation/cache metrics. #### `chunk_pool_system_free_cost_ns`[​](#chunk_pool_system_free_cost_ns "Direct link to chunk_pool_system_free_cost_ns") * Unit: ns * Description: Memory chunk allocation/cache metrics. #### `chunk_pool_system_free_count`[​](#chunk_pool_system_free_count "Direct link to chunk_pool_system_free_count") * Unit: Count * Description: Memory chunk allocation/cache metrics. #### `clone_mem_bytes`[​](#clone_mem_bytes "Direct link to clone_mem_bytes") * Unit: Bytes * Description: Memory used for replica clone. #### `cluster_snapshot_consecutive_failures`[​](#cluster_snapshot_consecutive_failures "Direct link to cluster_snapshot_consecutive_failures") * Unit: Count * Description: Number of consecutive failed automated cluster snapshot jobs since the last successful one. A persistently increasing value indicates the snapshot storage volume is misconfigured or unreachable. #### `cluster_snapshot_last_finished_time`[​](#cluster_snapshot_last_finished_time "Direct link to cluster_snapshot_last_finished_time") * Unit: Milliseconds * Description: Epoch time (in milliseconds) of the last finished automated cluster snapshot, or 0 if none has finished. #### `column_metadata_mem_bytes`[​](#column_metadata_mem_bytes "Direct link to column_metadata_mem_bytes") * Unit: Bytes * Description: Memory used by column metadata. #### `column_partial_update_apply_duration_us`[​](#column_partial_update_apply_duration_us "Direct link to column_partial_update_apply_duration_us") * Unit: us * Description: Total time spent on partial updates for columns' APPLY tasks (Column mode). #### `column_partial_update_apply_total`[​](#column_partial_update_apply_total "Direct link to column_partial_update_apply_total") * Unit: Count * Description: Total number of APPLY for partial updates by column (Column mode) #### `column_pool_mem_bytes`[​](#column_pool_mem_bytes "Direct link to column_pool_mem_bytes") * Unit: Bytes * Description: Memory used by the column pools. #### `column_zonemap_index_mem_bytes`[​](#column_zonemap_index_mem_bytes "Direct link to column_zonemap_index_mem_bytes") * Unit: Bytes * Description: Memory used by column zonemaps. #### `compaction_bytes_total`[​](#compaction_bytes_total "Direct link to compaction_bytes_total") * Unit: Bytes * Description: Total merged bytes from base compactions and cumulative compactions. #### `compaction_deltas_total`[​](#compaction_deltas_total "Direct link to compaction_deltas_total") * Unit: Count * Description: Total number of merged rowsets from base compactions and cumulative compactions. #### `compaction_mem_bytes`[​](#compaction_mem_bytes "Direct link to compaction_mem_bytes") * Unit: Bytes * Description: Memory used by compactions. #### `consistency_mem_bytes`[​](#consistency_mem_bytes "Direct link to consistency_mem_bytes") * Unit: Bytes * Description: Memory used by replica consistency checks. #### `cpu`[​](#cpu "Direct link to cpu") * Unit: - * Description: CPU usage information returned by `/proc/stat`. #### `cpu_guest`[​](#cpu_guest "Direct link to cpu_guest") * Unit: - * Type: Average * Description: cpu\_guest usage rate. #### `cpu_idle`[​](#cpu_idle "Direct link to cpu_idle") * Unit: - * Type: Average * Description: cpu\_idle usage rate. #### `cpu_iowait`[​](#cpu_iowait "Direct link to cpu_iowait") * Unit: - * Type: Average * Description: cpu\_iowait usage rate. #### `cpu_irq`[​](#cpu_irq "Direct link to cpu_irq") * Unit: - * Type: Average * Description: cpu\_irq usage rate. #### `cpu_nice`[​](#cpu_nice "Direct link to cpu_nice") * Unit: - * Type: Average * Description: cpu\_nice usage rate. #### `cpu_softirq`[​](#cpu_softirq "Direct link to cpu_softirq") * Unit: - * Type: Average * Description: cpu\_softirq usage rate. #### `cpu_steal`[​](#cpu_steal "Direct link to cpu_steal") * Unit: - * Type: Average * Description: cpu\_steal usage rate. #### `cpu_system`[​](#cpu_system "Direct link to cpu_system") * Unit: - * Type: Average * Description: cpu\_system usage rate. #### `cpu_user`[​](#cpu_user "Direct link to cpu_user") * Unit: - * Type: Average * Description: cpu\_user usage rate. #### `cpu_util`[​](#cpu_util "Direct link to cpu_util") * Unit: - * Type: Average * Description: CPU usage rate. #### `cumulative_compaction_task_byte_per_second`[​](#cumulative_compaction_task_byte_per_second "Direct link to cumulative_compaction_task_byte_per_second") * Unit: Bytes/s * Description: Rate of bytes processed during cumulative compactions. #### `cumulative_compaction_task_cost_time_ms`[​](#cumulative_compaction_task_cost_time_ms "Direct link to cumulative_compaction_task_cost_time_ms") * Unit: ms * Description: Total time spent on cumulative compactions. --- ### Metrics d through h note Metrics for materialized views and shared-data clusters are detailed in the corresponding sections: * [Metrics for asynchronous materialized view metrics](https://docs.starrocks.io/docs/administration/management/monitoring/metrics-materialized_view.md) * [Metrics for Shared-data Dashboard metrics, and Starlet Dashboard metrics](https://docs.starrocks.io/docs/administration/management/monitoring/metrics-shared-data.md) For more information on how to build a monitoring service for your StarRocks cluster, see [Monitor and Alert](https://docs.starrocks.io/docs/administration/management/monitoring/Monitor_and_Alert.md). #### `data_stream_receiver_count`[​](#data_stream_receiver_count "Direct link to data_stream_receiver_count") * Unit: Count * Description: Cumulative number of instances serving as Exchange receivers in BE. #### `datacache_disk_quota_bytes`[​](#datacache_disk_quota_bytes "Direct link to datacache_disk_quota_bytes") * Unit: Bytes * Type: Gauge * Description: The configured disk quota for datacache. #### `datacache_disk_used_bytes`[​](#datacache_disk_used_bytes "Direct link to datacache_disk_used_bytes") * Unit: Bytes * Type: Gauge * Description: The current disk usage of datacache. #### `datacache_mem_quota_bytes`[​](#datacache_mem_quota_bytes "Direct link to datacache_mem_quota_bytes") * Unit: Bytes * Type: Gauge * Description: The configured memory quota for datacache. #### `datacache_mem_used_bytes`[​](#datacache_mem_used_bytes "Direct link to datacache_mem_used_bytes") * Unit: Bytes * Type: Gauge * Description: The current memory usage of datacache. #### `datacache_meta_used_bytes`[​](#datacache_meta_used_bytes "Direct link to datacache_meta_used_bytes") * Unit: Bytes * Type: Gauge * Description: The memory usage for datacache metadata. #### `date_column_pool_bytes`[​](#date_column_pool_bytes "Direct link to date_column_pool_bytes") * Unit: Bytes * Description: Memory used by the DATE column pool. #### `datetime_column_pool_bytes`[​](#datetime_column_pool_bytes "Direct link to datetime_column_pool_bytes") * Unit: Bytes * Description: Memory used by the DATETIME column pool. #### `decimal_column_pool_bytes`[​](#decimal_column_pool_bytes "Direct link to decimal_column_pool_bytes") * Unit: Bytes * Description: Memory used by the DECIMAL column pool. #### `delta_column_group_get_hit_cache`[​](#delta_column_group_get_hit_cache "Direct link to delta_column_group_get_hit_cache") * Unit: Count * Description: Total number of delta column group cache hits (for Primary Key tables only). #### `delta_column_group_get_non_pk_hit_cache`[​](#delta_column_group_get_non_pk_hit_cache "Direct link to delta_column_group_get_non_pk_hit_cache") * Unit: Count * Description: Total number of hits in the delta column group cache (for non-Primary Key tables). #### `delta_column_group_get_non_pk_total`[​](#delta_column_group_get_non_pk_total "Direct link to delta_column_group_get_non_pk_total") * Unit: Count * Description: Total number of times to get delta column group (for non-Primary Key tables only). #### `delta_column_group_get_total`[​](#delta_column_group_get_total "Direct link to delta_column_group_get_total") * Unit: Count * Description: Total number of times to get delta column groups (for Primary Key tables). #### `disk_bytes_read`[​](#disk_bytes_read "Direct link to disk_bytes_read") * Unit: Bytes * Description: Total bytes read from the disk(sectors read \* 512). #### `disk_bytes_written`[​](#disk_bytes_written "Direct link to disk_bytes_written") * Unit: Bytes * Description: Total bytes written to disk. #### `disk_free`[​](#disk_free "Direct link to disk_free") * Unit: Bytes * Type: Average * Description: Free disk capacity. #### `disk_io_svctm`[​](#disk_io_svctm "Direct link to disk_io_svctm") * Unit: ms * Type: Average * Description: Disk IO service time. #### `disk_io_time_ms`[​](#disk_io_time_ms "Direct link to disk_io_time_ms") * Unit: ms * Description: Time spent on I/Os. #### `disk_io_time_weigthed`[​](#disk_io_time_weigthed "Direct link to disk_io_time_weigthed") * Unit: ms * Description: Weighted time spent on I/Os. #### `disk_io_util`[​](#disk_io_util "Direct link to disk_io_util") * Unit: - * Type: Average * Description: Disk usage. #### `disk_read_time_ms`[​](#disk_read_time_ms "Direct link to disk_read_time_ms") * Unit: ms * Description: Time spent on reading from disk. Unit: ms. #### `disk_reads_completed`[​](#disk_reads_completed "Direct link to disk_reads_completed") * Unit: Count * Description: Number of successfully completed disk reads. #### `disk_sync_total (Deprecated)`[​](#disk_sync_total-deprecated "Direct link to disk_sync_total-deprecated") #### `disk_used`[​](#disk_used "Direct link to disk_used") * Unit: Bytes * Type: Average * Description: Used disk capacity. #### `disk_write_time_ms`[​](#disk_write_time_ms "Direct link to disk_write_time_ms") * Unit: ms * Description: Time spent on disk writing. Unit: ms. #### `disk_writes_completed`[​](#disk_writes_completed "Direct link to disk_writes_completed") * Unit: Count * Description: Total number of successfully completed disk writes. #### `disks_avail_capacity`[​](#disks_avail_capacity "Direct link to disks_avail_capacity") * Description: Available capacity of a specific disk. #### `disks_data_used_capacity`[​](#disks_data_used_capacity "Direct link to disks_data_used_capacity") * Description: Used capacity of each disk (represented by a storage path). #### `disks_state`[​](#disks_state "Direct link to disks_state") * Unit: - * Description: State of each disk. `1` indicates that the disk is in use, and `0` indicates that it is not in use. #### `disks_total_capacity`[​](#disks_total_capacity "Direct link to disks_total_capacity") * Description: Total capacity of the disk. #### `double_column_pool_bytes`[​](#double_column_pool_bytes "Direct link to double_column_pool_bytes") * Unit: Bytes * Description: Memory used by the DOUBLE column pool. #### `encryption_keys_created`[​](#encryption_keys_created "Direct link to encryption_keys_created") * Unit: Count * Type: Cumulative * Description: number of file encryption keys created for file encryption #### `encryption_keys_in_cache`[​](#encryption_keys_in_cache "Direct link to encryption_keys_in_cache") * Unit: Count * Type: Instantaneous * Description: number of encryption keys currently in key cache #### `encryption_keys_unwrapped`[​](#encryption_keys_unwrapped "Direct link to encryption_keys_unwrapped") * Unit: Count * Type: Cumulative * Description: number of encryption meta unwrapped for file decryption #### `engine_requests_total`[​](#engine_requests_total "Direct link to engine_requests_total") * Unit: Count * Description: Total count of all types of requests between BE and FE, including CREATE TABLE, Publish Version, and tablet clone. #### `fd_num_limit`[​](#fd_num_limit "Direct link to fd_num_limit") * Unit: Count * Description: Maximum number of file descriptors. #### `fd_num_used`[​](#fd_num_used "Direct link to fd_num_used") * Unit: Count * Description: Number of file descriptors currently in use. #### `fe_cancelled_broker_load_job`[​](#fe_cancelled_broker_load_job "Direct link to fe_cancelled_broker_load_job") * Unit: Count * Type: Average * Description: Number of cancelled broker jobs. #### `fe_cancelled_delete_load_job`[​](#fe_cancelled_delete_load_job "Direct link to fe_cancelled_delete_load_job") * Unit: Count * Type: Average * Description: Number of cancelled delete jobs. #### `fe_cancelled_hadoop_load_job`[​](#fe_cancelled_hadoop_load_job "Direct link to fe_cancelled_hadoop_load_job") * Unit: Count * Type: Average * Description: Number of cancelled hadoop jobs. #### `fe_cancelled_insert_load_job`[​](#fe_cancelled_insert_load_job "Direct link to fe_cancelled_insert_load_job") * Unit: Count * Type: Average * Description: Number of cancelled insert jobs. #### `fe_checkpoint_push_per_second`[​](#fe_checkpoint_push_per_second "Direct link to fe_checkpoint_push_per_second") * Unit: Count/s * Type: Average * Description: Number of FE checkpoints. #### `fe_committed_broker_load_job`[​](#fe_committed_broker_load_job "Direct link to fe_committed_broker_load_job") * Unit: Count * Type: Average * Description: Number of committed broker jobs. #### `fe_committed_delete_load_job`[​](#fe_committed_delete_load_job "Direct link to fe_committed_delete_load_job") * Unit: Count * Type: Average * Description: Number of committed delete jobs. #### `fe_committed_hadoop_load_job`[​](#fe_committed_hadoop_load_job "Direct link to fe_committed_hadoop_load_job") * Unit: Count * Type: Average * Description: Number of committed hadoop jobs. #### `fe_committed_insert_load_job`[​](#fe_committed_insert_load_job "Direct link to fe_committed_insert_load_job") * Unit: Count * Type: Average * Description: Number of committed insert jobs. #### `fe_connection_total`[​](#fe_connection_total "Direct link to fe_connection_total") * Unit: Count * Type: Cumulative * Description: Total number of FE connections. #### `fe_connections_per_second`[​](#fe_connections_per_second "Direct link to fe_connections_per_second") * Unit: Count/s * Type: Average * Description: New connection rate of FE. #### `fe_edit_log_read`[​](#fe_edit_log_read "Direct link to fe_edit_log_read") * Unit: Count/s * Type: Average * Description: Read speed of FE edit log. #### `fe_edit_log_size_bytes`[​](#fe_edit_log_size_bytes "Direct link to fe_edit_log_size_bytes") * Unit: Bytes/s * Type: Average * Description: Size of FE edit log. #### `fe_edit_log_write`[​](#fe_edit_log_write "Direct link to fe_edit_log_write") * Unit: Bytes/s * Type: Average * Description: Write speed of FE edit log. #### `fe_finished_broker_load_job`[​](#fe_finished_broker_load_job "Direct link to fe_finished_broker_load_job") * Unit: Count * Type: Average * Description: Number of finished broker jobs. #### `fe_finished_delete_load_job`[​](#fe_finished_delete_load_job "Direct link to fe_finished_delete_load_job") * Unit: Count * Type: Average * Description: Number of finished delete jobs. #### `fe_finished_hadoop_load_job`[​](#fe_finished_hadoop_load_job "Direct link to fe_finished_hadoop_load_job") * Unit: Count * Type: Average * Description: Number of completedhadoop jobs. #### `fe_finished_insert_load_job`[​](#fe_finished_insert_load_job "Direct link to fe_finished_insert_load_job") * Unit: Count * Type: Average * Description: Number of completed insert jobs. #### `fe_loading_broker_load_job`[​](#fe_loading_broker_load_job "Direct link to fe_loading_broker_load_job") * Unit: Count * Type: Average * Description: Number of loading broker jobs. #### `fe_loading_delete_load_job`[​](#fe_loading_delete_load_job "Direct link to fe_loading_delete_load_job") * Unit: Count * Type: Average * Description: Number of loading delete jobs. #### `fe_loading_hadoop_load_job`[​](#fe_loading_hadoop_load_job "Direct link to fe_loading_hadoop_load_job") * Unit: Count * Type: Average * Description: Number of loading hadoop jobs. #### `fe_loading_insert_load_job`[​](#fe_loading_insert_load_job "Direct link to fe_loading_insert_load_job") * Unit: Count * Type: Average * Description: Number of loading insert jobs. #### `fe_pending_broker_load_job`[​](#fe_pending_broker_load_job "Direct link to fe_pending_broker_load_job") * Unit: Count * Type: Average * Description: Number of pending broker jobs. #### `fe_pending_delete_load_job`[​](#fe_pending_delete_load_job "Direct link to fe_pending_delete_load_job") * Unit: Count * Type: Average * Description: Number of pending delete jobs. #### `fe_pending_hadoop_load_job`[​](#fe_pending_hadoop_load_job "Direct link to fe_pending_hadoop_load_job") * Unit: Count * Type: Average * Description: Number of pending hadoop jobs. #### `fe_pending_insert_load_job`[​](#fe_pending_insert_load_job "Direct link to fe_pending_insert_load_job") * Unit: Count * Type: Average * Description: Number of pending insert jobs. #### `fe_rollup_running_alter_job`[​](#fe_rollup_running_alter_job "Direct link to fe_rollup_running_alter_job") * Unit: Count * Type: Average * Description: Number of jobs created in rollup. #### `fe_schema_change_running_job`[​](#fe_schema_change_running_job "Direct link to fe_schema_change_running_job") * Unit: Count * Type: Average * Description: Number of jobs in schema change. #### `float_column_pool_bytes`[​](#float_column_pool_bytes "Direct link to float_column_pool_bytes") * Unit: Bytes * Description: Memory used by the FLOAT column pool. #### `fragment_endpoint_count`[​](#fragment_endpoint_count "Direct link to fragment_endpoint_count") * Unit: Count * Description: Cumulative number of instances serving as Exchange senders in BE. #### `fragment_request_duration_us`[​](#fragment_request_duration_us "Direct link to fragment_request_duration_us") * Unit: us * Description: Cumulative execution time of fragment instances (for non-pipeline engine). #### `fragment_requests_total`[​](#fragment_requests_total "Direct link to fragment_requests_total") * Unit: Count * Description: Total fragment instances executing on a BE (for non-pipeline engine). #### `hive_write_bytes`[​](#hive_write_bytes "Direct link to hive_write_bytes") * Unit: Bytes * Type: Cumulative * Labels: `write_type` (`insert` or `overwrite`) * Description: Total written bytes from Hive write tasks (`INSERT`, `INSERT OVERWRITE`). This represents the total size of data files written to the Hive table. `write_type` distinguishes between the operation types. #### `hive_write_duration_ms_total`[​](#hive_write_duration_ms_total "Direct link to hive_write_duration_ms_total") * Unit: Millisecond * Type: Cumulative * Labels: `write_type` (`insert` or `overwrite`) * Description: Total execution time of Hive write tasks (`INSERT`, `INSERT OVERWRITE`) in milliseconds. The duration of each task is added after it ends. `write_type` distinguishes between the operation types. #### `hive_write_files`[​](#hive_write_files "Direct link to hive_write_files") * Unit: Count * Type: Cumulative * Labels: `write_type` (`insert` or `overwrite`) * Description: Total number of data files written to Hive from write tasks (`INSERT`, `INSERT OVERWRITE`). This represents the count of data files written to the Hive table. `write_type` distinguishes between the operation types. #### `hive_write_rows`[​](#hive_write_rows "Direct link to hive_write_rows") * Unit: Rows * Type: Cumulative * Labels: `write_type` (`insert` or `overwrite`) * Description: Total written rows from Hive write tasks (`INSERT`, `INSERT OVERWRITE`). This represents the number of rows written to the Hive table. `write_type` distinguishes between the operation types. #### `hive_write_total`[​](#hive_write_total "Direct link to hive_write_total") * Unit: Count * Type: Cumulative * Labels: * `status` (`success` or `failed`) * `reason` (`none`, `timeout`, `oom`, `access_denied`, `unknown`) * `write_type` (`insert` or `overwrite`) * Description: Total number of `INSERT` or `INSERT OVERWRITE` tasks that target Hive tables. The metric is incremented by 1 after each task ends, regardless of success or failure. `write_type` distinguishes between the operation types. #### `http_request_send_bytes (Deprecated)`[​](#http_request_send_bytes-deprecated "Direct link to http_request_send_bytes-deprecated") #### `http_requests_total (Deprecated)`[​](#http_requests_total-deprecated "Direct link to http_requests_total-deprecated") --- ### Metrics i through p note Metrics for materialized views and shared-data clusters are detailed in the corresponding sections: * [Metrics for asynchronous materialized view metrics](https://docs.starrocks.io/docs/administration/management/monitoring/metrics-materialized_view.md) * [Metrics for Shared-data Dashboard metrics, and Starlet Dashboard metrics](https://docs.starrocks.io/docs/administration/management/monitoring/metrics-shared-data.md) For more information on how to build a monitoring service for your StarRocks cluster, see [Monitor and Alert](https://docs.starrocks.io/docs/administration/management/monitoring/Monitor_and_Alert.md). #### `iceberg_compaction_duration_ms_total`[​](#iceberg_compaction_duration_ms_total "Direct link to iceberg_compaction_duration_ms_total") * Unit: Millisecond * Type: Cumulative * Labels: `compaction_type` (`manual` or `auto`) * Description: Total time spent running Iceberg compaction tasks. #### `iceberg_compaction_input_files_total`[​](#iceberg_compaction_input_files_total "Direct link to iceberg_compaction_input_files_total") * Unit: Count * Type: Cumulative * Labels: `compaction_type` (`manual` or `auto`) * Description: Total number of data files read by Iceberg compaction tasks. #### `iceberg_compaction_output_files_total`[​](#iceberg_compaction_output_files_total "Direct link to iceberg_compaction_output_files_total") * Unit: Count * Type: Cumulative * Labels: `compaction_type` (`manual` or `auto`) * Description: Total number of data files produced by Iceberg compaction tasks. #### `iceberg_compaction_removed_delete_files_total`[​](#iceberg_compaction_removed_delete_files_total "Direct link to iceberg_compaction_removed_delete_files_total") * Unit: Count * Type: Cumulative * Labels: `compaction_type` (`manual` or `auto`) * Description: Total number of delete files removed by Iceberg manual compaction tasks. #### `iceberg_compaction_total`[​](#iceberg_compaction_total "Direct link to iceberg_compaction_total") * Unit: Count * Type: Cumulative * Labels: `compaction_type` (`manual` or `auto`) * Description: Total number of Iceberg compaction (`rewrite_data_files`) tasks. #### `iceberg_delete_bytes`[​](#iceberg_delete_bytes "Direct link to iceberg_delete_bytes") * Unit: Bytes * Type: Cumulative * Labels: `delete_type` (`position` or `metadata`) * Description: Total deleted bytes from Iceberg `DELETE` tasks. For `metadata` delete, this represents the size of deleted data files. For `position` delete, this represents the size of position delete files created. #### `iceberg_delete_duration_ms_total`[​](#iceberg_delete_duration_ms_total "Direct link to iceberg_delete_duration_ms_total") * Unit: Millisecond * Type: Cumulative * Labels: `delete_type` (`position` or `metadata`) * Description: Total execution time of Iceberg `DELETE` tasks in milliseconds. The duration of each task is added after it ends. `delete_type` distinguishes between two delete methods. #### `iceberg_delete_rows`[​](#iceberg_delete_rows "Direct link to iceberg_delete_rows") * Unit: Rows * Type: Cumulative * Labels: `delete_type` (`position` or `metadata`) * Description: Total deleted rows from Iceberg `DELETE` tasks. For `metadata` delete, this represents the number of rows in deleted data files. For `position` delete, this represents the number of position deletes created. #### `iceberg_delete_total`[​](#iceberg_delete_total "Direct link to iceberg_delete_total") * Unit: Count * Type: Cumulative * Labels: * `status` (`success` or `failed`) * `reason` (`none`, `timeout`, `oom`, `access_denied`, `unknown`) * `delete_type` (`position` or `metadata`) * Description: Total number of `DELETE` tasks that target Iceberg tables. The metric is incremented by 1 after each task ends, regardless of success or failure. `delete_type` distinguishes between two delete methods: `position` (generates position delete files) and `metadata` (metadata-level delete). #### `iceberg_metadata_table_query_total`[​](#iceberg_metadata_table_query_total "Direct link to iceberg_metadata_table_query_total") * Unit: Count * Type: Cumulative * Labels: `metadata_table` (`refs`, `history`, `metadata_log_entries`, `snapshots`, `manifests`, `files`, `partitions`, or `properties`) * Description: Total number of SQL queries that access Iceberg metadata tables. Each query is counted under the `metadata_table` label that identifies the metadata table being accessed. #### `iceberg_time_travel_query_total`[​](#iceberg_time_travel_query_total "Direct link to iceberg_time_travel_query_total") * Unit: Count * Type: Cumulative * Labels: `time_travel_type` (`branch`, `tag`, `snapshot`, or `timestamp`) for the categorized series. * Description: Total number of Iceberg time travel queries. The unlabeled series counts each time travel query once. The labeled series count each distinct time travel type used by the query. `snapshot` means `FOR VERSION AS OF `, `branch` and `tag` mean `FOR VERSION AS OF `, and `timestamp` means `FOR TIMESTAMP AS OF ...`. #### `iceberg_write_bytes`[​](#iceberg_write_bytes "Direct link to iceberg_write_bytes") * Unit: Bytes * Type: Cumulative * Labels: `write_type` (`insert`, `overwrite`, or `ctas`) * Description: Total written bytes from Iceberg write tasks (`INSERT`, `INSERT OVERWRITE`, `CTAS`). This represents the total size of data files written to the Iceberg table. `write_type` distinguishes between the operation types. #### `iceberg_write_duration_ms_total`[​](#iceberg_write_duration_ms_total "Direct link to iceberg_write_duration_ms_total") * Unit: Millisecond * Type: Cumulative * Labels: `write_type` (`insert`, `overwrite`, or `ctas`) * Description: Total execution time of Iceberg write tasks (`INSERT`, `INSERT OVERWRITE`, `CTAS`) in milliseconds. The duration of each task is added after it ends. `write_type` distinguishes between the operation types. #### `iceberg_write_files`[​](#iceberg_write_files "Direct link to iceberg_write_files") * Unit: Count * Type: Cumulative * Labels: `write_type` (`insert`, `overwrite`, or `ctas`) * Description: Total number of data files written to Iceberg from write tasks (`INSERT`, `INSERT OVERWRITE`, `CTAS`). This represents the count of data files written to the Iceberg table. `write_type` distinguishes between the operation types. #### `iceberg_write_rows`[​](#iceberg_write_rows "Direct link to iceberg_write_rows") * Unit: Rows * Type: Cumulative * Labels: `write_type` (`insert`, `overwrite`, or `ctas`) * Description: Total written rows from Iceberg write tasks (`INSERT`, `INSERT OVERWRITE`, `CTAS`). This represents the number of rows written to the Iceberg table. `write_type` distinguishes between the operation types. #### `iceberg_write_total`[​](#iceberg_write_total "Direct link to iceberg_write_total") * Unit: Count * Type: Cumulative * Labels: * `status` (`success` or `failed`) * `reason` (`none`, `timeout`, `oom`, `access_denied`, `unknown`) * `write_type` (`insert`, `overwrite`, or `ctas`) * Description: Total number of `INSERT`, `INSERT OVERWRITE`, or `CTAS` tasks that target Iceberg tables. The metric is incremented by 1 after each task ends, regardless of success or failure. `write_type` distinguishes between the operation types. #### `int128_column_pool_bytes`[​](#int128_column_pool_bytes "Direct link to int128_column_pool_bytes") * Unit: Bytes * Description: Memory used by the INT128 column pool. #### `int16_column_pool_bytes`[​](#int16_column_pool_bytes "Direct link to int16_column_pool_bytes") * Unit: Bytes * Description: Memory used by the INT16 column pool. #### `int32_column_pool_bytes`[​](#int32_column_pool_bytes "Direct link to int32_column_pool_bytes") * Unit: Bytes * Description: Memory used by the INT32 column pool. #### `int64_column_pool_bytes`[​](#int64_column_pool_bytes "Direct link to int64_column_pool_bytes") * Unit: Bytes * Description: Memory used by the INT64 column pool. #### `int8_column_pool_bytes`[​](#int8_column_pool_bytes "Direct link to int8_column_pool_bytes") * Unit: Bytes * Description: Memory used by the INT8 column pool. #### `jemalloc_active_bytes`[​](#jemalloc_active_bytes "Direct link to jemalloc_active_bytes") * Unit: Bytes * Description: Total bytes in active pages allocated by the application. #### `jemalloc_allocated_bytes`[​](#jemalloc_allocated_bytes "Direct link to jemalloc_allocated_bytes") * Unit: Bytes * Description: Total number of bytes allocated by the application. #### `jemalloc_mapped_bytes`[​](#jemalloc_mapped_bytes "Direct link to jemalloc_mapped_bytes") * Unit: Bytes * Description: Total number of bytes in active extents mapped by the allocator. #### `jemalloc_metadata_bytes`[​](#jemalloc_metadata_bytes "Direct link to jemalloc_metadata_bytes") * Unit: Bytes * Description: Total number of bytes dedicated to metadata, comprising base allocations used for bootstrap-sensitive allocator metadata structures and internal allocations. The usage of transparent huge pages is not included in this item. #### `jemalloc_metadata_thp`[​](#jemalloc_metadata_thp "Direct link to jemalloc_metadata_thp") * Unit: Count * Description: Number of Transparent Huge Pages used for metadata. #### `jemalloc_resident_bytes`[​](#jemalloc_resident_bytes "Direct link to jemalloc_resident_bytes") * Unit: Bytes * Description: Maximum number of bytes in physically resident data pages mapped by the allocator, comprising all pages dedicated to allocator metadata, pages backing active allocations, and unused dirty pages. #### `jemalloc_retained_bytes`[​](#jemalloc_retained_bytes "Direct link to jemalloc_retained_bytes") * Unit: Bytes * Description: Total bytes of virtual memory mappings that were retained rather than returned to the operating system via operations like munmap(2). #### `jit_cache_mem_bytes`[​](#jit_cache_mem_bytes "Direct link to jit_cache_mem_bytes") * Unit: Bytes * Description: Memory used by jit compiled function cache. #### `lake_compaction_failed`[​](#lake_compaction_failed "Direct link to lake_compaction_failed") * Unit: Count * Description: Counter of failed lake compaction jobs. #### `lake_compaction_partial_success`[​](#lake_compaction_partial_success "Direct link to lake_compaction_partial_success") * Unit: Count * Description: Counter of partially successful lake compaction jobs. #### `lake_compaction_running`[​](#lake_compaction_running "Direct link to lake_compaction_running") * Unit: Count * Description: Number of currently running lake compaction jobs. #### `lake_compaction_running_tasks`[​](#lake_compaction_running_tasks "Direct link to lake_compaction_running_tasks") * Unit: Count * Description: Number of tablets currently being compacted across all running shared-data (lake) compaction jobs. This is the same unit the scheduler caps with the `lake_compaction_max_tasks` config, and is finer-grained than `lake_compaction_running`, which counts compaction jobs (one per partition) — a single job fans out into one tablet-level task per tablet. Carries an `is_leader` label; follower FEs export the metric with `is_leader="false"` and value 0, so dashboards should filter on `is_leader="true"`. #### `lake_compaction_score_at_trigger`[​](#lake_compaction_score_at_trigger "Direct link to lake_compaction_score_at_trigger") * Unit: Score * Type: Gauge * Description: Compaction score of the most recent partition that triggered a lake compaction job, rounded to the nearest integer. The value is the partition's *max* tablet-level score (`Quantiles.getMax()`), matching the criterion the scheduler uses to pick partitions for compaction. Updated once per partition per trigger; the gauge holds the value of the most recent update. This gauge does not decay: on the leader FE, when no compactions are running it retains the last trigger's value (it is not reset to 0). The value is process-local (an in-memory counter on the leader, not persisted), so after an FE leader failover the newly promoted leader starts at 0 and reports 0 until its first compaction trigger — it does not inherit the previous leader's value. Alert on it together with `lake_compaction_running > 0` rather than reading it in isolation. Carries an `is_leader` label; follower FEs export `is_leader="false"` and return 0, so dashboards should filter on `is_leader="true"`. #### `lake_compaction_success`[​](#lake_compaction_success "Direct link to lake_compaction_success") * Unit: Count * Description: Counter of successful lake compaction jobs. #### `lake_vacuum_del_file_batch_size_minute`[​](#lake_vacuum_del_file_batch_size_minute "Direct link to lake_vacuum_del_file_batch_size_minute") * Unit: Count (files per batch) * Type: Gauge * Description: Recent (60s) mean of files per `DeleteObjects` batch sent by Vacuum on shared-data clusters. #### `lake_vacuum_del_file_retries_minute`[​](#lake_vacuum_del_file_retries_minute "Direct link to lake_vacuum_del_file_retries_minute") * Unit: Count * Type: Gauge * Description: Number of Vacuum delete retries triggered in the last 60s on shared-data clusters. Surfaces transient object-storage throttling (SlowDown / try-again). #### `load_bytes`[​](#load_bytes "Direct link to load_bytes") * Unit: Bytes * Description: Total loaded bytes. #### `load_channel_count`[​](#load_channel_count "Direct link to load_channel_count") * Unit: Count * Description: Total number of loading channels. #### `load_mem_bytes`[​](#load_mem_bytes "Direct link to load_mem_bytes") * Unit: Bytes * Description: Memory cost of data loading. #### `load_rows`[​](#load_rows "Direct link to load_rows") * Unit: Count * Description: Total loaded rows. #### `load_rpc_threadpool_size`[​](#load_rpc_threadpool_size "Direct link to load_rpc_threadpool_size") * Unit: Count * Description: The current size of the RPC thread pool, which is used for handling Routine Load and loading via table functions. The default value is 10, with a maximum value of 1000. This value is dynamically adjusted based on the usage of the thread pool. #### `local_column_pool_bytes (Deprecated)`[​](#local_column_pool_bytes-deprecated "Direct link to local_column_pool_bytes-deprecated") #### `max_disk_io_util_percent`[​](#max_disk_io_util_percent "Direct link to max_disk_io_util_percent") * Unit: - * Description: Maximum disk I/O utilization percentage. #### `max_network_receive_bytes_rate`[​](#max_network_receive_bytes_rate "Direct link to max_network_receive_bytes_rate") * Unit: Bytes * Description: Total bytes received over the network (maximum value among all network interfaces). #### `max_network_send_bytes_rate`[​](#max_network_send_bytes_rate "Direct link to max_network_send_bytes_rate") * Unit: Bytes * Description: Total bytes sent over the network (maximum value among all network interfaces). #### `memory_pool_bytes_total`[​](#memory_pool_bytes_total "Direct link to memory_pool_bytes_total") * Unit: Bytes * Description: Memory used by the memory pool. #### `memtable_flush_duration_us`[​](#memtable_flush_duration_us "Direct link to memtable_flush_duration_us") * Unit: us * Description: Total time spent on memtable flush. #### `memtable_flush_queue_count`[​](#memtable_flush_queue_count "Direct link to memtable_flush_queue_count") * Unit: Count * Description: Queued task count in the memtable flush thread pool. #### `memtable_flush_total`[​](#memtable_flush_total "Direct link to memtable_flush_total") * Unit: Count * Description: Total number of memtable flushes. #### `merge_commit_append_pipe`[​](#merge_commit_append_pipe "Direct link to merge_commit_append_pipe") * Unit: microsecond * Type: Summary * Description: Time spent appending data to the stream load pipe during merge commit. #### `merge_commit_fail_total`[​](#merge_commit_fail_total "Direct link to merge_commit_fail_total") * Unit: Count * Type: Cumulative * Description: Merge commit requests that failed. #### `merge_commit_pending`[​](#merge_commit_pending "Direct link to merge_commit_pending") * Unit: microsecond * Type: Summary * Description: Time merge commit tasks spend waiting in the pending queue before execution. #### `merge_commit_pending_bytes`[​](#merge_commit_pending_bytes "Direct link to merge_commit_pending_bytes") * Unit: Bytes * Type: Instantaneous * Description: Total bytes of data held by pending merge commit tasks. #### `merge_commit_pending_total`[​](#merge_commit_pending_total "Direct link to merge_commit_pending_total") * Unit: Count * Type: Instantaneous * Description: Merge commit tasks currently waiting in the execution queue. #### `merge_commit_register_pipe_total`[​](#merge_commit_register_pipe_total "Direct link to merge_commit_register_pipe_total") * Unit: Count * Type: Cumulative * Description: Stream load pipes registered for merge commit operations. #### `merge_commit_request`[​](#merge_commit_request "Direct link to merge_commit_request") * Unit: microsecond * Type: Summary * Description: End-to-end processing latency for merge commit requests. #### `merge_commit_request_bytes`[​](#merge_commit_request_bytes "Direct link to merge_commit_request_bytes") * Unit: Bytes * Type: Cumulative * Description: Total bytes of data received across merge commit requests. #### `merge_commit_request_total`[​](#merge_commit_request_total "Direct link to merge_commit_request_total") * Unit: Count * Type: Cumulative * Description: Total number of merge commit requests received by BE. #### `merge_commit_send_rpc_total`[​](#merge_commit_send_rpc_total "Direct link to merge_commit_send_rpc_total") * Unit: Count * Type: Cumulative * Description: RPC requests sent to FE for starting merge commit operations. #### `merge_commit_success_total`[​](#merge_commit_success_total "Direct link to merge_commit_success_total") * Unit: Count * Type: Cumulative * Description: Merge commit requests that finished successfully. #### `merge_commit_unregister_pipe_total`[​](#merge_commit_unregister_pipe_total "Direct link to merge_commit_unregister_pipe_total") * Unit: Count * Type: Cumulative * Description: Stream load pipes unregistered from merge commit operations. Latency metrics expose percentile series such as `merge_commit_request_latency_99` and `merge_commit_request_latency_90`, reported in microseconds. The end-to-end latency obeys: `merge_commit_request = merge_commit_pending + merge_commit_wait_plan + merge_commit_append_pipe + merge_commit_wait_finish` > **Note**: Before v3.4.11, v3.5.12, and v4.0.4, these latency metrics were reported in nanoseconds. #### `merge_commit_wait_finish`[​](#merge_commit_wait_finish "Direct link to merge_commit_wait_finish") * Unit: microsecond * Type: Summary * Description: Time spent waiting for merge commit load operations to finish. #### `merge_commit_wait_plan`[​](#merge_commit_wait_plan "Direct link to merge_commit_wait_plan") * Unit: microsecond * Type: Summary * Description: Combined latency for the RPC request and waiting for the stream load pipe to become available. #### `meta_request_duration`[​](#meta_request_duration "Direct link to meta_request_duration") * Unit: us * Description: Total meta read/write duration. #### `meta_request_total`[​](#meta_request_total "Direct link to meta_request_total") * Unit: Count * Description: Total number of meta read/write requests. #### `metadata_mem_bytes (Deprecated)`[​](#metadata_mem_bytes-deprecated "Direct link to metadata_mem_bytes-deprecated") #### `network_receive_bytes`[​](#network_receive_bytes "Direct link to network_receive_bytes") * Unit: Bytes * Description: Total bytes received via network. #### `network_receive_packets`[​](#network_receive_packets "Direct link to network_receive_packets") * Unit: Count * Description: Total number of packets received through the network. #### `network_send_bytes`[​](#network_send_bytes "Direct link to network_send_bytes") * Unit: Bytes * Description: Number of bytes sent over the network. #### `network_send_packets`[​](#network_send_packets "Direct link to network_send_packets") * Unit: Count * Description: Total number of packets sent through the network. #### `ordinal_index_mem_bytes`[​](#ordinal_index_mem_bytes "Direct link to ordinal_index_mem_bytes") * Unit: Bytes * Description: Memory used by ordinal indexes. #### `page_cache_capacity`[​](#page_cache_capacity "Direct link to page_cache_capacity") * Description: Capacity of the storage page cache. #### `page_cache_hit_count`[​](#page_cache_hit_count "Direct link to page_cache_hit_count") * Unit: Count * Description: Total number of hits in the storage page cache. #### `page_cache_insert_count`[​](#page_cache_insert_count "Direct link to page_cache_insert_count") * Unit: Count * Description: Total number of insert operations in the storage page cache. #### `page_cache_insert_evict_count`[​](#page_cache_insert_evict_count "Direct link to page_cache_insert_evict_count") * Unit: Count * Description: Total number of cache entries evicted during insert operations due to capacity constraints. #### `page_cache_lookup_count`[​](#page_cache_lookup_count "Direct link to page_cache_lookup_count") * Unit: Count * Description: Total number of storage page cache lookups. #### `page_cache_release_evict_count`[​](#page_cache_release_evict_count "Direct link to page_cache_release_evict_count") * Unit: Count * Description: Total number of cache entries evicted during release operations when cache usage exceeds capacity. #### `pip_query_ctx_cnt`[​](#pip_query_ctx_cnt "Direct link to pip_query_ctx_cnt") * Unit: Count * Description: Total number of currently running queries in the BE. #### `pipe_driver_execution_time`[​](#pipe_driver_execution_time "Direct link to pipe_driver_execution_time") * Description: Cumulative time spent by PipelineDriver executors on processing PipelineDrivers. #### `pipe_driver_queue_len`[​](#pipe_driver_queue_len "Direct link to pipe_driver_queue_len") * Unit: Count * Description: Current number of ready drivers in the ready queue waiting for scheduling in BE. #### `pipe_driver_schedule_count`[​](#pipe_driver_schedule_count "Direct link to pipe_driver_schedule_count") * Unit: Count * Description: Cumulative number of driver scheduling times for pipeline executors in the BE. #### `pipe_poller_block_queue_len`[​](#pipe_poller_block_queue_len "Direct link to pipe_poller_block_queue_len") * Unit: Count * Description: Current length of the block queue of PipelineDriverPoller in the pipeline engine. #### `pipe_prepare_pool_queue_len`[​](#pipe_prepare_pool_queue_len "Direct link to pipe_prepare_pool_queue_len") * Unit: Count * Description: Queued task count in the pipeline PREPARE thread pool. This is an instantaneous value. #### `pipe_scan_executor_queuing`[​](#pipe_scan_executor_queuing "Direct link to pipe_scan_executor_queuing") * Unit: Count * Description: Current number of pending asynchronous I/O tasks launched by Scan Operators. #### `pk_index_compaction_queue_count`[​](#pk_index_compaction_queue_count "Direct link to pk_index_compaction_queue_count") * Unit: Count * Description: Queued task count in the Primary Key index compaction thread pool. #### `pk_index_sst_read_error_total`[​](#pk_index_sst_read_error_total "Direct link to pk_index_sst_read_error_total") * Type: Counter * Unit: Count * Description: Total number of SST file read failures in the lake Primary Key persistent index. Incremented when SST multi-get (read) operations fail. #### `pk_index_sst_write_error_total`[​](#pk_index_sst_write_error_total "Direct link to pk_index_sst_write_error_total") * Type: Counter * Unit: Count * Description: Total number of SST file write failures in the lake Primary Key persistent index. Incremented when SST file build fails. #### `plan_fragment_count`[​](#plan_fragment_count "Direct link to plan_fragment_count") * Unit: Count * Description: Number of currently running query plan fragments. #### `process_fd_num_limit_hard`[​](#process_fd_num_limit_hard "Direct link to process_fd_num_limit_hard") * Unit: Count * Description: Hard limit on the maximum number of file descriptors. #### `process_fd_num_limit_soft`[​](#process_fd_num_limit_soft "Direct link to process_fd_num_limit_soft") * Unit: Count * Description: Soft limit on the maximum number of file descriptors. Please note that this item indicates the soft limit. You can set the hard limit using the `ulimit` command. #### `process_fd_num_used`[​](#process_fd_num_used "Direct link to process_fd_num_used") * Unit: Count * Description: Number of file descriptors currently in use in this BE process. #### `process_mem_bytes`[​](#process_mem_bytes "Direct link to process_mem_bytes") * Unit: Bytes * Description: Memory used by this process. #### `process_thread_num`[​](#process_thread_num "Direct link to process_thread_num") * Unit: Count * Description: Total number of threads in this process. #### `publish_version_queue_count`[​](#publish_version_queue_count "Direct link to publish_version_queue_count") * Unit: Count * Description: Queued task count in the Publish Version thread pool. #### `push_request_duration_us`[​](#push_request_duration_us "Direct link to push_request_duration_us") * Unit: us * Description: Total time spent on Spark Load. #### `push_request_write_bytes`[​](#push_request_write_bytes "Direct link to push_request_write_bytes") * Unit: Bytes * Description: Total bytes written via Spark Load. #### `push_request_write_bytes_per_second`[​](#push_request_write_bytes_per_second "Direct link to push_request_write_bytes_per_second") * Unit: Bytes/s * Description: Data writing rate of Spark Load. #### `push_request_write_rows`[​](#push_request_write_rows "Direct link to push_request_write_rows") * Unit: Count * Description: Total rows written via Spark Load. #### `push_requests_total`[​](#push_requests_total "Direct link to push_requests_total") * Unit: Count * Description: Total number of successful and failed Spark Load requests. --- ### Metrics q through r #### `query_cache_capacity`[​](#query_cache_capacity "Direct link to query_cache_capacity") * Description: Capacity of query cache. #### `query_cache_hit_count`[​](#query_cache_hit_count "Direct link to query_cache_hit_count") * Unit: Count * Description: Number of query cache hits. #### `query_cache_hit_ratio`[​](#query_cache_hit_ratio "Direct link to query_cache_hit_ratio") * Unit: - * Description: Hit ratio of query cache. #### `query_cache_lookup_count`[​](#query_cache_lookup_count "Direct link to query_cache_lookup_count") * Unit: Count * Description: Total number of query cache lookups. #### `query_cache_usage`[​](#query_cache_usage "Direct link to query_cache_usage") * Unit: Bytes * Description: Current query cache usages. #### `query_cache_usage_ratio`[​](#query_cache_usage_ratio "Direct link to query_cache_usage_ratio") * Unit: - * Description: Current query cache usage ratio. #### `query_mem_bytes`[​](#query_mem_bytes "Direct link to query_mem_bytes") * Unit: Bytes * Description: Memory used by queries. #### `query_scan_bytes`[​](#query_scan_bytes "Direct link to query_scan_bytes") * Unit: Bytes * Description: Total number of scanned bytes. #### `query_scan_bytes_per_second`[​](#query_scan_bytes_per_second "Direct link to query_scan_bytes_per_second") * Unit: Bytes/s * Description: Estimated rate of scanned bytes per second. #### `query_scan_rows`[​](#query_scan_rows "Direct link to query_scan_rows") * Unit: Count * Description: Total number of scanned rows. #### `query_spill_trigger_total`[​](#query_spill_trigger_total "Direct link to query_spill_trigger_total") * Unit: Count * Labels: `storage_type` * Description: Number of spillable operator instances that triggered at least one spill, broken down by storage backend (`local`, `remote`). Incremented once per operator instance at the first flush callback. #### `query_spill_bytes_write_total`[​](#query_spill_bytes_write_total "Direct link to query_spill_bytes_write_total") * Unit: Bytes * Labels: `storage_type` * Description: Cumulative payload bytes written by spillable operators to spill storage, broken down by storage backend (`local`, `remote`). #### `query_spill_bytes_read_total`[​](#query_spill_bytes_read_total "Direct link to query_spill_bytes_read_total") * Unit: Bytes * Labels: `storage_type` * Description: Cumulative payload bytes read back from spill storage during restore, broken down by storage backend. #### `query_spill_blocks_write_total`[​](#query_spill_blocks_write_total "Direct link to query_spill_blocks_write_total") * Unit: Count * Labels: `storage_type` * Description: Number of spill blocks allocated for writing, broken down by storage backend. Useful for estimating IO count scale on the write path. #### `query_spill_blocks_read_total`[​](#query_spill_blocks_read_total "Direct link to query_spill_blocks_read_total") * Unit: Count * Labels: `storage_type` * Description: Number of spill blocks opened for reading, broken down by storage backend. Useful for estimating IO count scale on the read path. #### `query_spill_write_io_duration_ns_total`[​](#query_spill_write_io_duration_ns_total "Direct link to query_spill_write_io_duration_ns_total") * Unit: Nanoseconds * Labels: `storage_type` * Description: Cumulative wall-clock time spent in write-side spill IO (block append and flush), broken down by storage backend. Useful for tracking write-side spill performance. #### `query_spill_read_io_duration_ns_total`[​](#query_spill_read_io_duration_ns_total "Direct link to query_spill_read_io_duration_ns_total") * Unit: Nanoseconds * Labels: `storage_type` * Description: Cumulative wall-clock time spent in read-side spill IO (block reads during restore), broken down by storage backend. Useful for tracking read-side spill performance. #### `readable_blocks_total (Deprecated)`[​](#readable_blocks_total-deprecated "Direct link to readable_blocks_total-deprecated") #### `recycle_bin_database_num`[​](#recycle_bin_database_num "Direct link to recycle_bin_database_num") * Unit: Count * Description: Number of databases currently held in the FE catalog recycle bin. #### `recycle_bin_partition_num`[​](#recycle_bin_partition_num "Direct link to recycle_bin_partition_num") * Unit: Count * Description: Number of partitions currently held in the FE catalog recycle bin. #### `recycle_bin_table_num`[​](#recycle_bin_table_num "Direct link to recycle_bin_table_num") * Unit: Count * Description: Number of tables currently held in the FE catalog recycle bin. #### `resource_group_bigquery_count`[​](#resource_group_bigquery_count "Direct link to resource_group_bigquery_count") * Unit: Count * Description: Number of queries in each resource group that have triggered the limit for big queries. This is an instantaneous value. #### `resource_group_concurrency_overflow_count`[​](#resource_group_concurrency_overflow_count "Direct link to resource_group_concurrency_overflow_count") * Unit: Count * Description: Number of queries in each resource group that have triggered the concurrency limit. This is an instantaneous value. #### `resource_group_connector_scan_use_ratio (Deprecated)`[​](#resource_group_connector_scan_use_ratio-deprecated "Direct link to resource_group_connector_scan_use_ratio-deprecated") * Unit: - * Description: Ratio of external table scan thread time slices used by each resource group to the total used by all resource groups. This is an average value over the time interval between two metric retrievals. #### `resource_group_cpu_limit_ratio`[​](#resource_group_cpu_limit_ratio "Direct link to resource_group_cpu_limit_ratio") * Unit: - * Description: Ratio of the CPU core limit for each resource group to the total CPU core limit for all resource groups. This is an instantaneous value. #### `resource_group_cpu_use_ratio (Deprecated)`[​](#resource_group_cpu_use_ratio-deprecated "Direct link to resource_group_cpu_use_ratio-deprecated") * Unit: - * Description: Ratio of pipeline thread time slices used by each resource group to the total used by all resource groups. This is an average value over the time interval between two metric retrievals. #### `resource_group_inuse_cpu_cores`[​](#resource_group_inuse_cpu_cores "Direct link to resource_group_inuse_cpu_cores") * Unit: Count * Description: Estimated number of CPU cores currently in use by each resource group. This is an average value over the time interval between two metric retrievals. #### `resource_group_mem_inuse_bytes`[​](#resource_group_mem_inuse_bytes "Direct link to resource_group_mem_inuse_bytes") * Unit: Bytes * Description: Currently used memory by each resource group, measured in bytes. This is an instantaneous value. #### `resource_group_mem_limit_bytes`[​](#resource_group_mem_limit_bytes "Direct link to resource_group_mem_limit_bytes") * Unit: Bytes * Description: Memory limit for each resource group, measured in bytes. This is an instantaneous value. #### `resource_group_running_queries`[​](#resource_group_running_queries "Direct link to resource_group_running_queries") * Unit: Count * Description: Number of queries currently running in each resource group. This is an instantaneous value. #### `resource_group_scan_use_ratio (Deprecated)`[​](#resource_group_scan_use_ratio-deprecated "Direct link to resource_group_scan_use_ratio-deprecated") * Unit: - * Description: Ratio of internal table scan thread time slices used by each resource group to the total used by all resource groups. This is an average value over the time interval between two metric retrievals. #### `resource_group_total_queries`[​](#resource_group_total_queries "Direct link to resource_group_total_queries") * Unit: Count * Description: Total number of queries executed in each resource group, including those currently running. This is an instantaneous value. #### `result_block_queue_count`[​](#result_block_queue_count "Direct link to result_block_queue_count") * Unit: Count * Description: Number of results in the result block queue. #### `result_buffer_block_count`[​](#result_buffer_block_count "Direct link to result_buffer_block_count") * Unit: Count * Description: Number of blocks in the result buffer. #### `routine_load_task_count`[​](#routine_load_task_count "Direct link to routine_load_task_count") * Unit: Count * Description: Number of currently running Routine Load tasks. #### `rowset_count_generated_and_in_use`[​](#rowset_count_generated_and_in_use "Direct link to rowset_count_generated_and_in_use") * Unit: Count * Description: Number of rowset IDs currently in use. #### `rowset_metadata_mem_bytes`[​](#rowset_metadata_mem_bytes "Direct link to rowset_metadata_mem_bytes") * Unit: Bytes * Description: Total number of bytes for rowset metadata. #### `running_base_compaction_task_num`[​](#running_base_compaction_task_num "Direct link to running_base_compaction_task_num") * Unit: Count * Description: Total number of running base compaction tasks. #### `running_cumulative_compaction_task_num`[​](#running_cumulative_compaction_task_num "Direct link to running_cumulative_compaction_task_num") * Unit: Count * Description: Total number of running cumulative compactions. #### `running_update_compaction_task_num`[​](#running_update_compaction_task_num "Direct link to running_update_compaction_task_num") * Unit: Count * Description: Total number of currently running Primary Key table compaction tasks. --- ### Metrics s #### `schema_change_mem_bytes`[​](#schema_change_mem_bytes "Direct link to schema_change_mem_bytes") * Unit: Bytes * Description: Memory used for Schema Change. #### `segment_flush_queue_count`[​](#segment_flush_queue_count "Direct link to segment_flush_queue_count") * Unit: Count * Description: Number of queued tasks in the segment flush thread pool. #### `segment_metadata_mem_bytes`[​](#segment_metadata_mem_bytes "Direct link to segment_metadata_mem_bytes") * Unit: Bytes * Description: Memory used by segment metadata. #### `segment_read`[​](#segment_read "Direct link to segment_read") * Unit: Count * Description: Total number of segment reads. #### `segment_replicate_queue_count`[​](#segment_replicate_queue_count "Direct link to segment_replicate_queue_count") * Unit: Count * Description: Queued task count in the Segment Replicate thread pool. #### `segment_zonemap_mem_bytes`[​](#segment_zonemap_mem_bytes "Direct link to segment_zonemap_mem_bytes") * Unit: Bytes * Description: Memory used by segment zonemap. #### `short_key_index_mem_bytes`[​](#short_key_index_mem_bytes "Direct link to short_key_index_mem_bytes") * Unit: Bytes * Description: Memory used by the short key index. #### `small_file_cache_count`[​](#small_file_cache_count "Direct link to small_file_cache_count") * Unit: Count * Description: Number of small file caches. #### `snmp`[​](#snmp "Direct link to snmp") * Unit: - * Description: Metrics returned by `/proc/net/snmp`. #### `starrocks_be_clone_task_copy_bytes`[​](#starrocks_be_clone_task_copy_bytes "Direct link to starrocks_be_clone_task_copy_bytes") * Unit: Bytes * Type: Cumulative * Description: The total file size copied by Clone tasks in the BE node, including both INTER\_NODE and INTRA\_NODE types. #### `starrocks_be_clone_task_copy_duration_ms`[​](#starrocks_be_clone_task_copy_duration_ms "Direct link to starrocks_be_clone_task_copy_duration_ms") * Unit: ms * Type: Cumulative * Description: The total time for copy consumed by Clone tasks in the BE node, including both INTER\_NODE and INTRA\_NODE types. #### `starrocks_be_exec_state_report_active_threads`[​](#starrocks_be_exec_state_report_active_threads "Direct link to starrocks_be_exec_state_report_active_threads") * Unit: Count * Type: Instantaneous * Description: The number of tasks being executed in the thread pool that reports the execution status of the Fragment instance. #### `starrocks_be_exec_state_report_queue_count`[​](#starrocks_be_exec_state_report_queue_count "Direct link to starrocks_be_exec_state_report_queue_count") * Unit: Count * Type: Instantaneous * Description: The number of tasks queued in the thread pool that reports the execution status of the Fragment instance, up to a maximum of 1000. #### `starrocks_be_exec_state_report_running_threads`[​](#starrocks_be_exec_state_report_running_threads "Direct link to starrocks_be_exec_state_report_running_threads") * Unit: Count * Type: Instantaneous * Description: The number of threads in the thread pool that reports the execution status of the Fragment instance, with a minimum of 1 and a maximum of 2. #### `starrocks_be_exec_state_report_threadpool_size`[​](#starrocks_be_exec_state_report_threadpool_size "Direct link to starrocks_be_exec_state_report_threadpool_size") * Unit: Count * Type: Instantaneous * Description: The maximum number of threads in the thread pool that reports the execution status of the Fragment instance, defaults to 2. #### `starrocks_be_files_scan_num_bytes_read`[​](#starrocks_be_files_scan_num_bytes_read "Direct link to starrocks_be_files_scan_num_bytes_read") * Unit: Bytes * Description: Total bytes read from external storage. Labels: `file_format`, `scan_type`. #### `starrocks_be_files_scan_num_files_read`[​](#starrocks_be_files_scan_num_files_read "Direct link to starrocks_be_files_scan_num_files_read") * Unit: Count * Description: Number of files read from external storage (CSV, Parquet, ORC, JSON, Avro). Labels: `file_format`, `scan_type`. #### `starrocks_be_files_scan_num_raw_rows_read`[​](#starrocks_be_files_scan_num_raw_rows_read "Direct link to starrocks_be_files_scan_num_raw_rows_read") * Unit: Count * Description: Total raw rows read from external storage before format validation and predicate filtering. Labels: `file_format`, `scan_type`. #### `starrocks_be_files_scan_num_rows_return`[​](#starrocks_be_files_scan_num_rows_return "Direct link to starrocks_be_files_scan_num_rows_return") * Unit: Count * Description: Number of rows returned after predicate filtering. Labels: `file_format`, `scan_type`. #### `starrocks_be_files_scan_num_valid_rows_read`[​](#starrocks_be_files_scan_num_valid_rows_read "Direct link to starrocks_be_files_scan_num_valid_rows_read") * Unit: Count * Description: Number of valid rows read (excluding rows with invalid format). Labels: `file_format`, `scan_type`. #### `starrocks_be_mem_pool_mem_limit_bytes`[​](#starrocks_be_mem_pool_mem_limit_bytes "Direct link to starrocks_be_mem_pool_mem_limit_bytes") * Unit: Bytes * Type: Instantaneous * Description: Memory limit for each memory pool, measured in bytes. #### `starrocks_be_mem_pool_mem_usage_bytes`[​](#starrocks_be_mem_pool_mem_usage_bytes "Direct link to starrocks_be_mem_pool_mem_usage_bytes") * Unit: Bytes * Type: Instantaneous * Description: Currently used total memory by each memory pool, measured in bytes. #### `starrocks_be_mem_pool_mem_usage_ratio`[​](#starrocks_be_mem_pool_mem_usage_ratio "Direct link to starrocks_be_mem_pool_mem_usage_ratio") * Unit: - * Type: Instantaneous * Description: Ratio of the memory usage of the memory pool to the memory limit of the memory pool. #### `starrocks_be_mem_pool_workgroup_count`[​](#starrocks_be_mem_pool_workgroup_count "Direct link to starrocks_be_mem_pool_workgroup_count") * Unit: Count * Type: Instantaneous * Description: Number of resource groups assigned to each memory pool. #### `starrocks_be_pipe_prepare_pool_queue_len`[​](#starrocks_be_pipe_prepare_pool_queue_len "Direct link to starrocks_be_pipe_prepare_pool_queue_len") * Unit: Count * Type: Instantaneous * Description: Instantaneous value of pipeline prepare thread pool task queue length. #### `starrocks_be_priority_exec_state_report_active_threads`[​](#starrocks_be_priority_exec_state_report_active_threads "Direct link to starrocks_be_priority_exec_state_report_active_threads") * Unit: Count * Type: Instantaneous * Description: The number of tasks being executed in the thread pool that reports the final execution state of the Fragment instance. #### `starrocks_be_priority_exec_state_report_queue_count`[​](#starrocks_be_priority_exec_state_report_queue_count "Direct link to starrocks_be_priority_exec_state_report_queue_count") * Unit: Count * Type: Instantaneous * Description: The number of tasks queued in the thread pool that reports the final execution status of the Fragment instance, up to a maximum of 2147483647. #### `starrocks_be_priority_exec_state_report_running_threads`[​](#starrocks_be_priority_exec_state_report_running_threads "Direct link to starrocks_be_priority_exec_state_report_running_threads") * Unit: Count * Type: Instantaneous * Description: The number of threads in the thread pool that reports the final execution status of the Fragment instance, with a minimum of 1 and a maximum of 2. #### `starrocks_be_priority_exec_state_report_threadpool_size`[​](#starrocks_be_priority_exec_state_report_threadpool_size "Direct link to starrocks_be_priority_exec_state_report_threadpool_size") * Unit: Count * Type: Instantaneous * Description: The maximum number of threads in the thread pool that reports the final execution status of the Fragment instance, defaults to 2. #### `starrocks_be_resource_group_cpu_limit_ratio`[​](#starrocks_be_resource_group_cpu_limit_ratio "Direct link to starrocks_be_resource_group_cpu_limit_ratio") * Unit: - * Type: Instantaneous * Description: Instantaneous value of resource group cpu quota ratio. #### `starrocks_be_resource_group_cpu_use_ratio`[​](#starrocks_be_resource_group_cpu_use_ratio "Direct link to starrocks_be_resource_group_cpu_use_ratio") * Unit: - * Type: Average * Description: The ratio of CPU time used by the resource group to the CPU time of all resource groups. #### `starrocks_be_resource_group_mem_inuse_bytes`[​](#starrocks_be_resource_group_mem_inuse_bytes "Direct link to starrocks_be_resource_group_mem_inuse_bytes") * Unit: Bytes * Type: Instantaneous * Description: Instantaneous value of resource group memory usage. #### `starrocks_be_resource_group_mem_limit_bytes`[​](#starrocks_be_resource_group_mem_limit_bytes "Direct link to starrocks_be_resource_group_mem_limit_bytes") * Unit: Bytes * Type: Instantaneous * Description: Instantaneous value of resource group memory quota. #### `starrocks_be_segment_file_not_found_total`[​](#starrocks_be_segment_file_not_found_total "Direct link to starrocks_be_segment_file_not_found_total") * Unit: Count * Description: Total number of times a segment file was not found (file missing) during segment open. A continuously increasing value may indicate data loss or storage inconsistency. #### `starrocks_be_staros_shard_info_fallback_total`[​](#starrocks_be_staros_shard_info_fallback_total "Direct link to starrocks_be_staros_shard_info_fallback_total") * Unit: Count * Type: Cumulative * Description: Shared-data only. Total number of actual starmgr RPCs (`g_starlet->get_shard_info()`) that the BE's StarOSWorker had to issue because the requested shard info was not in the local cache (i.e. the FE had not pushed the shard to this BE before a query/compaction/lake operation referenced it). Only counted when the starlet readiness check passes and the RPC is actually dispatched; starlet-not-ready timeouts are not included. Should normally be near zero. A sustained or rising rate is a strong signal that FE-side task or node selection is scheduling work on a BE that does not yet have the shard, or that shard push propagation from FE is lagging. Recommended alert: high per-BE rate over a 5-minute window. #### `starrocks_be_staros_shard_info_fallback_failed_total`[​](#starrocks_be_staros_shard_info_fallback_failed_total "Direct link to starrocks_be_staros_shard_info_fallback_failed_total") * Unit: Count * Type: Cumulative * Description: Shared-data only. Subset of `starrocks_be_staros_shard_info_fallback_total` where the starmgr RPC returned a non-OK status. Use the ratio `failed_total / fallback_total` to alert on transient starmgr errors separately from routine successful fallbacks. #### `starrocks_be_staros_shard_count`[​](#starrocks_be_staros_shard_count "Direct link to starrocks_be_staros_shard_count") * Unit: Count * Type: Instantaneous * Description: Shared-data only. Number of shards currently assigned to this BE's StarOSWorker (size of the worker's local shard table). Updated synchronously inside `StarOSWorker::add_shard` and `StarOSWorker::remove_shard` (push-on-mutation), so the value reflects the last shard table mutation rather than being recomputed at scrape time. The gauge is not reset on BE shutdown and will retain its last value until the next mutation. Use it to observe shard distribution balance across BEs and to detect drift from the FE-side placement. #### `starrocks_fe_alter_duration_ms`[​](#starrocks_fe_alter_duration_ms "Direct link to starrocks_fe_alter_duration_ms") * Unit: ms * Type: Summary * Labels: `execution_mode` (`fse`, `legacy_fse`, or `rewrite`), `is_leader` * Description: Time in milliseconds to apply an ALTER TABLE change, per statement. Reported only by the Leader FE (`is_leader="true"`). Includes the 0.75/0.95/0.98/0.99/0.999 quantiles plus `_sum` and `_count`. The `execution_mode` label indicates how the change was applied: * `fse`: the current Fast Schema Evolution (FSE), applied immediately while the statement runs. * `legacy_fse`: the legacy FSE path, which runs in the background and is usually far slower. Occurs only in shared-data clusters with `cloud_native_fast_schema_evolution_v2` disabled (enabled by default, which uses `fse`). * `rewrite`: the change had to physically rewrite the table data, which also runs in the background. #### `starrocks_fe_alter_operation_total`[​](#starrocks_fe_alter_operation_total "Direct link to starrocks_fe_alter_operation_total") * Unit: Count * Type: Cumulative * Labels: `type` (`add_column`, `drop_column`, or `modify_column`), `is_leader` * Description: Number of ALTER TABLE column operations, by type. A single statement can contain several operations — for example, `ADD COLUMN a, DROP COLUMN b` — and each is counted separately under its type. Renames, reorders, and comment-only changes are not counted. Reported only by the Leader FE (`is_leader="true"`). #### `starrocks_fe_clone_task_copy_bytes`[​](#starrocks_fe_clone_task_copy_bytes "Direct link to starrocks_fe_clone_task_copy_bytes") * Unit: Bytes * Type: Cumulative * Description: The total file size copied by Clone tasks in the cluster, including both INTER\_NODE and INTRA\_NODE types. #### `starrocks_fe_clone_task_copy_duration_ms`[​](#starrocks_fe_clone_task_copy_duration_ms "Direct link to starrocks_fe_clone_task_copy_duration_ms") * Unit: ms * Type: Cumulative * Description: The total time for copy consumed by Clone tasks in the cluster, including both INTER\_NODE and INTRA\_NODE types. #### `starrocks_fe_clone_task_success`[​](#starrocks_fe_clone_task_success "Direct link to starrocks_fe_clone_task_success") * Unit: Count * Type: Cumulative * Description: The number of successfully executed Clone tasks in the cluster. #### `starrocks_fe_clone_task_total`[​](#starrocks_fe_clone_task_total "Direct link to starrocks_fe_clone_task_total") * Unit: Count * Type: Cumulative * Description: The total number of Clone tasks in the cluster. #### `starrocks_fe_last_finished_job_timestamp`[​](#starrocks_fe_last_finished_job_timestamp "Direct link to starrocks_fe_last_finished_job_timestamp") * Unit: ms * Type: Instantaneous * Description: Indicates the end time of the last query or loading under the specific warehouse. For a shared-nothing cluster, this item only monitors the default warehouse. #### `starrocks_fe_memory_usage`[​](#starrocks_fe_memory_usage "Direct link to starrocks_fe_memory_usage") * Unit: Bytes or Count * Type: Instantaneous * Description: Indicates the memory statistics for various modules under the specific warehouse. For a shared-nothing cluster, this item only monitors the default warehouse. #### `starrocks_fe_meta_log_count`[​](#starrocks_fe_meta_log_count "Direct link to starrocks_fe_meta_log_count") * Unit: Count * Type: Instantaneous * Description: The number of Edit Logs without a checkpoint. A value within `100000` is considered reasonable.. #### `starrocks_fe_publish_version_daemon_loop_total`[​](#starrocks_fe_publish_version_daemon_loop_total "Direct link to starrocks_fe_publish_version_daemon_loop_total") * Unit: Count * Type: Cumulative * Description: Total number of `publish-version-daemon` loop runs on this FE node. The following metrics are `summary`-type metrics that provide latency distributions for different phases of a transaction. These metrics are reported exclusively by the Leader FE node. Each metric includes the following outputs: * **Quantiles**: Latency values at different percentile boundaries. These are exposed via the `quantile` label, which can have values of `0.75`, `0.95`, `0.98`, `0.99`, and `0.999`. * **`_sum`**: The total cumulative time spent in this phase, for example, `starrocks_fe_txn_total_latency_ms_sum`. * **`_count`**: The total number of transactions recorded for this phase, for example, `starrocks_fe_txn_total_latency_ms_count`. All transaction metrics share the following labels: * `type`: Categorizes transactions by their load job source type (for example, `all`, `stream_load`, `routine_load`). This allows for monitoring both overall transaction performance and the performance of specific load types. The reported groups can be configured via the FE parameter [`txn_latency_metric_report_groups`](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#txn_latency_metric_report_groups). * `is_leader`: Indicates whether the reporting FE node is the Leader. Only the Leader FE (`is_leader="true"`) reports actual metric values. Followers will have `is_leader="false"` and report no data. #### `starrocks_fe_query_resource_group`[​](#starrocks_fe_query_resource_group "Direct link to starrocks_fe_query_resource_group") * Unit: Count * Type: Cumulative * Description: Indicates the total number of queries executed under the specific resource group. #### `starrocks_fe_query_resource_group`[​](#starrocks_fe_query_resource_group-1 "Direct link to starrocks_fe_query_resource_group-1") * Unit: Count * Type: Cumulative * Description: The number of queries for each resource group. #### `starrocks_fe_query_resource_group_err`[​](#starrocks_fe_query_resource_group_err "Direct link to starrocks_fe_query_resource_group_err") * Unit: Count * Type: Cumulative * Description: Indicates the number of failed queries under the specific resource group. #### `starrocks_fe_query_resource_group_err`[​](#starrocks_fe_query_resource_group_err-1 "Direct link to starrocks_fe_query_resource_group_err-1") * Unit: Count * Type: Cumulative * Description: The number of incorrect queries for each resource group. #### `starrocks_fe_query_resource_group_latency`[​](#starrocks_fe_query_resource_group_latency "Direct link to starrocks_fe_query_resource_group_latency") * Unit: ms * Type: Cumulative * Description: Indicates the latency statistics for queries under the specific resource group. #### `starrocks_fe_query_resource_group_latency`[​](#starrocks_fe_query_resource_group_latency-1 "Direct link to starrocks_fe_query_resource_group_latency-1") * Unit: Seconds * Type: Average * Description: the query latency percentile for each resource group. #### `starrocks_fe_routine_load_error_rows`[​](#starrocks_fe_routine_load_error_rows "Direct link to starrocks_fe_routine_load_error_rows") * Unit: Count * Description: The total number of error rows encountered during data loading by all Routine Load jobs. #### `starrocks_fe_routine_load_jobs`[​](#starrocks_fe_routine_load_jobs "Direct link to starrocks_fe_routine_load_jobs") * Unit: Count * Description: The total number of Routine Load jobs in different states. For example: ```plaintext starrocks_fe_routine_load_jobs{state="NEED_SCHEDULE"} 0 starrocks_fe_routine_load_jobs{state="RUNNING"} 1 starrocks_fe_routine_load_jobs{state="PAUSED"} 0 starrocks_fe_routine_load_jobs{state="STOPPED"} 0 starrocks_fe_routine_load_jobs{state="CANCELLED"} 1 starrocks_fe_routine_load_jobs{state="UNSTABLE"} 0 ``` #### `starrocks_fe_routine_load_max_lag_of_partition`[​](#starrocks_fe_routine_load_max_lag_of_partition "Direct link to starrocks_fe_routine_load_max_lag_of_partition") * Unit: - * Description: The maximum Kafka partition offset lag for each Routine Load job. It is collected only when the FE configuration `enable_routine_load_lag_metrics` is set to `true` and the offset lag is greater than or equal to the FE configuration `min_routine_load_lag_for_metrics`. By default, `enable_routine_load_lag_metrics` is `false`, and `min_routine_load_lag_for_metrics` is `10000`. #### `starrocks_fe_routine_load_max_lag_time_of_partition`[​](#starrocks_fe_routine_load_max_lag_time_of_partition "Direct link to starrocks_fe_routine_load_max_lag_time_of_partition") * Unit: Seconds * Description: The maximum Kafka partition offset timestamp lag for each Routine Load job. It is collected only when the FE configuration `enable_routine_load_lag_time_metrics` is set to `true`. By default, `enable_routine_load_lag_time_metrics` is `false`. #### `starrocks_fe_routine_load_paused`[​](#starrocks_fe_routine_load_paused "Direct link to starrocks_fe_routine_load_paused") * Unit: Count * Description: The total number of times Routine Load jobs are paused. #### `starrocks_fe_routine_load_receive_bytes`[​](#starrocks_fe_routine_load_receive_bytes "Direct link to starrocks_fe_routine_load_receive_bytes") * Unit: Byte * Description: The total amount of data loaded by all Routine Load jobs. #### `starrocks_fe_routine_load_rows`[​](#starrocks_fe_routine_load_rows "Direct link to starrocks_fe_routine_load_rows") * Unit: Count * Description: The total number of rows loaded by all Routine Load jobs. #### `starrocks_fe_safe_mode`[​](#starrocks_fe_safe_mode "Direct link to starrocks_fe_safe_mode") * Unit: - * Type: Instantaneous * Description: Indicates whether Safe Mode is enabled. Valid values: `0` (disabled) and `1` (enabled). When Safe Mode is enabled, the cluster no longer accepts any loading requests. #### `starrocks_fe_scheduled_pending_tablet_num`[​](#starrocks_fe_scheduled_pending_tablet_num "Direct link to starrocks_fe_scheduled_pending_tablet_num") * Unit: Count * Type: Instantaneous * Description: The number of Clone tasks in Pending state FE scheduled, including both BALANCE and REPAIR types. #### `starrocks_fe_scheduled_running_tablet_num`[​](#starrocks_fe_scheduled_running_tablet_num "Direct link to starrocks_fe_scheduled_running_tablet_num") * Unit: Count * Type: Instantaneous * Description: The number of Clone tasks in Running state FE scheduled, including both BALANCE and REPAIR types. #### `starrocks_fe_slow_lock_held_time_ms`[​](#starrocks_fe_slow_lock_held_time_ms "Direct link to starrocks_fe_slow_lock_held_time_ms") * Unit: ms * Type: Summary * Description: Histogram tracking the lock held time (in milliseconds) when slow locks are detected. This metric is updated when lock wait time exceeds the `slow_lock_threshold_ms` configuration parameter. It tracks the maximum lock held time among all lock owners when a slow lock event is detected. Each metric includes quantile values (0.75, 0.95, 0.98, 0.99, 0.999), `_sum`, and `_count` outputs. Note: This metric may not accurately reflect the exact lock held time under high contention, because the metric is updated once the wait time exceeds the threshold, but the held time may continue to increase until the owner completes its operation and releases the lock. However, this metric can still be updated even when deadlock occurs. #### `starrocks_fe_slow_lock_wait_time_ms`[​](#starrocks_fe_slow_lock_wait_time_ms "Direct link to starrocks_fe_slow_lock_wait_time_ms") * Unit: ms * Type: Summary * Description: Histogram tracking the lock wait time (in milliseconds) when slow locks are detected. This metric is updated when lock wait time exceeds the `slow_lock_threshold_ms` configuration parameter. It accurately tracks how long threads wait to acquire locks during lock contention scenarios. Each metric includes quantile values (0.75, 0.95, 0.98, 0.99, 0.999), `_sum`, and `_count` outputs. This metric provides precise wait time measurements. Note: This metric cannot be updated when deadlock occurs, hence it cannot be used to detect deadlock situations. #### `starrocks_fe_sql_block_hit_count`[​](#starrocks_fe_sql_block_hit_count "Direct link to starrocks_fe_sql_block_hit_count") * Unit: Count * Description: The number of times blacklisted SQL has been intercepted. #### `starrocks_fe_tablet_pre_split_eligibility_skipped`[​](#starrocks_fe_tablet_pre_split_eligibility_skipped "Direct link to starrocks_fe_tablet_pre_split_eligibility_skipped") * Unit: Count * Type: Cumulative * Labels: `reason` — the SkipReason enum value (lower-cased). Per-load values: `not_range_distribution`, `table_not_normal`, `has_materialized_view_or_rollup`, `unsupported_sort_key`, `metadata_not_resolved`, `multiple_base_index_tablets`, `partition_not_empty`, `disabled_by_config`, `disabled_by_session`. Multi-partition (P2-a) per-partition values: `unsupported_partition_column_type` (partition source column type cannot be projected, e.g. STRUCT/ARRAY), `invalid_partition_value` (sampled partition cell can't be formatted into an `AddPartitionClause`, e.g. null in a non-nullable column or unparseable date), `grouper_empty` (every sample row was dropped by the formatter/analyzer), `stale_catalog_state` (partition was seen by the grouper but disappeared before the coordinator re-resolved it under READ lock — concurrent partition drop/replace), `partition_not_eligible_post_create` (the post-pre-create eligibility re-check failed, typically because the partition is non-empty or now has multiple tablets). * Description: Total Sample-Based Tablet Pre-Split invocations that the FE-side eligibility gate declined before any sampler ran, broken down by the specific reason. Operators can use this counter to attribute "pre-split not running" to a single eligibility branch at a glance. In the multi-partition (P2-a) path the same counter also records per-partition skip reasons emitted by the grouper and the per-partition re-resolve. #### `starrocks_fe_tablet_pre_split_sampler_invocations`[​](#starrocks_fe_tablet_pre_split_sampler_invocations "Direct link to starrocks_fe_tablet_pre_split_sampler_invocations") * Unit: Count * Type: Cumulative * Description: Total sampler invocations driven by Sample-Based Tablet Pre-Split. Incremented once per eligible invocation when the production sampler pipeline starts a sample. #### `starrocks_fe_tablet_pre_split_sampler_failed`[​](#starrocks_fe_tablet_pre_split_sampler_failed "Direct link to starrocks_fe_tablet_pre_split_sampler_failed") * Unit: Count * Type: Cumulative * Labels: `reason` — the post-eligibility failure category (lower-cased SkipReason), one of `sample_failed` (sampler executor threw), `timeout_pre_submit` (sample + plan + build phase exceeded `tablet_pre_split_pre_submit_timeout_seconds`), `submit_failed` (`TabletReshardJobMgr` rejected admission), `pre_create_failed` (multi-partition path: `LocalMetastore.addPartitions` threw while pre-creating a target partition — that one partition is dropped from the combined submit and falls back to BE runtime auto-create; sibling partitions in the same load continue). * Description: Total times the sampler attempted but did not produce an admitted reshard job, broken down by reason. Distinct from `tablet_pre_split_eligibility_skipped` (sampler never ran) and from `tablet_pre_split_tier_used` (which records the tier that succeeded). Meta-tier → data-tier fallback alone is not a failure; it is tracked via `tablet_pre_split_tier_used{tier=data_tier}`. #### `starrocks_fe_tablet_pre_split_tier_used`[​](#starrocks_fe_tablet_pre_split_tier_used "Direct link to starrocks_fe_tablet_pre_split_tier_used") * Unit: Count * Type: Cumulative * Labels: `tier` — `meta_tier` (boundaries computed from Parquet/ORC row-group statistics; no row data read) or `data_tier` (boundaries computed from actual row samples collected via a FILES sub-query — covers both direct data-tier invocations and meta-tier → data-tier fallbacks). * Description: Total Sample-Based Tablet Pre-Split invocations by which sampler tier produced the boundaries. #### `starrocks_fe_tablet_pre_split_boundaries_planned`[​](#starrocks_fe_tablet_pre_split_boundaries_planned "Direct link to starrocks_fe_tablet_pre_split_boundaries_planned") * Unit: Count * Type: Histogram * Description: Number of boundary tuples produced by the planner per invocation. Equals `effectiveTabletCount - 1` (a K-tablet split needs K-1 cut points). #### `starrocks_fe_tablet_pre_split_partitions_total`[​](#starrocks_fe_tablet_pre_split_partitions_total "Direct link to starrocks_fe_tablet_pre_split_partitions_total") * Unit: Count * Type: Cumulative * Description: Multi-partition (P2-a) counter. Total predicted target partitions counted by the Sample-Based Tablet Pre-Split coordinator — one increment per `PartitionSamples` entry that survived the grouper. Combined with `tablet_pre_split_partitions_capped` and the `tablet_pre_split_pre_create{result=...}` family this tells operators how many partitions each multi-partition invocation actually acts on. Stays at zero for the single-partition path. #### `starrocks_fe_tablet_pre_split_partitions_capped`[​](#starrocks_fe_tablet_pre_split_partitions_capped "Direct link to starrocks_fe_tablet_pre_split_partitions_capped") * Unit: Count * Type: Cumulative * Description: Multi-partition (P2-a) counter. Number of predicted target partitions the grouper dropped because the per-load count exceeded `tablet_pre_split_max_partitions_per_load`. The grouper keeps the partitions with the highest sample counts and drops the lowest-count tail; dropped partitions fall back to BE runtime auto-create with no pre-split. Sustained non-zero values mean the cap is biting — consider raising `tablet_pre_split_max_partitions_per_load` or reducing partition cardinality on the load. #### `starrocks_fe_tablet_pre_split_pre_create`[​](#starrocks_fe_tablet_pre_split_pre_create "Direct link to starrocks_fe_tablet_pre_split_pre_create") * Unit: Count * Type: Cumulative * Labels: `result` — `succeeded` (`LocalMetastore.addPartitions` returned normally — the partition was created or silently deduped), `failed` (`addPartitions` threw, e.g. concurrent ALTER or journal failure; the affected partition falls back to BE runtime auto-create and is also recorded under `tablet_pre_split_sampler_failed{reason=pre_create_failed}`), `already_exists` (the partition was found in the catalog at pre-create time — concurrent loader race; the coordinator reuses the existing partition). * Description: Multi-partition (P2-a) counter. Number of partition pre-create attempts the coordinator issued via `LocalMetastore.addPartitions`, broken down by outcome. Total attempts = sum of all three labels. Stays at zero for the single-partition path. #### `starrocks_fe_tablet_pre_split_pre_submit_wait_ms`[​](#starrocks_fe_tablet_pre_split_pre_submit_wait_ms "Direct link to starrocks_fe_tablet_pre_split_pre_submit_wait_ms") * Unit: ms * Type: Histogram * Description: Wall-clock time spent in the pre-submit phase of Sample-Based Tablet Pre-Split (sample + plan + build reshard job). Capped by `tablet_pre_split_pre_submit_timeout_seconds`. #### `starrocks_fe_tablet_pre_split_post_submit_wait_ms`[​](#starrocks_fe_tablet_pre_split_post_submit_wait_ms "Direct link to starrocks_fe_tablet_pre_split_post_submit_wait_ms") * Unit: ms * Type: Histogram * Description: Wall-clock time the coordinator spent awaiting `FINISHED` on the admitted Sample-Based Tablet Pre-Split reshard job. Fires on all production load kinds — INSERT-from-FILES and INSERT-from-table (both via `InsertPreSplitHook`, called from `StmtExecutor` before `StatementPlanner.plan` opens the load txn) and Broker Load (via `BrokerLoadPreSplitHook`, called from `BrokerLoadJob.createLoadingTask` before `beginTxn` opens `T_load`), all sync-awaiting through the shared `PreSplitFlow` — and on the optional `runPreSplit` synchronous-await wrapper used by tests. In all cases the trigger load itself plans against the post-split layout. #### `starrocks_fe_tablet_pre_split_post_submit_hard_cap`[​](#starrocks_fe_tablet_pre_split_post_submit_hard_cap "Direct link to starrocks_fe_tablet_pre_split_post_submit_hard_cap") * Unit: Count * Type: Cumulative * Description: Total Sample-Based Tablet Pre-Split post-submit hard-cap events. Incremented when the admitted reshard job did not reach `FINISHED` within `tablet_pre_split_post_submit_wait_seconds`. Fires on every production load kind on timeout — INSERT-from-FILES, INSERT-from-table, and Broker Load (all sync-await through the shared `PreSplitFlow`) — as well as the `runPreSplit` synchronous-await wrapper. The load then proceeds without abort against the currently visible tablet layout (still the original layout if the daemon hasn't transitioned, or partially / fully post-split if the daemon raced past the wait); `tablet_pre_split_load_abort` is NOT incremented because the load itself is not aborted. #### `starrocks_fe_tablet_pre_split_load_abort`[​](#starrocks_fe_tablet_pre_split_load_abort "Direct link to starrocks_fe_tablet_pre_split_load_abort") * Unit: Count * Type: Cumulative * Description: Total load transactions aborted because Sample-Based Tablet Pre-Split could not confirm the admitted reshard job reached `FINISHED` in time. Sibling counter of `tablet_pre_split_post_submit_hard_cap`. Production load paths proceed without abort against the currently visible layout on post-submit timeout rather than abort, so this counter stays at zero in production today; it only fires when a caller uses the strict `runPreSplit` wrapper (tests, or a future caller that opts into abort-on-timeout). #### `starrocks_fe_tablet_max_compaction_score`[​](#starrocks_fe_tablet_max_compaction_score "Direct link to starrocks_fe_tablet_max_compaction_score") * Unit: Count * Type: Instantaneous * Description: Indicates the highest Compaction Score on each BE node. #### `starrocks_fe_tablet_num`[​](#starrocks_fe_tablet_num "Direct link to starrocks_fe_tablet_num") * Unit: Count * Type: Instantaneous * Description: Indicates the number of tablets on each BE node. #### `starrocks_fe_txn_max_committed_pending_publish_ms`[​](#starrocks_fe_txn_max_committed_pending_publish_ms "Direct link to starrocks_fe_txn_max_committed_pending_publish_ms") * Unit: ms * Type: Instantaneous * Description: The longest time, in milliseconds, that a transaction is currently sitting in the `COMMITTED` status pending publish to `VISIBLE`, that is, the age of the oldest committed-but-not-yet-published transaction. Unlike `starrocks_fe_txn_publish_*` metrics, which are summaries recorded after a transaction finishes, this is a live gauge of the worst-case in-flight wait. The value is reported per database via the `db` label and only by the Leader FE node (`is_leader="true"`). It returns `0` when no committed transaction is pending publish. A high or continuously growing value indicates that version publishing is stuck or lagging behind commits. #### `starrocks_fe_txn_publish_ack_latency_ms`[​](#starrocks_fe_txn_publish_ack_latency_ms "Direct link to starrocks_fe_txn_publish_ack_latency_ms") * Unit: ms * Type: Summary * Description: The final acknowledgment latency, from `ready-to-finish` time to the final `finish` time when the transaction is marked as `VISIBLE`. This metric includes final acknowledgment steps after the transaction is ready to finish. #### `starrocks_fe_txn_publish_can_finish_latency_ms`[​](#starrocks_fe_txn_publish_can_finish_latency_ms "Direct link to starrocks_fe_txn_publish_can_finish_latency_ms") * Unit: ms * Type: Summary * Description: The latency from `publish` task completion to the moment `canTxnFinish()` first returns true, measured from `publish version finish` time to `ready-to-finish` time. #### `starrocks_fe_txn_publish_execute_latency_ms`[​](#starrocks_fe_txn_publish_execute_latency_ms "Direct link to starrocks_fe_txn_publish_execute_latency_ms") * Unit: ms * Type: Summary * Description: The active execution time of the `publish` task, from when the task is picked up to when it finishes. This metric represents the actual time being spent to make the transaction's changes visible. #### `starrocks_fe_txn_publish_latency_ms`[​](#starrocks_fe_txn_publish_latency_ms "Direct link to starrocks_fe_txn_publish_latency_ms") * Unit: ms * Type: Summary * Description: The latency of the `publish` phase, from `commit` time to `finish` time. This is the duration it takes for a committed transaction to become visible to queries. It is the sum of the `schedule`, `execute`, `can_finish`, and `ack` sub-phases. #### `starrocks_fe_txn_publish_schedule_latency_ms`[​](#starrocks_fe_txn_publish_schedule_latency_ms "Direct link to starrocks_fe_txn_publish_schedule_latency_ms") * Unit: ms * Type: Summary * Description: The time a transaction spends waiting to be published after it has been committed, measured from `commit` time to when the publish task is picked up. This metric reflects scheduling delays or queueing time in the `publish` pipeline. #### `starrocks_fe_txn_total_latency_ms`[​](#starrocks_fe_txn_total_latency_ms "Direct link to starrocks_fe_txn_total_latency_ms") * Unit: ms * Type: Summary * Description: The total latency for a transaction to complete, measured from the `prepare` time to the `finish` time. This metric represents the full end-to-end duration of a transaction. #### `starrocks_fe_txn_write_latency_ms`[​](#starrocks_fe_txn_write_latency_ms "Direct link to starrocks_fe_txn_write_latency_ms") * Unit: ms * Type: Summary * Description: The latency of the `write` phase of a transaction, from `prepare` time to `commit` time. This metric isolates the performance of the data writing and preparation stage before the transaction is ready to be published. #### `starrocks_fe_unfinished_backup_job`[​](#starrocks_fe_unfinished_backup_job "Direct link to starrocks_fe_unfinished_backup_job") * Unit: Count * Type: Instantaneous * Description: Indicates the number of running BACKUP tasks under the specific warehouse. For a shared-nothing cluster, this item only monitors the default warehouse. For a shared-data cluster, this value is always `0`. #### `starrocks_fe_unfinished_query`[​](#starrocks_fe_unfinished_query "Direct link to starrocks_fe_unfinished_query") * Unit: Count * Type: Instantaneous * Description: Indicates the number of queries currently running under the specific warehouse. For a shared-nothing cluster, this item only monitors the default warehouse. #### `starrocks_fe_unfinished_restore_job`[​](#starrocks_fe_unfinished_restore_job "Direct link to starrocks_fe_unfinished_restore_job") * Unit: Count * Type: Instantaneous * Description: Indicates the number of running RESTORE tasks under the specific warehouse. For a shared-nothing cluster, this item only monitors the default warehouse. For a shared-data cluster, this value is always `0`. #### `storage_page_cache_mem_bytes`[​](#storage_page_cache_mem_bytes "Direct link to storage_page_cache_mem_bytes") * Unit: Bytes * Description: Memory used by storage page cache. #### `stream_load`[​](#stream_load "Direct link to stream_load") * Unit: - * Description: Total loaded rows and received bytes. #### `stream_load_pipe_count`[​](#stream_load_pipe_count "Direct link to stream_load_pipe_count") * Unit: Count * Description: Number of currently running Stream Load tasks. #### `streaming_load_bytes`[​](#streaming_load_bytes "Direct link to streaming_load_bytes") * Unit: Bytes * Description: Total bytes loaded by Stream Load. #### `streaming_load_current_processing`[​](#streaming_load_current_processing "Direct link to streaming_load_current_processing") * Unit: Count * Description: Number of currently running Stream Load tasks. #### `streaming_load_duration_ms`[​](#streaming_load_duration_ms "Direct link to streaming_load_duration_ms") * Unit: ms * Description: Total time spent on Stream Load. #### `streaming_load_requests_total`[​](#streaming_load_requests_total "Direct link to streaming_load_requests_total") * Unit: Count * Description: Total number of Stream Load requests. ###### SPLIT[​](#split "Direct link to SPLIT") --- ### Metrics t through z #### `tablet_base_max_compaction_score`[​](#tablet_base_max_compaction_score "Direct link to tablet_base_max_compaction_score") * Unit: - * Description: Highest base compaction score of tablets in this BE. #### `tablet_cumulative_max_compaction_score`[​](#tablet_cumulative_max_compaction_score "Direct link to tablet_cumulative_max_compaction_score") * Unit: - * Description: Highest cumulative compaction score of tablets in this BE. #### `tablet_metadata_mem_bytes`[​](#tablet_metadata_mem_bytes "Direct link to tablet_metadata_mem_bytes") * Unit: Bytes * Description: Memory used by tablet metadata. #### `tablet_schema_mem_bytes`[​](#tablet_schema_mem_bytes "Direct link to tablet_schema_mem_bytes") * Unit: Bytes * Description: Memory used by tablet schema. #### `tablet_update_max_compaction_score`[​](#tablet_update_max_compaction_score "Direct link to tablet_update_max_compaction_score") * Unit: - * Description: Highest compaction score of tablets in Primary Key tables in the current BE. #### `threadpool_task_exception_total`[​](#threadpool_task_exception_total "Direct link to threadpool_task_exception_total") * Unit: Count * Description: Cumulative number of task exceptions caught and swallowed by ThreadPool worker threads across the BE process. Increments only when [`enable_threadpool_catch_task_exception`](https://docs.starrocks.io/docs/administration/management/BE_parameters/log_server_meta.md#enable_threadpool_catch_task_exception) is `true`. When that item is `false` (default), this metric stays unchanged because there is no enclosing catch clause. Use it to alert on swallowed failures while catch mode is enabled; pool name and exception detail remain in the BE ERROR logs. #### `thrift_connections_total`[​](#thrift_connections_total "Direct link to thrift_connections_total") * Unit: Count * Description: Total number of thrift connections (including finished connections). #### `thrift_current_connections (Deprecated)`[​](#thrift_current_connections-deprecated "Direct link to thrift_current_connections-deprecated") #### `thrift_opened_clients`[​](#thrift_opened_clients "Direct link to thrift_opened_clients") * Unit: Count * Description: Number of currently opened thrift clients. #### `thrift_used_clients`[​](#thrift_used_clients "Direct link to thrift_used_clients") * Unit: Count * Description: Number of thrift clients in use currently. #### `total_column_pool_bytes (Deprecated)`[​](#total_column_pool_bytes-deprecated "Direct link to total_column_pool_bytes-deprecated") #### `transaction_streaming_load_bytes`[​](#transaction_streaming_load_bytes "Direct link to transaction_streaming_load_bytes") * Unit: Bytes * Description: Total loading bytes of transaction load. #### `transaction_streaming_load_current_processing`[​](#transaction_streaming_load_current_processing "Direct link to transaction_streaming_load_current_processing") * Unit: Count * Description: Number of currently running transactional Stream Load tasks. #### `transaction_streaming_load_duration_ms`[​](#transaction_streaming_load_duration_ms "Direct link to transaction_streaming_load_duration_ms") * Unit: ms * Description: Total time spent on Stream Load transaction Interface. #### `transaction_streaming_load_requests_total`[​](#transaction_streaming_load_requests_total "Direct link to transaction_streaming_load_requests_total") * Unit: Count * Description: Total number of transaction load requests. #### `txn_request`[​](#txn_request "Direct link to txn_request") * Unit: - * Description: Transaction requests of BEGIN, COMMIT, ROLLBACK, and EXEC. #### `uint8_column_pool_bytes`[​](#uint8_column_pool_bytes "Direct link to uint8_column_pool_bytes") * Unit: Bytes * Description: Bytes used by the UINT8 column pool. #### `unused_rowsets_count`[​](#unused_rowsets_count "Direct link to unused_rowsets_count") * Unit: Count * Description: Total number of unused rowsets. Please note that these rowsets will be reclaimed later. #### `update_apply_queue_count`[​](#update_apply_queue_count "Direct link to update_apply_queue_count") * Unit: Count * Description: Queued task count in the Primary Key table transaction APPLY thread pool. #### `update_compaction_duration_us`[​](#update_compaction_duration_us "Direct link to update_compaction_duration_us") * Unit: us * Description: Total time spent on Primary Key table compactions. #### `update_compaction_outputs_bytes_total`[​](#update_compaction_outputs_bytes_total "Direct link to update_compaction_outputs_bytes_total") * Unit: Bytes * Description: Total bytes written by Primary Key table compactions. #### `update_compaction_outputs_total`[​](#update_compaction_outputs_total "Direct link to update_compaction_outputs_total") * Unit: Count * Description: Total number of Primary Key table compactions. #### `update_compaction_task_byte_per_second`[​](#update_compaction_task_byte_per_second "Direct link to update_compaction_task_byte_per_second") * Unit: Bytes/s * Description: Estimated rate of Primary Key table compactions. #### `update_compaction_task_cost_time_ns`[​](#update_compaction_task_cost_time_ns "Direct link to update_compaction_task_cost_time_ns") * Unit: ns * Description: Total time spent on the Primary Key table compactions. #### `update_del_vector_bytes_total`[​](#update_del_vector_bytes_total "Direct link to update_del_vector_bytes_total") * Unit: Bytes * Description: Total memory used for caching DELETE vectors in Primary Key tables. #### `update_del_vector_deletes_new`[​](#update_del_vector_deletes_new "Direct link to update_del_vector_deletes_new") * Unit: Count * Description: Total number of newly generated DELETE vectors used in Primary Key tables. #### `update_del_vector_deletes_total (Deprecated)`[​](#update_del_vector_deletes_total-deprecated "Direct link to update_del_vector_deletes_total-deprecated") #### `update_del_vector_dels_num (Deprecated)`[​](#update_del_vector_dels_num-deprecated "Direct link to update_del_vector_dels_num-deprecated") #### `update_del_vector_num`[​](#update_del_vector_num "Direct link to update_del_vector_num") * Unit: Count * Description: Number of the DELETE vector cache items in Primary Key tables. #### `update_mem_bytes`[​](#update_mem_bytes "Direct link to update_mem_bytes") * Unit: Bytes * Description: Memory used by Primary Key table APPLY tasks and Primary Key index. #### `update_primary_index_bytes_total`[​](#update_primary_index_bytes_total "Direct link to update_primary_index_bytes_total") * Unit: Bytes * Description: Total memory cost of the Primary Key index. #### `update_primary_index_num`[​](#update_primary_index_num "Direct link to update_primary_index_num") * Unit: Count * Description: Number of Primary Key indexes cached in memory. #### `update_rowset_commit_apply_duration_us`[​](#update_rowset_commit_apply_duration_us "Direct link to update_rowset_commit_apply_duration_us") * Unit: us * Description: Total time spent on Primary Key table APPLY tasks. #### `update_rowset_commit_apply_total`[​](#update_rowset_commit_apply_total "Direct link to update_rowset_commit_apply_total") * Unit: Count * Description: Total number of COMMIT and APPLY for Primary Key tables. #### `update_rowset_commit_request_failed`[​](#update_rowset_commit_request_failed "Direct link to update_rowset_commit_request_failed") * Unit: Count * Description: Total number of failed rowset COMMIT requests in Primary Key tables. #### `update_rowset_commit_request_total`[​](#update_rowset_commit_request_total "Direct link to update_rowset_commit_request_total") * Unit: Count * Description: Total number of rowset COMMIT requests in Primary Key tables. #### `wait_base_compaction_task_num`[​](#wait_base_compaction_task_num "Direct link to wait_base_compaction_task_num") * Unit: Count * Description: Number of base compaction tasks waiting for execution. #### `wait_cumulative_compaction_task_num`[​](#wait_cumulative_compaction_task_num "Direct link to wait_cumulative_compaction_task_num") * Unit: Count * Description: Number of cumulative compaction tasks waiting for execution. #### `writable_blocks_total (Deprecated)`[​](#writable_blocks_total-deprecated "Direct link to writable_blocks_total-deprecated") --- ### General Monitoring Metrics note Metrics for materialized views and shared-data clusters are detailed in the corresponding sections: * [Metrics for asynchronous materialized view metrics](https://docs.starrocks.io/docs/administration/management/monitoring/metrics-materialized_view.md) * [Metrics for Shared-data Dashboard metrics, and Starlet Dashboard metrics](https://docs.starrocks.io/docs/administration/management/monitoring/metrics-shared-data.md) For more information on how to build a monitoring service for your StarRocks cluster, see [Monitor and Alert](https://docs.starrocks.io/docs/administration/management/monitoring/Monitor_and_Alert.md). Monitoring metrics are listed alphabetically in these files: * [a - c](https://docs.starrocks.io/docs/administration/management/monitoring/metric_details/a-c.md) * [d - h](https://docs.starrocks.io/docs/administration/management/monitoring/metric_details/d-h.md) * [i - p](https://docs.starrocks.io/docs/administration/management/monitoring/metric_details/i-p.md) * [q - r](https://docs.starrocks.io/docs/administration/management/monitoring/metric_details/q-r.md) * [s](https://docs.starrocks.io/docs/administration/management/monitoring/metric_details/s.md) * [t - z](https://docs.starrocks.io/docs/administration/management/monitoring/metric_details/t-z.md) --- ### Monitoring Metrics for Asynchronous Materialized Views From v3.1 onwards, StarRocks supports metrics for asynchronous materialized views. To allow Prometheus to access the materialized view metadata in your cluster, you must add the following configurations in the Prometheus configuration file **prometheus/prometheus.yml**: ```yaml global: .... scrape_configs: - job_name: 'dev' metrics_path: '/metrics' # Add the following configurations. basic_auth: username: 'root' password: '' params: 'with_materialized_view_metrics' : ['all'] .... ``` * `username`: The username used to log into your StarRocks cluster. Unless using the root account, the user must be granted both the `user_admin` and `db_admin` roles. * `password`: The password used to log into your StarRocks cluster. * `'with_materialized_view_metrics'`: The scope of the metrics to collect. Valid values: * `'all'`: All metrics relevant to materialized views are collected. * `'minified'`: Gauge metrics and metrics whose values are `0` will not be collected. #### Metric items[​](#metric-items "Direct link to Metric items") ##### mv\_refresh\_jobs[​](#mv_refresh_jobs "Direct link to mv_refresh_jobs") * Type: Counter * Description: Total number of refresh jobs triggered for the materialized view. A refresh job corresponds to one user-initiated or scheduled refresh; a single job may execute multiple task runs internally. Each job is counted once when it reaches a terminal state. MERGED task runs (sub-tasks merged into a later batch) are not counted. ##### mv\_refresh\_total\_success\_jobs[​](#mv_refresh_total_success_jobs "Direct link to mv_refresh_total_success_jobs") * Type: Counter * Description: Number of refresh jobs that completed successfully. Counted once per job. ##### mv\_refresh\_total\_failed\_jobs[​](#mv_refresh_total_failed_jobs "Direct link to mv_refresh_total_failed_jobs") * Type: Counter * Description: Number of refresh jobs that failed. Counted once per job. ##### mv\_refresh\_total\_empty\_jobs[​](#mv_refresh_total_empty_jobs "Direct link to mv_refresh_total_empty_jobs") * Type: Counter * Description: Number of canceled refresh jobs of the materialized view because the data to refresh is empty. ##### mv\_refresh\_total\_retry\_meta\_count[​](#mv_refresh_total_retry_meta_count "Direct link to mv_refresh_total_retry_meta_count") * Type: Counter * Description: Number of times when the materialized view refresh job checks whether the base table is updated. ##### mv\_query\_total\_count[​](#mv_query_total_count "Direct link to mv_query_total_count") * Type: Counter * Description: Number of times when the materialized view is used in the pre-processing of a query. ##### mv\_query\_total\_hit\_count[​](#mv_query_total_hit_count "Direct link to mv_query_total_hit_count") * Type: Counter * Description: Number of times when the materialized view is considered able to rewrite a query in the query plan. This value may appear higher because the final query plan may skip rewriting due to a high cost. ##### mv\_query\_total\_considered\_count[​](#mv_query_total_considered_count "Direct link to mv_query_total_considered_count") * Type: Counter * Description: Number of times when the materialized view rewrites a query (excluding direct queries against the materialized view). ##### mv\_query\_total\_matched\_count[​](#mv_query_total_matched_count "Direct link to mv_query_total_matched_count") * Type: Counter * Description: Number of times when the materialized view is involved in the final plan of a query (including direct queries against the materialized view). ##### mv\_refresh\_pending\_jobs[​](#mv_refresh_pending_jobs "Direct link to mv_refresh_pending_jobs") * Type: Gauge * Description:| Number of currently pending refresh jobs of the materialized view. ##### mv\_refresh\_running\_jobs[​](#mv_refresh_running_jobs "Direct link to mv_refresh_running_jobs") * Type: Gauge * Description:| Number of currently running refresh jobs of the materialized view. ##### mv\_row\_count[​](#mv_row_count "Direct link to mv_row_count") * Type: Gauge * Description:| Row count of the materialized view. ##### mv\_storage\_size[​](#mv_storage_size "Direct link to mv_storage_size") * Type: Gauge * Description:| Size of the materialized view. Unit: Byte. ##### mv\_inactive\_state[​](#mv_inactive_state "Direct link to mv_inactive_state") * Type: Gauge * Description:| Status of the materialized view. Valid values: `0`(active) and `1`(inactive). ##### mv\_partition\_count[​](#mv_partition_count "Direct link to mv_partition_count") * Type: Gauge * Description:| Number of partitions in the materialized view. The value is `0` if the materialized view is not partitioned. ##### mv\_refresh\_duration[​](#mv_refresh_duration "Direct link to mv_refresh_duration") * Type: Histogram * Description: Wall-clock duration of refresh jobs, in milliseconds. For multi-batch jobs, measured from the first task run start to the final task run completion. ##### mv\_global\_count[​](#mv_global_count "Direct link to mv_global_count") * Type: Gauge * Description: Current number of asynchronous materialized views in the cluster, with labels `refresh_mode` (the materialized view's refresh mode) and `status` (`ACTIVE` or `INACTIVE`). This metric is always emitted, regardless of the per-materialized-view metrics privilege. ##### mv\_global\_query\_rewrite\_queries\_total[​](#mv_global_query_rewrite_queries_total "Direct link to mv_global_query_rewrite_queries_total") * Type: Counter * Description: Number of queries grouped by materialized view rewrite outcome, with label `state`: `HIT` (the query was rewritten to use a materialized view), `NO_HIT` (rewrite was enabled but no materialized view was used), or `DISABLED` (materialized view rewrite was disabled by the session variable or the FE configuration). Counted once per query. ##### mv\_global\_query\_mv\_usage\_total[​](#mv_global_query_mv_usage_total "Direct link to mv_global_query_mv_usage_total") * Type: Counter * Description: Number of times materialized views are used by queries, with labels `usage_type` (`REWRITE` if a query was rewritten to use the materialized view, or `DIRECT` if a query reads the materialized view directly) and `refresh_mode`. ##### mv\_global\_refresh\_jobs\_total[​](#mv_global_refresh_jobs_total "Direct link to mv_global_refresh_jobs_total") * Type: Counter * Description: Total number of materialized view refresh jobs across all materialized views, with label `warehouse_name`. This is the fleet-level aggregate of `mv_refresh_jobs`: counted once per job on its terminal task run, excluding `MERGED` runs. Always emitted, regardless of the per-materialized-view metrics privilege. ##### mv\_global\_refresh\_success\_jobs\_total[​](#mv_global_refresh_success_jobs_total "Direct link to mv_global_refresh_success_jobs_total") * Type: Counter * Description: Total number of successful materialized view refresh jobs across all materialized views, by `warehouse_name`. ##### mv\_global\_refresh\_failed\_jobs\_total[​](#mv_global_refresh_failed_jobs_total "Direct link to mv_global_refresh_failed_jobs_total") * Type: Counter * Description: Total number of failed materialized view refresh jobs across all materialized views, by `warehouse_name`. ##### mv\_global\_refresh\_duration[​](#mv_global_refresh_duration "Direct link to mv_global_refresh_duration") * Type: Histogram * Description: Per-job wall-clock duration of materialized view refresh jobs, in milliseconds, by `warehouse_name`. ##### mv\_global\_refresh\_pending\_jobs[​](#mv_global_refresh_pending_jobs "Direct link to mv_global_refresh_pending_jobs") * Type: Gauge * Description: Current number of pending materialized view refresh jobs across all materialized views, aggregated by `warehouse_name`. ##### mv\_global\_refresh\_running\_jobs[​](#mv_global_refresh_running_jobs "Direct link to mv_global_refresh_running_jobs") * Type: Gauge * Description: Current number of running materialized view refresh jobs across all materialized views, aggregated by `warehouse_name`. --- ### Monitoring Metrics for Shared-data Clusters StarRocks provides two Dashboard templates for shared-data clusters: * [Shared-data Dashboard](#shared-data-dashboard) * [Starlet Dashboard](#starlet-dashboard) #### Shared-data Dashboard[​](#shared-data-dashboard "Direct link to Shared-data Dashboard") Shared-data Dashboard includes the following categories of monitoring metrics: * [Publish Version](#publish-version) * [Metadata](#metadata) * [Metacache](#metacache) * [Vacuum](#vacuum) * [Loading](#loading) ##### Publish Version[​](#publish-version "Direct link to Publish Version") ###### Latency / QPS[​](#latency--qps "Direct link to Latency / QPS") * Description: Quantile latency, average latency, and QPS of Public Version tasks. ###### Queued Tasks[​](#queued-tasks "Direct link to Queued Tasks") * Description: The number of Public Version tasks in the queue. ##### Metadata[​](#metadata "Direct link to Metadata") ###### Get Tablet Metadata[​](#get-tablet-metadata "Direct link to Get Tablet Metadata") * Description: Quantile latency, average latency, and QPS of Get Tablet Metadata tasks. ###### Put Tablet Metadata[​](#put-tablet-metadata "Direct link to Put Tablet Metadata") * Description: Quantile latency, average latency, and QPS of Put Tablet Metadata tasks. ###### Get Txn Log[​](#get-txn-log "Direct link to Get Txn Log") * Description: Quantile latency, average latency, and QPS of Get Txn Log tasks. ###### Put Txn Log[​](#put-txn-log "Direct link to Put Txn Log") * Description: Quantile latency, average latency, and QPS of Put Txn Log tasks. ##### Metacache[​](#metacache "Direct link to Metacache") ###### Metacache Usage[​](#metacache-usage "Direct link to Metacache Usage") * Description: Metacache utilization rate. ###### Delvec Cache Miss Per Minute[​](#delvec-cache-miss-per-minute "Direct link to Delvec Cache Miss Per Minute") * Description: Number of cache misses in Delvec Cache per minute. ###### Metadata Cache Miss Per Minute[​](#metadata-cache-miss-per-minute "Direct link to Metadata Cache Miss Per Minute") * Description: Number of cache misses in Metadata Cache per minute. ###### Txn Log Cache Miss Per Minute[​](#txn-log-cache-miss-per-minute "Direct link to Txn Log Cache Miss Per Minute") * Description: Number of cache misses in Txn Log Cache per minute. ###### Segment Cache Miss Per Minute[​](#segment-cache-miss-per-minute "Direct link to Segment Cache Miss Per Minute") * Description: Number of cache misses in Segment Cache per minute. ##### Vacuum[​](#vacuum "Direct link to Vacuum") ###### Vacuum Deletes[​](#vacuum-deletes "Direct link to Vacuum Deletes") * Description: Quantile latency, average latency, and QPS of Vacuum Deletes tasks. ###### Errors[​](#errors "Direct link to Errors") * Description: Number of failed Vacuum Deletes operations. ##### Loading[​](#loading "Direct link to Loading") ###### Queue Size[​](#queue-size "Direct link to Queue Size") * Description: Queue size of BE Async Delta Writer. #### Starlet Dashboard[​](#starlet-dashboard "Direct link to Starlet Dashboard") Starlet Dashboard includes the following categories of monitoring metrics: * [FSLIB READ IO METRICS](#fslib-read-io-metrics) * [FSLIB WRITE IO METRICS](#fslib-write-io-metrics) * [S3 IO METRICS](#s3-io-metrics) * [FSLIB CACHE METRICS](#fslib-cache-metrics) * [FSLIB FS METRICS](#fslib-fs-metrics) ##### FSLIB READ IO METRICS[​](#fslib-read-io-metrics "Direct link to FSLIB READ IO METRICS") ###### fslib read io\_latency (quantile)[​](#fslib-read-io_latency-quantile "Direct link to fslib read io_latency (quantile)") * Type: Histogram * Description: Quantile latency for S3 reads. ###### fslib read io\_latency (average)[​](#fslib-read-io_latency-average "Direct link to fslib read io_latency (average)") * Type: Counter * Description: Average latency for S3 reads. ###### fslib total read data[​](#fslib-total-read-data "Direct link to fslib total read data") * Type: Counter * Description: Total data size for S3 reads. ###### fslib read iosize (quantile)[​](#fslib-read-iosize-quantile "Direct link to fslib read iosize (quantile)") * Type: Histogram * Description: Quantile I/O size for S3 reads. ###### fslib read iosize (average)[​](#fslib-read-iosize-average "Direct link to fslib read iosize (average)") * Type: Counter * Description: Average I/O size for S3 reads. ###### fslib read throughput[​](#fslib-read-throughput "Direct link to fslib read throughput") * Type: Counter * Description: I/O throughput per second for S3 reads. ###### fslib read iops[​](#fslib-read-iops "Direct link to fslib read iops") * Type: Counter * Description: Number of I/O operations per second for S3 reads. ##### FSLIB WRITE IO METRICS[​](#fslib-write-io-metrics "Direct link to FSLIB WRITE IO METRICS") ###### fslib write io\_latency (quantile)[​](#fslib-write-io_latency-quantile "Direct link to fslib write io_latency (quantile)") * Type: Histogram * Description: Quantile latency for application writes. Please note that this value may appear lower because this metric monitors only data written to the buffer. ###### fslib write io\_latency (average)[​](#fslib-write-io_latency-average "Direct link to fslib write io_latency (average)") * Type: Counter * Description: Average latency for application writes. Please note that this value may appear lower because this metric monitors only data written to the buffer. ###### fslib total write data[​](#fslib-total-write-data "Direct link to fslib total write data") * Type: Counter * Description: Total data size for application writes. ###### fslib write iosize (quantile)[​](#fslib-write-iosize-quantile "Direct link to fslib write iosize (quantile)") * Type: Histogram * Description: Quantile I/O size for application writes. ###### fslib write iosize (average)[​](#fslib-write-iosize-average "Direct link to fslib write iosize (average)") * Type: Counter * Description: Average I/O size for application writes. ###### fslib write throughput[​](#fslib-write-throughput "Direct link to fslib write throughput") * Type: Counter * Description: I/O throughput per second for application writes. ##### S3 IO METRICS[​](#s3-io-metrics "Direct link to S3 IO METRICS") ###### fslib s3 single upload iops[​](#fslib-s3-single-upload-iops "Direct link to fslib s3 single upload iops") * Type: Counter * Description: Number of invocations per second for S3 Put Object. ###### fslib s3 single upload iosize (quantile)[​](#fslib-s3-single-upload-iosize-quantile "Direct link to fslib s3 single upload iosize (quantile)") * Type: Histogram * Description: Quantile I/O size for S3 Put Object. ###### fslib s3 single upload latency (quantile)[​](#fslib-s3-single-upload-latency-quantile "Direct link to fslib s3 single upload latency (quantile)") * Type: Histogram * Description: Quantile latency for S3 Put Object. ###### fslib s3 multi upload iops[​](#fslib-s3-multi-upload-iops "Direct link to fslib s3 multi upload iops") * Type: Counter * Description: Number of invocations per second for S3 Multi Upload Object. ###### fslib s3 multi upload iosize (quantile)[​](#fslib-s3-multi-upload-iosize-quantile "Direct link to fslib s3 multi upload iosize (quantile)") * Type: Histogram * Description: Quantile I/O size for S3 Multi Upload Object. ###### fslib s3 multi upload latency (quantile)[​](#fslib-s3-multi-upload-latency-quantile "Direct link to fslib s3 multi upload latency (quantile)") * Type: Histogram * Description: Quantile latency for S3 Multi Upload Object. ###### fslib s3 complete multi upload latency (quantile)[​](#fslib-s3-complete-multi-upload-latency-quantile "Direct link to fslib s3 complete multi upload latency (quantile)") * Type: Histogram * Description: Quantile latency for S3 Complete Multi Upload Object. ##### FSLIB CACHE METRICS[​](#fslib-cache-metrics "Direct link to FSLIB CACHE METRICS") ###### fslib cache hit ratio[​](#fslib-cache-hit-ratio "Direct link to fslib cache hit ratio") * Type: Counter * Description: Cache hit ratio. ###### fslib cache hits/misses[​](#fslib-cache-hitsmisses "Direct link to fslib cache hits/misses") * Type: Counter * Description: Number of cache hits/misses per second. ##### FSLIB FS METRICS[​](#fslib-fs-metrics "Direct link to FSLIB FS METRICS") ###### fslib alive fs instances count[​](#fslib-alive-fs-instances-count "Direct link to fslib alive fs instances count") * Type: Gauge * Description: Number of file system instances that are alive. ###### fslib open files[​](#fslib-open-files "Direct link to fslib open files") * Type: Counter * Description: Cumulative number of opened files. ###### fslib create files[​](#fslib-create-files "Direct link to fslib create files") * Type: Counter * Description: Average number of files created per second. ###### filesystem meta operations[​](#filesystem-meta-operations "Direct link to filesystem meta operations") * Type: Counter * Description: Average number of directory listing operations per second. ###### fslib async caches[​](#fslib-async-caches "Direct link to fslib async caches") * Type: Counter * Description: Cumulative number of files in asynchronous cache. ###### fslib create files (TOTAL)[​](#fslib-create-files-total "Direct link to fslib create files (TOTAL)") * Type: Counter * Description: Cumulative number of files created. ###### fslib async tasks[​](#fslib-async-tasks "Direct link to fslib async tasks") * Type: Counter * Description: Cumulative number of asynchronous tasks in the queue. --- ### Monitor and Alert with Prometheus and Grafana StarRocks provides a monitor and alert solution by using Prometheus and Grafana. This allows you to visualize the running of your cluster, facilitating monitoring and troubleshooting. #### Overview[​](#overview "Direct link to Overview") StarRocks provides a Prometheus-compatible information collection interface. Prometheus can retrieve metric information of StarRocks by connecting to the HTTP ports of BE and FE nodes and storing the information in its own time-series database. Grafana can then use Prometheus as a data source to visualize the metric information. By using the dashboard templates provided by StarRocks, you can easily monitor your StarRocks cluster and set alerts for it with Grafana. ![MA-1](/assets/images/monitor1-c933907ca154e75553acfa00488e5904.png) Follow these steps to integrate your StarRocks cluster with Prometheus and Grafana: 1. Install necessary components - Prometheus and Grafana. 2. Understand the core monitoring metrics of StarRocks. 3. Set alert channel and alert rule. #### Step 1: Install Monitoring Components[​](#step-1-install-monitoring-components "Direct link to Step 1: Install Monitoring Components") The default ports of Prometheus and Grafana do not conflict with those of StarRocks. However, it is recommended to deploy them on a different server from that of your StarRocks clusters for production. This reduces the risk of resource conflicts and avoids potential alert failure due to the server's abnormal shutdown. Additionally, please note that Prometheus and Grafana cannot monitor their own service's availability. Therefore, in a production environment, it is recommended to use Supervisor to set up a heartbeat service for them. The following tutorial deploys monitoring components on the monitoring node (IP: 192.168.110.23) using the root OS user. They monitor the following StarRocks cluster (which uses default ports). When setting up a monitoring service for your own StarRocks cluster based on this tutorial, you only need to replace the IP addresses. | **Host** | **IP** | **OS user** | **Services** | | -------- | --------------- | ----------- | ------------ | | node01 | 192.168.110.101 | root | 1 FE + 1 BE | | node02 | 192.168.110.102 | root | 1 FE + 1 BE | | node03 | 192.168.110.103 | root | 1 FE + 1 BE | > **NOTE** > > Prometheus and Grafana can only monitor FE, BE, and CN nodes, not Broker nodes. ##### 1.1 Deploy Prometheus[​](#11-deploy-prometheus "Direct link to 1.1 Deploy Prometheus") ###### 1.1.1 Download Prometheus[​](#111-download-prometheus "Direct link to 1.1.1 Download Prometheus") For StarRocks, you only need to download the installation package of the Prometheus server. Download the package to the monitoring node. [Click here to download Prometheus](https://prometheus.io/download/). Take the LTS version v2.45.0 as an example, click the package to download it. ![MA-2](/assets/images/monitor2-c1ff652ccf35625c91e46ffcb24cd63e.png) Alternatively, you can download it using the `wget` command: ```bash # The following example downloads the LTS version v2.45.0. # You can download other versions by replacing the version number in the command. wget https://github.com/prometheus/prometheus/releases/download/v2.45.0/prometheus-2.45.0.linux-amd64.tar.gz ``` After the download is complete, upload or copy the installation package to the directory **/opt** on the monitoring node. ###### 1.1.2 Install Prometheus[​](#112-install-prometheus "Direct link to 1.1.2 Install Prometheus") 1. Navigate to **/opt** and decompress the Prometheus installation package. ```bash cd /opt tar xvf prometheus-2.45.0.linux-amd64.tar.gz ``` 2. For ease of management, rename the decompressed directory to **prometheus**. ```bash mv prometheus-2.45.0.linux-amd64 prometheus ``` 3. Create a data storage path for Prometheus. ```bash mkdir prometheus/data ``` 4. For ease of management, you can create a system service startup file for Prometheus. ```bash vim /etc/systemd/system/prometheus.service ``` Add the following content to the file: ```properties [Unit] Description=Prometheus service After=network.target [Service] User=root Type=simple ExecReload=/bin/sh -c "/bin/kill -1 `/usr/bin/pgrep prometheus`" ExecStop=/bin/sh -c "/bin/kill -9 `/usr/bin/pgrep prometheus`" ExecStart=/opt/prometheus/prometheus --config.file=/opt/prometheus/prometheus.yml --storage.tsdb.path=/opt/prometheus/data --storage.tsdb.retention.time=30d --storage.tsdb.retention.size=30GB [Install] WantedBy=multi-user.target ``` Then, save and exit the editor. > **NOTE** > > If you deploy Prometheus under a different path, please make sure to synchronize the path in the ExecStart command in the file above. Additionally, the file configures the expiration conditions for Prometheus data storage to be "30 days or more" or "greater than 30 GB". You can modify this according to your needs. 5. Modify the Prometheus configuration file **prometheus/prometheus.yml**. This file has strict requirements for the format of the content. Please pay special attention to spaces and indentation when making modifications. ```bash vim prometheus/prometheus.yml ``` Add the following content to the file: ```yaml global: scrape_interval: 15s # Set the global scrape interval to 15s. The default is 1 min. evaluation_interval: 15s # Set the global rule evaluation interval to 15s. The default is 1 min. scrape_configs: - job_name: 'StarRocks_Cluster01' # A cluster being monitored corresponds to a job. You can customize the StarRocks cluster name here. metrics_path: '/metrics' # Specify the Restful API for retrieving monitoring metrics. static_configs: # The following configuration specifies an FE group, which includes 3 FE nodes. # Here, you need to fill in the IP and HTTP ports corresponding to each FE. # If you modified the HTTP ports during cluster deployment, make sure to adjust them accordingly. - targets: ['192.168.110.101:8030','192.168.110.102:8030','192.168.110.103:8030'] labels: group: fe # The following configuration specifies a BE group, which includes 3 BE nodes. # Here, you need to fill in the IP and HTTP ports corresponding to each BE. # If you modified the HTTP ports during cluster deployment, make sure to adjust them accordingly. - targets: ['192.168.110.101:8040','192.168.110.102:8040','192.168.110.103:8040'] labels: group: be ``` note Please note that Prometheus is unable to detect the service changes (`targets`) after the cluster has been scaled in or out. For example, for clusters deployed on AWS, you can grant the EC2 instance that hosts the Prometheus service the `ec2:DescribeInstances` and `ec2:DescribeTags` permissions, and add the `ec2_sd_configs` and `relabel_configs` properties to **prometheus/prometheus.yml**. For detailed instructions, see [Appendix - Enable Service Detection for Prometheus](#enable-service-detection-for-prometheus). After you have modified the configuration file, you can use `promtool` to verify whether the modification is valid. ```bash ./prometheus/promtool check config prometheus/prometheus.yml ``` The following prompt indicates that the check has passed. You can then proceed. ```bash SUCCESS: prometheus/prometheus.yml is valid prometheus config file syntax ``` 6. Start Prometheus. ```bash systemctl daemon-reload systemctl start prometheus.service ``` 7. Check the status of Prometheus. ```bash systemctl status prometheus.service ``` If `Active: active (running)` is returned, it indicates that Prometheus has started successfully. You can also use `netstat` to check the status of the default Prometheus port (9090). ```bash netstat -nltp | grep 9090 ``` 8. Set Prometheus to start on boot. ```bash systemctl enable prometheus.service ``` **Other commands**: * Stop Prometheus. ```bash systemctl stop prometheus.service ``` * Restart Prometheus. ```bash systemctl restart prometheus.service ``` * Reload configurations on runtime. ```bash systemctl reload prometheus.service ``` * Disable start on boot. ```bash systemctl disable prometheus.service ``` ###### 1.1.3 Access Prometheus[​](#113-access-prometheus "Direct link to 1.1.3 Access Prometheus") You can access the Prometheus Web UI through a browser, and the default port is 9090. For the monitoring node in this tutorial, you need to visit `192.168.110.23:9090`. On the Prometheus homepage, navigate to **Status** --> **Targets** in the top menu. Here, you can see all the monitored nodes for each group job configured in the **prometheus.yml** file. Usually, the status of all nodes should be UP, indicating that the service communication is normal. ![MA-3](/assets/images/monitor3-0edea7a4319bf836da3e55c4a24559f9.jpeg) At this point, Prometheus is configured and set up. For more detailed information, you can refer to the [Prometheus Documentation](https://prometheus.io/docs/). ##### 1.2 Deploy Grafana[​](#12-deploy-grafana "Direct link to 1.2 Deploy Grafana") ###### 1.2.1 Download Grafana[​](#121-download-grafana "Direct link to 1.2.1 Download Grafana") [Click here to download Grafana](https://grafana.com/grafana/download). Alternatively, you can use the `wget` command to download the Grafana RPM installation package. ```bash # The following example downloads the LTS version v10.0.3. # You can download other versions by replacing the version number in the command. wget https://dl.grafana.com/enterprise/release/grafana-enterprise-10.0.3-1.x86_64.rpm ``` ###### 1.2.2 Install Grafana[​](#122-install-grafana "Direct link to 1.2.2 Install Grafana") 1. Use the `yum` command to install Grafana. This command will automatically install the dependencies required for Grafana. ```bash yum -y install grafana-enterprise-10.0.3-1.x86_64.rpm ``` 2. Start Grafana. ```bash systemctl start grafana-server.service ``` 3. Check the status of Grafana. ```bash systemctl status grafana-server.service ``` If `Active: active (running)` is returned, it indicates that Grafana has started successfully. You can also use `netstat` to check the status of the default Grafana port (3000). ```bash netstat -nltp | grep 3000 ``` 4. Set Grafana to start on boot. ```bash systemctl enable grafana-server.service ``` **Other commands**: * Stop Grafana. ```bash systemctl stop grafana-server.service ``` * Restart Grafana. ```bash systemctl restart grafana-server.service ``` * Disable start on boot. ```bash systemctl disable grafana-server.service ``` For more information, refer to the [Grafana Documentation](https://grafana.com/docs/grafana/latest/). ###### 1.2.3 Access Grafana[​](#123-access-grafana "Direct link to 1.2.3 Access Grafana") You can access the Grafana Web UI through a browser, and the default port is 3000. For the monitoring node in this tutorial, you need to visit `192.168.110.23:3000`. The default username and password required for login are both set to `admin`. Upon the initial login, Grafana will prompt you to change the default login password. If you want to skip this for now, you can click `Skip`. Then, you will be re-directed to the Grafana Web UI homepage. ![MA-4](/assets/images/monitor4-769f8f814df66a7014c2baba78a300f0.png) ###### 1.2.4 Configure data sources[​](#124-configure-data-sources "Direct link to 1.2.4 Configure data sources") Click on the menu button in the upper-left corner, expand **Administration**, and then click **Data sources**. ![MA-5](/assets/images/monitor5-ce53b5a566261841ef338f2dbd65e148.png) On the page that appears, click **Add data source**, and then choose **Prometheus**. ![MA-6](/assets/images/monitor6-12deb6c827e59ca679b17ea3823564e5.png) ![MA-7](/assets/images/monitor7-c79999ed4ef04037c3c97153a52b7f6c.png) To integrate Grafana with your Prometheus service, you need to modify the following configuration: * **Name**: The name of the data source. You can customize the name for the data source. ![MA-8](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAmwAAABoCAYAAABMm6waAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAABquSURBVHhe7d3/U1XnnQfw/glLtt1EK2C8UVtDivFLwKBGNmwhrKQkGiImBA1QFDQQQoAgguGqCGQSghZmDaQWthaiQeMAzTBki6bFYcMP5of1p+zMznZmZ2faaTc72yRtmnz283nOc+59zuUA96K5HK7vH15z7jnPc8659/LIefs85zl8K+6uuwkAAAAAvAuBDQAAAMDjENgAAAAAPA6BDQAAAMDjENgAAAAAPM4zge17319PL1S84tj29J79tHPnU45tR46cVHXNbRB9j+zIpEOHax1KSipo5b1rFXkdWp6S+ojrsQAAAGBunglsO9KzaHr6E8e2t9++oJjbpI7UNbd5QnYrDX0wRT2HU9zLY8xhDmDHj7+hlrb29m5avz5FkddmmV3X7ViLYWvpG9ReXeBaJnZXd5O/NMO1LGryGqm9uYq2upUBAMAd5c4IbGmV1HFxiianPlH7T09O0dCJMvK51XVTP6r2G6gPbqsblGONUp1dp7CfJrjO8Kns4H4xzA5i5rbQwGaWudWfXQYdaO5Wx7C1HGum4uxNLnUXBoENAACWEk8EtvgEH/X2XlChyBwWvT2BrYA6xiRc3aChMx3U2NBBXRdvqOOMvTn7BdshnMB2h4lGYLMD04r7d1BWQSP5209Tdd7tCW0IbAAAsJR4IrBVVNTTyMiv1VKC0abNW9X20MAm20ND3bzyrJ6vyd5KY3sRdX3AgevDQSpR6z7KaRilsUkJYWxinLpKH1V1rWBmGBymAXOdqSCnzzNxxgoB1n58nNfHaUL37I2dO0ppgffwKJWcnQocY+zsIA0b+8cl5FHj+WD55NgoNeb49L6LL5qBzbZp/0lqb6mnncv1tuU7aM9L7dTSavfC1VPuA8H6y3aUUe1x3UvX+gbVFuXQCl2mAltNGeVWtAfL9z9Gy3S5CmyHXqDKY/b+7VSZa4ZFH2UUNFOTPnf7sWZ6dkeiUb6JskqbyW+XH2+mfUa5dfwqqpb3Z4eyB/KD5+P6Bw7x5zUC21yfBwAAYpuHAtuHKohJOHELbNILJ2XvvPM+Xb16g4qKX3AcY1YbWmlIQs/kOPl3rXOtE3/4Mk1ynYnzXaoHrmfkJp/rGrVl302+1GxKPzGuzj10gl+nptHmjGzyX7ICmZ9fb17Dx3ENbBy0RgbJHzjmTeo7bJ0zs/2aS3lw/8Kz0gt4gwaaiiinuIuGJExODFKhfs+LreTHFVZwCGEGtlCyj9uxZnIPbHFrS+hI+2kqz7LWd1ac5qBURRmrOQgtT6U91bxet59Wq/pPUWVLNzUdzKHVyxNp9Q+rqIn3rcy1QpMKbLxeW/AYrU5MotRnmqml/STt22gdWwJVe3s7ledspRWJ6ynroAS7Ztqz1ipPekbC40kq/mESLVueRBmq/CQ9qwPjg4VS3kh7NkvI5nAn5cdrKEO9t+DxK3c/Rg8m8zHu2kr7Gjh01pXQjtU+Wr3lKSqX8BYIbHN/HgAAiG2eGRKVYCaBRcKbvd0MbPY9blJXQpvMFrXrzSetflQFMtl/euIaDZzx0y4JWao8hfxXeHugt43p8DX2Zp61Hs6Q6Cw9bI0Jurx4UL0H65jZ1DbC5ZOXqcouT7CCpWN/ozyt0M9hsoZ2Jen6i0xmgtrhzLSMw4RwK5N93I410yyB7a4CqubgV50nrzm0JKdSkoQ1u1yGENsbaXegLoe7bLs8iXY8uZ9ydySpdRXY6kt0uLPr87F3W+uqB+zgY7pMPEHlJ7vpyP5Ufm29v6Yi6z8Wlq1U3Bh8zyvuT6UH7zd6RLdXkZ8DWvHD1rocv+XQEzPLU/Q6W/Z0sxHY5v48AAAQ25bUpAMJalJHetginniwpoDqzozS8IQEKTY1RR35ckGtoT57KDREYHhywYFttnK/Nax6pYOS7XK9zd4/uVaHzKkbNHxxkNoOF4Q/SSIKJHwlcwgLZQc2t7LbG9hYYoZjSNRiB7Yk2nlIetE4WNXV04H8J2hTon0ct3vYnMdWgc1x/kTaU2Nv0+FJ9/TZdr54OnjM5am00xwSVd6gA9utujOOL2FThnvtdXtbILDN/XkAACC2eTawSU+aDJMKeW1vlzqhz2aLlK+wn8YkDI10Ubodnsb6qTAjm9INacn6vIsQ2IQvs5IazZDJ7zczUH9xLf6QaBKHHmtINMvuyXL0sFlWbH6Cdu+voVoZXmw5Sft0YPomAtvul/gc+pjq+DIkukWGO7lc9aDdSmCzzPZ5AAAgtnk2sHV2/lRNRBDy2t4udSLpXUvX94oN1BvDU3zBVZMOJi9TuT08KcOXG4L7paVnU7xd/7YHtvmGRMuo7fwoDZwqs8ru8lFJr9zjNkUdubr+Ilv8SQdWnUBvG1NDiHZg42Cz5+knaJMuCx2yDCuw3cKQ6IxAtmOewDbfkOg8nwcAAGKbZwObDHvK/WxCXtvbIw1scWl+GpAhzyn7sR6tHIasx3pMnverHq7A8OMHo9TG5f6z13hdJgjov6jA5VJ/op/3L85V2+rOSyC7QX0nWqkkk+tEFNg4EDbPNekghfe3AlpfbRGlP+unPnk0iRnwFlk0A5s81mNn6UlqaTcf65Fq3aRfX0Y77l9P3/v7EqrlQBUIbCllqjeutiBDzaRcsZnLWzhwFUrgCi+wzT/pQGaG6kkHJfL+gpMO1ISI5hrauYXfm0wg4HA155DojEkHBVRpziCd5/MAAEBs82xgk0kFsi7MCQayHlFgY/HpNc4H58p9YedaKScQfkIe6zE1RUOvVwYfwZFQRB3v67LzR61j5vfSmD7eQDXXiTCwzXisx0UJiUb5mgLHYz3kUSMdhe6zXBdDNAKbHMPm+uDc0MdgFDVaockOVdlVjsdgHCnLpyS9b1g9bCGP9Sh3nN9HGUXGPWryWI8Moxd3+WO0r47PofetLqhyzHCdGdjYPI/1mOvzAABAbFtSkw6E1Ik0sHnTo5SeblzgHbNIzXreJOHLf/wNx98KlSBhBjazTOqGH9gAAADA5JnAJn/Q3Xykh5DJBaHhLFb++Pv3G8ZpevIa9ZxopcaGLhpQEwusZ7+51fca+UPudq+ZTSYVyExQIa9DyyXIuR0LAAAA5uaZwHbHkb9kcO5a4K8gmH9dAQAAAMCEwAYAAADgcQhsAAAAAB6HwAYAAADgcQhsAAAAAB6HwAYAAADgcQhsAAAAAB6HwAYAAADgcbcc2MofWUW/b3yQqGUj20R00unrE7xkXx8Xm+lrPy/9stSatVctXx0TDwU1aY3aUctf1TKF/trArxt4eUSrd/ryFV6+kkpf1mm1IWq0l8UW+ks1L6tlqb2kVWkviocD/lypVWgvmNLoz4d5eTiNvjjEr9kX5aat9EUZL8t4edDy+QFeMlkqpdqPxTb6vMTps2KtSGynz57n5fOy1PZb/rRvO/12bxqVJq9x/TkCAACAd91yYPtd43oV1r62ndROCAlpvFRhTYLaJvpKBTZeNmuvCg5pslRhTWuy/LXRoIKaXqqgxo5YvpRlPS8VCWq85LD2ZZ2hNoX+wtSyRqTSX17mJYc1hcNawEuWP1eJLdaSw1pApVZh+UKWHNK+eIFfy5KDmnLIUP4wfc4krH2ughovDxoOGErFVvrsx7zksPaZKDEUa0VCBzYOa3+y7df2GQq303/uedj15wgAAADedcuBTfWunZLeNbuHjZfSs2b0sFnBTQvpYftK97BJaAv2sGmqd00vdQ+b1bNmLVXP2lw9bBLYbCq0pbr3rqngxkvdwzajd83oYZOwJr1samn3rs3WwyY9a0YPm2L2sOnQZgc3u4dtRu+ao4eNl9KzJj1sdu9aoIeNBXrYeCk9a0YPm+ply09z/TkCAACAd936kOj2e+l3HNqkd031tM3Vw8bMHjYrqNk9bJpLD9tXgV42HdRcetgsHMoCPWxmUDMYPWx2UHNw9LBxMKvipeph02bpYbM8rHrXVE+bWw+bCmy8NMKao4ct0MsmIY2XgaC2VfWwqaWjh01CGi/tsMakd031tDl62CSsbVNhDUOiAAAAS8+37lv7AAEAAACAd6nAds93EwEAAADAoxDYAAAAADwOgQ0AAADA4xDYAAAAADwOgQ0AAADA4xDYAAAAADwOgQ0AAADA4xDYAAAAADwOgQ0AAADA4xDYAAAAADwOgQ0AAADA4xDYAAAAADwOgQ0AAADA4xDYAAAAADwOgQ0AAADA4xDYAAAAADwOgQ0AAADA4xDYAAAAADwOgQ0AAADA4xDYAAAAADwOgQ0AAADA4xDYAAAAADwOgQ3uOM9997v0aUIcUeLfEMkyQZYAGreL/+N2cTL+Htf2AwCwGBDY4I4iYe1rt4s0wAxxdDr+btd2BAAQbVEJbCsS76N771tHq1bfDxA10uak7Zlt8U/oUYMIfM3tZcPmNIAlZ/3GLfx7cJ3j91+sXovdfteLlavW0rqkjZT0wOaYEJXAhrAGi0XantkWzYvxG/d8m9b+7d0UdxdAUM53vuNoJ251ALxu2fJEFdrM33+xfC0O/V0vJKzdsyzB9ftZiqIS2Ny+XIBoMduidd+aZSXCGswieG9jnGs5wFIgPW3m7z+334+xxPysQnql3L6XpQqBDWKe2RbtsCbc/kEACLQTiAUIbAhsEXP7YgGixWyLuBBDONBOIBYgsCGwRcztiwWIFrMt4kIM4UA7gViAwIbAFjG3LxYgWsy2iAsxhAPtBGIBAhsCW8TcvliAaDHbIi7EEA60E4gFCGwIbBFz+2IBosVsi7gQQzjQTiAWILDdWmC7NzmL0gr/iXJe/ZjyOv9AT5/53/Cc/pTr/5GeevP3tOv1/6Yn239LT7T+B+W2fEJZr/yaUp/ppJU/+KHrOeeCwAYxz2yLuBBDONBOIBYgsC08sElQcw1j81Fh7Q8qqP2o5d/pH5tuUGbdrymj+l8UeS3bpEyCm9u5Z4PABp61fUcW5T9TTC/XvqrIa9nmVncuZluM3Quxnwamb1BXnlvZEpGQR+UNNZST4FIWZbHbTuBOgsC2sMCWUTniHsbmY4e11/6Ldjb/mwpoP8g4QL77Uyhx5SpFXss2KZM66WUXXd+DG28EtucH6er0J3T9F8fogcD2g/TWrz6h6Xdfc9aFmPf9pA207/lyerPzbTpw8CXa9dRziryWbVLmtt9szLZ42y7E9aM0PT1KdW5liyIGAltuL41N36S+Mmu9bpD//Q/6Z9aLgtvWTgAWUcSBTV+Lp23XP6ILJ4qN6/Ickiup48rH1n5XOinVrU4Ymt7l/X81SM+p9dfoAh/vQvPMem7MzyoWEtgW3LPGZBhUetYkiG0t7rOCWuJKV1ImdaRuuD1tngps09Mf07mKh/R2BLY71f6iQ9TQ2EopW9JnlMk2KZM6oWWzMdvibbsQI7B94xDYAG7NQgPbe6/l0+NPFlNF5wRd5/XRzufc65tqh7nuTQ5XxfR45lb3OmFYzMAm96y5BbGwnP5U3bMmQ53SezZXWLNJHakr+4RzT5vHAhsb79c/qJDAlvIy/cRO7+zqlbepOFnqWT/Qq++O0fiULvt5C1V3/kY1tOmpj/mHvStwrqyqQXrvmlVvenyY6rOt7eANMuQpvWhmWPuHzB8p9rqUSZ1wh0fNthjJhTg+v4uGJnRbmbpBQ6+XkY+3qyAh27SBeqt+Wmm/s357EcXrY+06c4OmL/ZTx/tSbgU9Oc5Ebz8NyD4f9NMu3haffpR6RriuPsbw2aOUpo+h3lNOKw2MGec4U6nLnYEtPr9f9VYNNGRb7yGhiNou6eOyiUu9VLgmeFxXef00MT1OXe3jNKH/bU1c6qK9/B77Au9hivpefDS4z5oy8l+cCpxn7GIH7Q0McVrvsaepN/g9jV2mqjRdrs4nn0Hq6XLFDsY+Sq8epGF734lr1GOc2+07tssiFUk7AfCqhQa2YEB6iCp+xtfd63ytVNfbWa6hIT1zV88epFXJ+VT/84/ouv7dYV5vnzsr1/IxatLnNUNa4LXbMXX92ZifVUQa2G6pd+30/6jeNbk/TYY83QKaG6kr+4TTy+apwHbhbD+N89JK887AlnV0kC68M0hNT22lDWVDqv5oZz6XWYFtenyIqgvyqaLnI/XDvf5uJ+Xz/xDaLt1Uja1azpPdTaNcNt5zhP/3cITeGuf9rnTSNvO9wKLa+2yJGvoM3SbMbVIndNtszLYY/oW4hvomOXB0l1Fygo+Sn7cCUN+LPqs8tIcto4uGJSA15apQ5yuU+hxOCq1yFSa4fKi9knIythnBjwNPbRGlb0+h+IRK6uEwMtHvp0wOU75MDmf8HiZ6K63QtYGDTOA9cfmuDhri9aGmFD6HEdjSdL0zwcBYfo7/HYz1U2Gyj+KTy6iLA9dk/9FAuSsVoPg4vTWUk7yOkvN7+TPye+aQ1nOYP+eabVTSzeFs8jKVq30eJf8VPu4Ih0HjPNNXOoxQyetjg1SeuY58qfx5P5D3UWOcLxg6Q3vY4osHVXlfrXzH6yizaZQm5Tsutn4mbt+xvW+kwm8nAN5164GNNY3xv6uP6a3n+fVc19Bmo56sl3bT+XeGqe1QJm3I7aT3rlu3PUlZWIFNlUW3h01mg7qFsXDIcKjMBpVJBeH0rtmkruwjs0fd3pPJW4Gt2U7zE3Qie64hUavMSty6h81O36ENTjUiq2E83vkbfs3H1v9T2NYm68FGA4vv5dpmdb+auW39xi2KuU3qSF1z22zMthj+hVjCxU3qKdUB7a4U2lvdSlX5Eo54PTSwcXhJd4SEAuriMDL2Zp5aV2GCg0tyoNwKJJMcxuz1eDkmh58q46b7+Bcvcyjh8/C25OZrNP3hIJXoMpFZP0gDp8r4tQ5s+UVWGOOgY/bM2eeyA1py/lFqrC5yvJ8ZdA9bY+D9+KjxIv+bvNgaDHpmyFKvp6gj267PsuW+NDuE6e+0OFiuvpexXsoJPRavOwObz/oMjpDpo6p+DqJcR7a5fccLFX47AfCu2xLY7G1H57mGhga2ECqI6eu5VwNbRI/uCCH7yqM7ZIhTTTBwCWeuuK7sI4/8cHtPJo8FNn6dfIzOqyT+9pxDosEu0vADm9VIgvtbENi8xC2wufWwffOBLYXKe6XHhkPXpcvU1VSper0C5TPuYfNRepkxJKpNnClQ5SpMhNyPJQHELg/UCQ0cqufOCjGq3AxLDlZgG77CdabGyZ/uLE8+PGgNa46NU9+ZVirJXOcod6UClHNo0Rmi7Do6ZMl3EhIo7Z5Ka9hYh0odyIT6THo4eO7AZgXg4WYdmLX0dg6xen+373ihwm8nAN51u3vY5ryGhga20CFRgcA205IObLyedcK60dH8AR/8Kf+ApectVyYlLKyHLb9bhkv5tf7fAXiPBDNvDIlafJmVVNc+SEMytDd5jdrssBEa2FTYkCHRPDVcaQeMWw5sauZkuIFNhkHLVK9TaA+bsiaXSmq7qOeS3GN2k4ZPBc/t6rYENut9fVOBLedN/iwIbACubvc9bHNeQ0MC27aWCbV+rsyagLAUetgwJMrcvliHGY3kOer4Jf/QeJv9A67+hdyLNkFtBfmUXzuk7nW72lPJZeEHtlVPv22Nv//sGOU/mU/Fr03Q1Uud9Lg6J3iB26QD2WZOMIjKpAMOao0NlZQZ2JarbmYPBKzQwCbrdvBQiiIObAsZEk0r9FNjcS6/NsKQuteNw2O9fUM+B7UG6VUL7pf5ejDo2NtmiDSwqdfzDYkuNLCFOSSKwAYQsNDAFjpLdLxbX1/nuoaGBDZr+FSe/MDHKu1W97BNc115RMgDuu65WqNsjsA22llMWY/YT5CYnflZBSYdLIDbF+swI7AFtwWGRLNb6Lzc4CjbxifoPRkuVT/8CAIbc8xwuTZBb1UFZ5CCN6jHehw95QhtNtkmZd/4Yz2yrUkEQycKrEkEmdYN/sOnsq3yfB1OclMoOdlHcWUSrG5Qz+E82pyaR+VnrZmSkQS2OD3pQG7ytyYd+KlPJiFEOumA66Y1jNPk5Dg1qhmYudQ2wmHnUgftkmHdNbnkv8hBZ6SL0u1zu4k0sNmTDvg8e/Wkg44RPo9j0kH4gW1vN5e930s5yfwd8+e1Jh3oCQ8y6aD2sqrvmHSAwAYQsNDApq6PwuU5bLNeQ0MCm+O5bFzvvV/y7wIOYsV22TCv67IL8to1sGVS0ztWvfHu4sB7mI35WUWkgQ2P9WBuXyzAXCSQSS9a6cGX6MndBYq8lm2RhDVhtsVILsShj+lwPmKjgNquWL9IrIkFPtprPP5i7HwHdUl40ZMKwgpsTB7r0Temf5Hpc6abPW6Ox3pMcaC0HjUyMwxZ4SkQltIqqct8rMfIYPBxGrOJOLCxNRzSjPPIYz2Cjw+JLLDF5XFo/lCOI8FYtsljPS7TGIdUdXx5rEe1fmyJfSwENoCAiAPbEmd+VhFpYBN4cK7LFwswHxnylPvUZHKBkNfhDoOazLaIC7GTCmA6XDncpuCzVKGdQCxAYIs8sAn8aSqARWK2RVyIIRxoJxALENgWFtgE/vg7wCIw2yIuxBAOtBOIBQhsCw9sQu5pk+Ams0cjeuSHCm1/VPe0SXCT2aPyyA95dIfMBpWgFs49a6EQ2CDmmW0RF2IIB9oJxAIEtlsLbF6DwAYxz2yLuBBDONBOIBYgsCGwRcztiwWIFrMt4kIM4UA7gViAwIbAFjG3LxYgWsy2iAsxhAPtBGIBAhsCW8TcvliAaDHbIi7EEA60E4gFCGwIbBFz+2IBosVsi7gQQzjQTiAWILAhsEXs3vvWuX65AN80aXtmW1QX4URciGFuaCew1C1bvpLWb9zi+P0Xy9fi0N/1Yl3SRrpnWYLr97MURSWwrUi8D6ENok7anLQ9sy2qC7G2c8ND6n+gAKbUTQ872olbHQCvk7C2arUzxMTqtdjtd71YuWqtCm3S0xYLohLYALziK+NCTAlxxmsAm7NduLUjAIBoQ2CDO8o/x/+d42IMMJfr8d9xbUcAANGGwAZ3nEsrzNCGXjZwE0f/Gv9t1/YDALAYENgAAAAAPA6BDQAAAMDjENgAAAAAPA6BDQAAAMDjENgAAAAAPA6BDQAAAMDjENgAAAAAPA6BDQAAAMDjENgAAAAAPA6BDQAAAMDjENgAAAAAPA6BDQAAAMDjENgAAAAAPC2R/h/myl9s2S1DBgAAAABJRU5ErkJggg==) * **Prometheus Server URL**: The URL of the Prometheus server, which, in this tutorial, is `http://192.168.110.23:9090`. ![MA-9](/assets/images/monitor9-0b3e34d7be2872123104e38932fcc362.png) After the configuration is complete, click **Save & Test** to save and test the configuration. If **Successfully queried the Prometheus API** is displayed, it means the data source is accessible. ![MA-10](/assets/images/monitor10-2d17d7d8ca6a8d25231e381d60df8f39.png) ###### 1.2.5 Configure Dashboard[​](#125-configure-dashboard "Direct link to 1.2.5 Configure Dashboard") 1. Download the corresponding Dashboard template based on your StarRocks version. * [Dashboard template for All Architecture](https://releases.starrocks.io/resources/Dashboard-All-Arch-20260113.json) * [Dashboard template for Shared-data Cluster - General](https://releases.starrocks.io/resources/Dashboard-Shared-data-General-3.5.json) * [Dashboard template for Shared-data Cluster - Starlet](https://releases.starrocks.io/resources/Dashboard-Shared-data-Starlet-3.5.json) > **NOTE** > > The template file needs to be uploaded through the Grafana Web UI. Therefore, you need to download the template file to the machine you use to access Grafana, not the monitoring node itself. 2. Configure the Dashboard template. Click on the menu button in the upper-left corner and click **Dashboards**. ![MA-11](/assets/images/monitor11-e18189f47fceec908b80c7d127a51446.png) On the page that appears, expand the **New** button and click **Import**. ![MA-12](/assets/images/monitor12-40d0bdaad5a6097aaa9edbe3e19c2093.png) On the new page, click on **Upload Dashboard JSON file** and upload the template file you downloaded earlier. ![MA-13](/assets/images/monitor13-200a76c2389b70d7ad8799f3db961757.png) After uploading the file, you can rename the Dashboard. By default, it is named `StarRocks Overview`. Then, select the data source, which is the one you created earlier (`starrocks_monitor`). Then, click **Import**. ![MA-14](/assets/images/monitor14-d5df752b2e5dd0b98b25014c1dbf2f21.png) After the import is complete, you should see the StarRocks Dashboard displayed. ![MA-15](/assets/images/monitor15-210b53b9e827c45dffd4e1c1c37e415c.png) ###### 1.2.6 Monitor StarRocks via Grafana[​](#126-monitor-starrocks-via-grafana "Direct link to 1.2.6 Monitor StarRocks via Grafana") Log in to the Grafana Web UI, click on the menu button in the upper-left corner, and click **Dashboards**. ![MA-16](/assets/images/monitor16-5a986d52c1c7f54049d8d545a6a2f4a6.png) On the page that appears, select **StarRocks Overview** from the **General** directory. ![MA-17](/assets/images/monitor17-bda14636790d5f7f78a7f5565e6f771e.png) After you enter the StarRocks monitoring Dashboard, you can manually refresh the page in the upper-right corner or set the automatic refresh interval for monitoring the StarRocks cluster status. ![MA-18](/assets/images/monitor18-9a6f4f7963351ba69d2147ee5a54d88d.png) #### Step 2: Understand the core monitoring metrics[​](#step-2-understand-the-core-monitoring-metrics "Direct link to Step 2: Understand the core monitoring metrics") To accommodate the needs of development, operations, DBA, and more, StarRocks provides a wide range of monitoring metrics. This section only introduces some important metrics commonly used in business and their alert rules. For other metric details, please refer to [Monitoring Metrics](https://docs.starrocks.io/docs/administration/management/monitoring/metrics.md). ##### 2.1 Metrics for FE and BE status[​](#21-metrics-for-fe-and-be-status "Direct link to 2.1 Metrics for FE and BE status") | **Metric** | **Description** | **Alert rule** | **Note** | | ---------------- | --------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | Frontends Status | FE Node Status. The status of a live node is represented by `1`, while a node that is down (DEAD) will be displayed as `0`. | The status of all FE nodes should be alive, and any FE node with a status of DEAD should trigger an alert. | The failure of any FE or BE nodes is considered critical, and it requires prompt troubleshooting to identify the cause of failure. | | Backends Status | BE Node Status. The status of a live node is represented by `1`, while a node that is down (DEAD) will be displayed as `0`. | The status of all BE nodes should be alive, and any BE node with a status of DEAD should trigger an alert. | | ##### 2.2 Metrics for query failure[​](#22-metrics-for-query-failure "Direct link to 2.2 Metrics for query failure") | **Metric** | **Description** | **Alert rule** | **Note** | | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Query Error | The query failure (including timeout) rate within one minute. Its value is calculated as the number of failed queries in one minute divided by 60 seconds. | You can configure this based on the actual QPS of your business. 0.05, for example, can be used as a preliminary setting. You can adjust it later as needed. | Usually, the query failure rate should be kept low. Setting this threshold to 0.05 means allowing a maximum of 3 failed queries per minute. If you receive the alert from this item, you can check resource utilization or configure the query timeout appropriately. | ##### 2.3 Metrics for external operation failure[​](#23-metrics-for-external-operation-failure "Direct link to 2.3 Metrics for external operation failure") | **Metric** | **Description** | **Alert rule** | **Note** | | ------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Schema Change | The Schema Change operation failure rate. | Schema Change is a low-frequency operation. You can set this item to send an alert immediately upon failure. | Usually, Schema Change operations should not fail. If an alert is triggered for this item, you can consider increasing the memory limit of Schema Change operations, which is set to 2GB by default. | ##### 2.4 Metrics for internal operation failure[​](#24-metrics-for-internal-operation-failure "Direct link to 2.4 Metrics for internal operation failure") | **Metric** | **Description** | **Alert rule** | **Note** | | ------------------- | -------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | BE Compaction Score | The highest Compaction Score among all BE nodes, indicating the current compaction pressure. | In typical offline scenarios, this value is usually lower than 100. However, when there are a large number of loading tasks, the Compaction Score may increase significantly. In most cases, intervention is required when this value exceeds 800. | Usually, if the Compaction Score is greater than 1000, StarRocks will return an error "Too many versions". In such cases, you may consider reducing the loading concurrency and frequency. | | Clone | The tablet clone operation failure rate. | You can set this item to send an alert immediately upon failure. | If an alert is triggered for this item, you can check the status of BE nodes, disk status, and network status. | ##### 2.5 Metrics for service availability[​](#25-metrics-for-service-availability "Direct link to 2.5 Metrics for service availability") | **Metric** | **Description** | **Alert rule** | **Note** | | -------------- | ------------------------------------------------------ | --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Meta Log Count | The number of BDB metadata log entries on the FE node. | It is recommended to configure this item to trigger an immediate alert if it exceeds 100,000. | By default, the leader FE node triggers a checkpoint to flush the log to disk when the number of logs exceeds 50,000. If this value exceeds 50,000 by a large margin, it usually indicates a checkpoint failure. You can check whether the Xmx heap memory configuration is reasonable in **fe.conf**. | ##### 2.6 Metrics for system load[​](#26-metrics-for-system-load "Direct link to 2.6 Metrics for system load") | **Metric** | **Description** | **Alert rule** | **Note** | | -------------------- | --------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | BE CPU Idle | CPU idle rate of the BE node. | It is recommended to configure this item to trigger an alert if the idle rate is lower than 10% for 30 consecutive seconds. | This item is used to monitor CPU resource bottlenecks. CPU usage can fluctuate significantly, and setting a small polling interval may result in false alerts. Therefore, you need to adjust this item based on the actual business conditions. If you have multiple batch processing tasks or a large number of queries, you may consider setting a lower threshold. | | BE Mem | Memory usage for the BE node. | It is recommended to configure this item to 90% of the available memory size for each BE. | This value is equivalent to the value of Process Mem, and BE's default memory limit is 90% of the server's memory size (controlled by configuration `mem_limit` in **be.conf**). If you have deployed other services on the same server, be sure to adjust this value to avoid OOM. The alert threshold for this item should be set to 90% of BE's actual memory limit so that you can confirm whether BE memory resources have reached a bottleneck. | | Disks Avail Capacity | Available disk space ratio (percentage) of the local disks on each BE node. | It is recommended to configure this item to trigger an alert if the value is less than 20%. | It is recommended to reserve sufficient available space for StarRocks based on your business requirements. | | FE JVM Heap Stat | JVM heap memory usage percentage for each FE node in the cluster. | It is recommended to configure this item to trigger an alert if the value is greater than or equal to 80%. | If an alert is triggered for this item, it is recommended to increase the Xmx heap memory configuration in **fe.conf**; otherwise, it may affect query efficiency or lead to FE OOM issues. | #### Step 3: Configure alert via Email[​](#step-3-configure-alert-via-email "Direct link to Step 3: Configure alert via Email") ##### 3.1 Configure SMTP service[​](#31-configure-smtp-service "Direct link to 3.1 Configure SMTP service") Grafana supports various alerting solutions, such as email and webhooks. This tutorial uses email as an example. To enable email alerting, you first need to configure SMTP information in Grafana, allowing Grafana to send emails to your mailbox. Most commonly used email providers support SMTP services, and you need to enable SMTP service for your email account and obtain an authorization code. After completing these steps, modify the Grafana configuration file on the node where Grafana is deployed. ```bash vim /usr/share/grafana/conf/defaults.ini ``` Example: ```properties ###################### SMTP / Emailing ##################### [smtp] enabled = true host = user = johndoe@gmail.com # If the password contains # or ; you have to wrap it with triple quotes.Ex """#password;""" password = ABCDEFGHIJKLMNOP # The authorization password obtained after enabling SMTP. cert_file = key_file = skip_verify = true ## Verify SSL for SMTP server from_address = johndoe@gmail.com ## Address used when sending out emails. from_name = Grafana ehlo_identity = startTLS_policy = [emails] welcome_email_on_sign_up = false templates_pattern = emails/*.html, emails/*.txt content_types = text/html ``` You need to modify the following configuration items: * `enabled`: Whether to allow Grafana to send email alerts. Set this item to `true`. * `host`: The SMTP server address and port for your email, separated by a colon (`:`). Example: `smtp.gmail.com:465`. * `user`: SMTP username. * `password`: The authorization password obtained after enabling SMTP. * `skip_verify`: Whether to skip SSL verification for the SMTP server. Set this item to `true`. * `from_address`: The email address used to send alert emails. After the configuration is complete, restart Grafana. ```bash systemctl daemon-reload systemctl restart grafana-server.service ``` ##### 3.2 Create alert channel[​](#32-create-alert-channel "Direct link to 3.2 Create alert channel") You need to create an alert channel (Contact Point) in Grafana to specify how to notify contacts when an alert is triggered. 1. Log in to the Grafana Web UI, click on the menu button in the upper-left corner, expand **Alerting**, and select **Contact Points**. On the **Contact points** page, click **Add contact point** to create a new alert channel. ![MA-19](/assets/images/monitor19-6109fd0b12144f79f8f68466b5b47507.png) 2. In the **Name** field, customize the name of the contact point. Then, in the **Integration** dropdown list, select **Email**. ![MA-20](/assets/images/monitor20-8e6a36e70958a93278b465e375aea321.png) 3. In the **Addresses** field, enter the email addresses of the contacts to receive the alert. If there are multiple email addresses, separate the addresses using semicolons (`;`), commas (`,`), or line breaks. The configurations on the page can be left with their default values except for the following two items: * **Single email**: When enabled, if there are multiple contacts, the alert will be sent to them through a single email. It's recommended to enable this item. * **Disable resolved message**: By default, when the issue causing the alert is resolved, Grafana sends another notification notifying the service recovery. If you don't need this recovery notification, you can disable this item. It's not recommended to disable this option. 4. After the configuration is complete, click the **Test** button in the upper-right corner of the page. In the prompt that appears, click **Sent test notification**. If your SMTP service and address configuration are correct, the target email account should receive a test email with the subject "TestAlert Grafana". Once you confirm that you can receive the test alert email successfully, click the **Save contact point** button at the bottom of the page to complete the configuration. ![MA-21](/assets/images/monitor21-d97b1c38652ebb72514b2b5df4579fc6.png) ![MA-22](/assets/images/monitor22-4dca103ae0c8a75cfd54e43c39337abb.png) You can configure multiple notification methods for each contact point through "Add contact point integration", which will not be detailed here. For more details about Contact Points, you can refer to the [Grafana Documentation](https://grafana.com/docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/contact-points/) For subsequent demonstration, let's assume that in this step, you have created two contact points, "StarRocksDev" and "StarRocksOp", using different email addresses. ##### 3.3 Set notification policies[​](#33-set-notification-policies "Direct link to 3.3 Set notification policies") Grafana uses notification policies to associate contact points with alert rules. Notification policies use matching labels to provide a flexible way to route different alerts to different contacts, allowing for alert grouping during O\&M. 1. Log in to the Grafana Web UI, click on the menu button in the upper-left corner, expand **Alerting**, and select **Notification policies**. ![MA-23](/assets/images/monitor23-0a648b6966faea93ea94a1870594e82a.png) 2. On the **Notification policies** page, click the more (**...**) icon to the right of **Default policy** and click **Edit** to modify the Default policy. ![MA-24](/assets/images/monitor24-bf167112de0d0d807fa4e548e1d813fb.png) ![MA-25](/assets/images/monitor25-7f769039514cfe682b6360d88d18ca55.png) Notification policies use a tree-like structure, and the Default policy represents the default root policy for notification. When no other policies are set, all alert rules will default to matching this policy. It will then use the default contact point configured within it for notifications. 1. In the **Default contact point** field, select the contact point you created previously, for example, "StarRocksOp". 2. **Group by** is a key concept in Grafana Alerting, grouping alert instances with similar characteristics into a single funnel. This tutorial does not involve grouping, and you can use the default setting. ![MA-26](/assets/images/monitor26-f0f18c8a09e860caf8e1292f0a3df0ab.png) 3. Expand the **Timing options** field and configure **Group wait**, **Group interval**, and **Repeat interval**. * **Group wait**: The time waiting for the initial notification to send after the new alert creates a new group. Default 30 seconds. * **Group interval**: The interval at which alerts are sent for an existing group. Defaults to 5 minutes, which means that notifications will not be sent to this group any sooner than 5 minutes since the previous alert was sent. This means that notifications will not be sent any sooner than 5 minutes (default) since the last batch of updates were delivered, regardless of whether the alert rule interval for those alert instances was lower. Default 5 minutes. * **Repeat interval**: The waiting time to resend an alert after they have successfully been sent. The interval at which alerts are sent for an existing group. Defaults to 5 minutes, which means that notifications will not be sent to this group any sooner than 5 minutes since the previous alert was sent. You can configure the parameters as shown below so that Grafana will send the alert by these rules: 0 seconds (Group wait) after the **alert conditions are met**, Grafana will send the first alert email. After that, Grafana will re-send the alert every 1 minute (Group interval + Repeat interval). ![MA-27](/assets/images/monitor27-1555bd354163a5ba006ff518c875ff62.png) > **NOTE** > > The previous paragraph uses "meeting the alert conditions" rather than "reaching the alert threshold" to avoid false alerts. It's recommended to set the alert to be triggered a certain duration of time after the threshold has been reached. 3. After the configuration is complete, click **Update default policy**. 4. If you need to create a nested policy, click on **New nested policy** on the **Notification policies** page. Nested policies use labels to define matching rules. The labels defined in a nested policy can be used as conditions to match when configuring alert rules later. The following example configures a label as `Group=Development_team`. ![MA-28](/assets/images/monitor28-4ee3adad9577249db57b9f77193d8ea9.png) In the **Contact point** field, select "StarRocksDev". This way, when configuring alert rules with the label `Group=Development_team`, "StarRocksDev" is set to receive the alerts. You can have the nested policy inherit the timing options from the parent policy. After the configuration is complete, click **Save policy** to save the policy. ![MA-29](/assets/images/monitor29-757db2ff569cc44b57305940cc844a67.png) If you are interested in the details of notification policies or if your business has more complex alerting scenarios, you can refer to the [Grafana Documentation](https://grafana.com/docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/contact-points/) for more information. ##### 3.4 Define alert rules[​](#34-define-alert-rules "Direct link to 3.4 Define alert rules") After setting up notification policies, you also need to define alert rules for StarRocks. Log in to the Grafana Web UI, and search for and navigate to the previously configured StarRocks Overview Dashboard. ![MA-30](/assets/images/monitor30-a9c570096158e3748f7eaafb731ef8fe.png) ![MA-31](/assets/images/monitor31-39435dfb2795b8b967dbd7022ea11aee.jpeg) ###### 3.4.1 FE and BE status alert rule[​](#341-fe-and-be-status-alert-rule "Direct link to 3.4.1 FE and BE status alert rule") For a StarRocks cluster, the status of all FE and BE nodes must be alive. Any node with a status of DEAD should trigger an alert. The following example uses the Frontends Status and Backends Status metrics under StarRocks Overview to monitor FE and BE status. As you can configure multiple StarRocks clusters in Prometheus, note that the Frontends Status and Backends Status metrics are for all clusters that you have registered. ###### Configure the alert rule for FE[​](#configure-the-alert-rule-for-fe "Direct link to Configure the alert rule for FE") Follow these procedures to configure alerts for **Frontends Status**: 1. Click on the More (...) icon to the right of the **Frontends Status** monitoring item, and click **Edit**. ![MA-32](/assets/images/monitor32-ffeb20bf4d65aec43d33d6e55c77bf28.jpeg) 2. On the new page, choose **Alert**, then click **Create alert rule** from this panel to enter the rule creation page. ![MA-33](/assets/images/monitor33-2dd84ee6a971dcad5cf34fbc3289d1be.jpeg) 3. Set the rule name in the **Rule name** field. The default value is the title of the monitoring metric. If you have multiple clusters, you can add the cluster name as a prefix for differentiation, for example, "\[PROD]Frontends Status". ![MA-34](/assets/images/monitor34-462961937451b355d3dec8634360782e.jpeg) 4. Configure the alert rule as follows. 1. Choose **Grafana managed alert**. 2. For section **B**, modify the rule as `(up{group="fe"})`. 3. Click on the delete icon on the right of section **A** to remove section **A**. 4. For section **C**, modify the **Input** field to **B**. 5. For section **D**, modify the condition to `IS BELOW 1`. After completing these settings, the page will appear as shown below: ![MA-35](/assets/images/monitor35-9b9881f59ac229c3d6b340b22087cf62.jpeg) Click to view detailed instructions Configuring alert rules in Grafana typically involves three steps: 1. Retrieve the metric values from Prometheus through PromQL queries. PromQL is a data query DSL language developed by Prometheus, and it is also used in the JSON templates of Dashboards. The `expr` property of each monitoring item corresponds to the respective PromQL. You can click **Run queries** on the rule settings page to view the query results. 2. Apply functions and modes to process the result data from the above queries. Usually, you need to use the Last function to retrieve the latest value and use Strict mode to ensure that if the returned value is non-numeric data, it can be displayed as `NaN`. 3. Set rules for the processed query results. Taking FE as an example, if the FE node status is alive, the output result is `1`. If the FE node is down, the result is `0`. Therefore, you can set the rule to `IS BELOW 1`, meaning an alert will be triggered when this condition occurs. 5. Set up alert evaluation rules. According to the Grafana documentation, you need to configure the frequency for evaluating alert rules and the frequency at which their status changes. In simple terms, this involves configuring "how often to check with the alert rules" and "how long the abnormal state must persist after detection before triggering the alert (to avoid false alerts caused by transient spikes)". Each Evaluation group contains an independent evaluation interval to determine the frequency of checking the alert rules. You can create a new folder named **PROD** specifically for the StarRocks production cluster and create a new Evaluation group `01` within it. Then, configure this group to check every `10` seconds, and trigger the alert if the anomaly persists for `30` seconds. ![MA-36](/assets/images/monitor36-45dbde604bf936329236090fe7560a61.png) > **NOTE** > > The previously mentioned "Disable resolved message" option in the alert channel configuration section, which controls the timing of sending emails for cluster service recovery, is also influenced by the "Evaluate every" parameter above. In other words, when Grafana performs a new check and detects that the service has recovered, it sends an email to notify the contacts. 6. Add alert annotations. In the **Add details for your alert rule** section, click **Add annotation** to configure the content of the alert email. Please note not to modify the **Dashboard UID** and **Panel ID** fields. ![MA-37](/assets/images/monitor37-af96a30ba8b1a14ec11923250de8aa6d.jpeg) In the **Choose** drop-down list, select **Description**, and add the descriptive content for the alert email, for example, "FE node in your StarRocks production cluster failed, please check!" 7. Match notification policies. Specify the notification policy for the alert rule. By default, all alert rules match the Default policy. When the alert condition is met, Grafana will use the "StarRocksOp" contact point in the Default policy to send alert messages to the configured email group. ![MA-38](/assets/images/monitor38-249ddb362026eabbad3bdd3dba5f8f06.jpeg) If you want to use a nested policy, set the **Label** field to the corresponding nested policy, for example, `Group=Development_team`. Example: ![MA-39](/assets/images/monitor39-bcb6c9f6d736823e0dc4550a73901a1a.jpeg) When the alert condition is met, emails will be sent to "StarRocksDev" instead of "StarRocksOp" in the Default policy. 8. Once all configurations are complete, click **Save rule and exit**. ![MA-40](data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEAYABgAAD/2wBDAAMCAgMCAgMDAwMEAwMEBQgFBQQEBQoHBwYIDAoMDAsKCwsNDhIQDQ4RDgsLEBYQERMUFRUVDA8XGBYUGBIUFRT/2wBDAQMEBAUEBQkFBQkUDQsNFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBT/wAARCACkAZcDAREAAhEBAxEB/8QAHQABAAIDAQEBAQAAAAAAAAAAAAMEBQYIBwECCf/EAEQQAAEDAgIHBAUIBwkBAAAAAAABAgMEBQYRBxIVVJGT0hQXIdMTGDFBYQgWIjJRVXOyMzQ1ZHJ0gTY3RXGDobGzwSP/xAAcAQEAAQUBAQAAAAAAAAAAAAAAAgMEBQYHAQj/xABKEQACAQMBAggJBgoLAQEAAAAAAQIDBBEhBTEGEhdBUWGR0xMVUlRVgZPR0hYiMnGSlDQ1NkVzdKGxssIHFCNCcnWCs8Hw8WLh/9oADAMBAAIRAxEAPwD+dmzKbeZeSnUXBTyNmU28y8lOoDI2ZTbzLyU6gMjZlNvMvJTqAyNmU28y8lOoDI2ZTbzLyU6gMjZlNvMvJTqAyNmU28y8lOoDI2ZTbzLyU6gMjZlNvMvJTqAyNmU28y8lOoDI2ZTbzLyU6gMjZlNvMvJTqAyNmU28y8lOoDI2ZTbzLyU6gMjZlNvMvJTqAyNmU28y8lOoDI2ZTbzLyU6gMjZlNvMvJTqAyNmU28y8lOoDI2ZTbzLyU6gMjZlNvMvJTqAyNmU28y8lOoDI2ZTbzLyU6gMjZlNvMvJTqAyNmU28y8lOoDI2ZTbzLyU6gMjZlNvMvJTqAyNmU28y8lOoDI2ZTbzLyU6gMjZlNvMvJTqAyNmU28y8lOoDI2ZTbzLyU6gMjZlNvMvJTqAyNmU28y8lOoDI2ZTbzLyU6gMjZlNvMvJTqAyNmU28y8lOoDI2ZTbzLyU6gMjZlNvMvJTqAyNmU28y8lOoDI2ZTbzLyU6gMjZlNvMvJTqAyNmU28y8lOoDI2ZTbzLyU6gMjZlNvMvJTqAyNmU28y8lOoDI2ZTbzLyU6gMjZlNvMvJTqAyNmU28y8lOoDI2ZTbzLyU6gMkwPAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAb3hXQ3fMXWWG6Uk9FBTTK5GJUSPRy6qq1Vya1fei8DarDg5ebQoK4pSiovOMt8zxzJmIuNp0Lao6Uk210f+mX9XbEm+2nmyeWZH5HX/AJcO2Xwlt47t/Jl2L3j1dsSb7aebJ5Y+R1/5cO2Xwjx3b+TLsXvHq7Yk32082Tyx8jr/AMuHbL4R47t/Jl2L3j1dsSb7aebJ5Y+R1/5cO2Xwjx3b+TLsXvHq7Yk32082Tyx8jr/y4dsvhHju38mXYvePV2xJvtp5snlj5HX/AJcO2Xwjx3b+TLsXvHq7Yk32082Tyx8jr/y4dsvhHju38mXYvePV2xJvtp5snlj5HX/lw7ZfCPHdv5Muxe8ertiTfbTzZPLHyOv/AC4dsvhHju38mXYvePV2xJvtp5snlj5HX/lw7ZfCPHdv5Muxe8/EvyecSxxPe2qtkqtRVRjJpM3fBM2In+5CXBDaEU2pQfrfwnq21bN4w+xe880pqeSrqIoIm68srkYxueWaquSJ4mh1KkaUJVJvCSy/qRtNpa1r64p2lvHjVKklGK0WXJ4Sy8JZb53g3WLRHcZImvddLZEqoirG902afBco1T/cto7T2TJLN5Bf6avdHQuTLhn6OftKHen77oK/73tXGfyiXjLY/nsPs1u6HJnwz9Gy9pQ70d0Ff972rjP5Q8ZbH89h9mt3Q5M+Gfo2XtKHejugr/ve1cZ/KHjLY/nsPs1u6HJnwz9Gy9pQ70d0Ff8Ae9q4z+UPGWx/PYfZrd0OTPhn6Nl7Sh3o7oK/73tXGfyh4y2P57D7NbuhyZ8M/RsvaUO9HdBX/e9q4z+UPGWx/PYfZrd0OTPhn6Nl7Sh3o7oK/wC97Vxn8oeMtj+ew+zW7ocmfDP0bL2lDvR3QV/3vauM/lDxlsfz2H2a3dDkz4Z+jZe0od6O6Cv+97Vxn8oeMtj+ew+zW7ocmfDP0bL2lDvR3QV/3vauM/lDxlsfz2H2a3dDkz4Z+jZe0od6O6Cv+97Vxn8oeMtj+ew+zW7ocmfDP0bL2lDvR3QV/wB72rjP5Q8ZbH89h9mt3Q5M+Gfo2XtKHejugr/ve1cZ/KHjLY/nsPs1u6HJnwz9Gy9pQ70d0Ff972rjP5Q8ZbH89h9mt3Q5M+Gfo2XtKHejugr/AL3tXGfyh4y2P57D7NbuhyZ8M/RsvaUO9HdBX/e9q4z+UPGWx/PYfZrd0OTPhn6Nl7Sh3o7oK/73tXGfyh4y2P57D7NbuhyZ8M/RsvaUO9Hc9c3ZJFcrZNIq5MiY6bNyr7ETONETiVqF5s26rQt7e7hKc2opcWqstvCWXTS39JZXv9H3CvZ1rVvbuwcaVKMpyfHovEYptvCqtvCWcJN9CNDKpoBlvm3Ong6eBrve1Vd4cGgYHzcm3in4v6Qe4Hzcm3in4v6QMD5uTbxT8X9IGB83Jt4p+L+kDA+bk28U/F/SBgfNybeKfi/pAwPm5NvFPxf0gYHzcm3in4v6QMENVZJqSnfMssUjWZayMVc08cveifaDzBjwAAAAAAAAAAAdR6Fv7s7N/rf9zzunBr8VUf8AV/Ezn+1fwyfq/cjdzZzElmgttXdaj0FFSz1k+Su9FTxq92Se1ckTMpzqQprjTaS69CUYuTxFZZBJG6J7mParHtVUc1yZKi/YpNNNZR4008M/J6eAAAAAmpqSetl9HTwyTyZK7UiYrlyTxVckISlGCzJ4R6k5NJbyEmeAAAHGuHf7QWz+ai/Oh8q7Q/A63+GX7mfQPBL8otnfpqX8cTr+0fsmi/AZ+VD6H4N/iSx/RU/4Ec04cflVtX9Yrf7ki2bGaSW4bVW1FDNWRUdRLRwrqyVDInLGxfsc5EyT+pSlUhGShKSTe5Z1JqEpJtLcVCqQAAAAAAJkpJ3UzqlIZFp2uRjpkauojl9iKvsz+BDjRUuLnUkk2m0txCTIgAAAAmqqSeilWKohkp5URF1JWK12S+KLkpCMozWYvJJpreiEmRABZrrbV2uVkdZSzUkj2NlayeNWK5i+LXIip4ovuUpwqQqZcGnjTTp6CTi0k2t+q610lYqETE4g/wAN/nYv/TnPDP8ANn61R/mO2f0Yfn7/AC+5/kOOzlRiTdKj9Yl/iX/kEiMAs0tsrK6GeampJ6iKButLJFG5zY0+1yongn+Yeiy9wWrwisAAAAACRtNM+B87YnuhYqNdIjV1WqvsRV9wem8b9CMAr3L9l1f8LfztAZq4IgAAAAAAAAAA6j0Lf3Z2b/W/7nndODX4qo/6v4mc/wBq/hk/V+5G7mzmJPdNC2N6XDOje9W+rr7vgta+vY6LF1uoXTx6zWfq0ipk77XZNXPxXPw9uobXs53F3SqQjGrxU805PGcv6S5urXQzFhWjShVUsx42Fx0s46n9fabXddH81/0jXe6Y2pLZiOlhs9LVRXaGtS1UE7JFVsU9S9fpo9yNVMmoqqqfYYqlext7ONKycqcnOScWuPJNauMVuwusvXQlXrwnWfhI8TKkvm5XM5Pm6+fcQ4q0F4Wlr8RWWw0q7ansNLf7QkVW+eNPpKk8LHLl6RqoiK1zkz8Sdtti7Uada4l8yNR055STw0uK2tcNPfgjUtLfOEvp03KOHlKUW8pPn4yWmfUWaPQngWDEWLUmippaPClHRUc8dwur6Snqa6RF9LJJN4qxqL9FGtyTNMinLa99KhRlFvNaU2sR4zUI7klplvfrzEoWNCNV0qm+nCLlzJyl18yXUeRacMKYbwtimjbhitpqigq6KOpkp6WsSrZSyrmjo2y5Irm+CKir45KbPse5ubmlL+txacZNJtcVtczxzPmeDHX1GjS4kqTXzlqk84f19D957DNow0dv0nU+C4MLztkbZ1uc1e65TLrv7Kr0jazPwTW1XZ5555plkaq9o7RVjUvnWWkuKlxV5SWc/Vpu695koWtsq1pQlDPhEm3l+TLTHW1n9h+vk64MtdBhvDWJYqZzLvXsvFPNOsjlR8bIF1URqrkmS5+KIR4QXdadSvat/MioNLrckQ2TRi5Ua7XzvC49Xg5P958boN0eWixW213qutlLcayztr5LxUX30VVHM+NXt1KVU1XRJllmq5qmf2E57Y2jUq1KlvGTjCfFUVDKaTxrLemyVtZ2zo0XWa/tIptuWGs7sLnxz56zllUyVUzzy96HRzW2sPB8B4ca4d/tBbP5qL86HyrtD8Drf4ZfuZ9A8Evyi2d+mpfxxOv7R+yaL8Bn5UPofg3+JLH9FT/gRzThx+VW1f1it/uSLZsZpJ1tg7GVZeKfBeHrJfq3A+Iqa2spY8MXq2PW23ZVYq+lVzcvCTx+k5F+Hx5peWkITubmtTVam5NucZLjw6tfJ6F6+rZbW4SpUKUZunJYxp82eXo/Xu6OjpMJos0L4dvFLbocV2Cmprhd7jU0/p6m99ldkx6syo6dmayarkVF1/D/ADQuto7VuKbk7Oq3GEIvSHG3rPz5PCWV0HlG2puUpXEFlzcd/FW/DUcatp827ciphXQXhrFUVhqIY5I6S0Xiut+JZFmd9OKFHSsk9v0NZjNVdXLxUncbaubbjuX9+nGVNdDeE116vOvMQhs+nWl4CnnjRquLfTHV56sJNZ6TK4Z0QaOH4csNzu0dBDBiSWeeN1ffnUktDT+kVsbYI8l9M5qZZ66+3w95b3G09pKtUo0m80lHOIcZSljLcn/dT5scxVo29pOHh8LiylJJOTWIxeNOl8+vUeU6JsCWPEenSkwxXSJd7EtVUw+lhkVqTxsZIrHo5qp7dVF8FNl2jeV6GypXcVxanFTx0NtZWv1mMp0Kbv426fGg5Y+teoy+JMN4PvWibEGJbDh2Sy1FBeYLbCj66SoVY1Y5XOdnkmbly8MvDLwLO3r31LaFG2uKvHU4yk/mpdGEvq1LydC2lSuZQjh0+KlrvfGw361zcx7VWaHLNNZ7lge3JJbbTV4ktmv/APRXvajqP0kmSuVfFfHLP2Kpqkdq1lUp31T50owqdW6WFuMpG0hG3qUqenHjSz9bnhv/APDyvTHo8wHacGV1wsU1qtt2oK9tOyjob4twdVQqqoqyNciLHI1UzVG5plmbDsu/2hVuYQrqUoTjltw4qT36Nb092vUWF3a2tOnVUcKUGsfOy3rhprmfPp0Mi0G6NcPX7DUdyxJZYKiOtuTaGCruV67BCrck1mwMZm+WVFX2Kmr7Ez9pW2xf3NGt4G2qNNRcmow4z6m86KP7TH2NKnOE6tSOUmlq+Kt2frb/AGYMtctHGBNHVixTcLvYanEi23FbrNTRrXvp84VjRyayt9qp4+xEVVy8ci0o39/tCrb0qNRQ49PjN8VPXONMmTuLO3tFcTceMoOOFnH0lnDfr379C1jjRRgTRLTYju9wstZiWl21FbKGh7c6DszHU7J1c57Uzc5NfVRF+xM/eU7LaW0Npyo0KdRQk4OUpcVPOJOKwnpzZZCva29uqtdxzFcXEctfSjnV78cxpfysWsbplrGxo5GJQ0aNR/1kT0DMs/iZTgxl2Gu/jS/eUNrY41Hi7uJH/k9GuugrR5YLRsa5V1so7psdKxbxPfNSq7QseuiJSKmr6L3Z555cTCLbO0a1SVajGTip44qhlYTx9PfxvVgu6FjaxhSjXazOOW+NhrOdy3NLr6zWbzoxwhR6Ge8WOzVbW1lBFRwWp0kupT1qyOY+qV+eaxLq5tRVyVVyL6O0bx7S8WeEWks8bT6OM8XHldPPjUt6NrQnZ/1yUfopprXDlnClv3dK9R6Rdo7DZMOY0S42ea/OiwnaamSStuMrnvY5ckia5c1YiOTWzb9uRgIuvVrUvBTUP7aa0it+N76ejUv7LiKEHNcb+wzq+bXKXRnq3cxxqviq5Jkn2HWDUXv0MRiD/Df52L/05zwz/Nn61R/mO1/0Yfn7/L7n+Q47OVGJN0qP1iX+Jf8AkEiMA6W0i6SMWaIu7qx4ErJbZZJLHR10UdJEituVRLmsrpPD/wCiq76OqvsLupKX9fqUksxi1GMeZxwv3vOvOUYJK0hUejkm5PnTy9M//On/ACTYZ0dW3EV7xHe8cYIobLUVV6ZSLT1t62ZR0z3MR8kcLG60kkv0kdq/VTPL4FK3o0+LTpp/Sbx04UsYiv8A5emvUidarNudSX92KfQs4zmT5srXTrbIL7ovwDovs+Orjd7DV4nSy4oZaaOFbg+m1oXwq9Ee5ieOX2oiKqonsTNC3o1IulSlNZcpTi/9L346err6i5lTbq1VF6RhCX2ub9v7CxjvRFgLRHDijEVfZK3EtsZdaW32+0rXugSnbLStqHOklYms7LW1W/5eOZXklR4tOercpLPVF8y6dezUpqLqpVY6LiRljrk2tX0aftwVcEaIsDXGhxfimpoZIrRS18FDb7Riu57K9F6SNJHLLKxHKqongxE+snip6oqnSjx9W5SW/Gi6vK115ljJBPwlR8XRKKfTq3jf0aaPn3FzH2F8PYS0L6SKLC1xiuVmfd7VPEsNQlQkLnxvV0XpE+tqrmiL70yzLW4b4lGD1Uas0n0rwa1/4fWme0ceFqyW904Nrfh8d/8AvrOYCoCvcv2XV/wt/O0BmrgiAAAAAAAAAADqPQt/dnZv9b/ued04Nfiqj/q/iZz/AGr+GT9X7kbubOYk3PAml3EujqmqaS0VcLrfUvSSWhrKaOohc9EyR+q9Fyd8Uy9iGJvdl2t/JTrL5y3NNp9qLqhc1bZvwb0e9b0/UZCl0941p7/c7vNdI7hUXKJkFXDXUsU0EjGfUb6JW6qI1VzTJE/3UoPYlk6MaMYYUXlNNppve85zl85Wd/cOr4Vy1xjGFjHRjoKs+mzGtRjGhxW+9u2/Qwdlp6ttNC3UiycmrqIzUVPpO9qL7fghVjsiyjbztVT+ZJ5ay9+muc55luZCd5XqVIVZS1hu0Wm/q62U8N6VsTYVvtzu1DcEdVXRXLXsqYWTRVWs5XL6SNyK1fFVX2eGfhkVK+zbW4owoTj82H0cNprGmj3kY3daNZ11L5zznrzv03f90KGMsb3nH142le6tKqqSNsMerG2NkUbc9VjGNREa1M18EQr2lnRsafg6Cws5erbb6W3zka9xUuGnUe7RcyS9RkU0s4sTGdLixLu5t/po2QxVbYY25MazURqsRuqqavh4p4+8oeLLT+rytOJ8yTbay97eenO89lc1pSpzctYJKL00Sz73vM1W/KJ0gV1TBK+9sZ6D03oY46GnayJJWaj0aiR5ZK3w/wB/b4lnHYOzoxcfB78J/Olrh5XP0+7cXL2jcuUZcbc+MtFo8NZ3dDfbneQW/T3ja14cjs1PdWtp4oFpIah1NE6phgVMljZMrddrcvj4e7InV2LY1qzrzhq3lrLw30tZwyFG+uKEFCEt27RNr6m93/cHnpnTHgA41w7/AGgtn81F+dD5V2h+B1v8Mv3M+geCX5RbO/TUv44nX9o/ZNF+Az8qH0Pwb/Elj+ip/wACOacOPyq2r+sVv9yRbNjNJPS7N8orHdhsdPbKS7R6lLF6ClqpaSGSpp48stRkrmq5Ey/qnuyNfrbCsLiq6s4PMtWstJvpaTMjRv69CChB6Ldonj6slfDenzGuFbVTUFvukKNpJHS0tRUUcM89Ornaz0ZI9qqiOVVzT35lS42LZXNR1KkN6w0m0mlospNbuYhTva9JNRlz51SeG97Wed85hrZpRxRZ6LEVJR3Z8FPiHW2nG2KNUqM1dn7W/Qz1nfVy9uRdVNm2tXwTnDPgvo6vTGOvXct+SEbutCc6kZYc853a5z73uMnhHThjDBNljtdruELaSBzn0vaaSKd1I931nROe1VYq/AtrrZFneVPC1YvL0eG1ldDw9SVC7rW8eLB6ZzhpPD6Vnca7YMZ3rDOJ48RW2udDeWPfKlW9jZXaz0VHKqPRUVVRy+1PeZCvaULig7apH5jWMbtFu3FFV6ka3h0/nZznrMrgrS1inR7HcY7Hcm00VeqOnilp45mOemeq9GvaqI5M/ahbXezLW+4nh454u7Vp/VlNaFSldVqM5VIPWW/Ra+rcWrnpxxzeJZZKq/yPllq4K58jIIo3rPC1GRv1msRUVGoiZJ4L78yjT2NYUsKFPcmt73S3rV8//hUnfXFSLjOecpJ7v7ryux653nzGmmnFmPrS22XauhWi9Kk8sVLSRU/aJUTL0kmo1Nd3j7/D4HtpsizsqnhaMddyy28LoWXoe1b6vWg4Tej36LLxuy+c+YP0zYrwLZnWu1VsDaRJu0wJU0kU7qaZUyWSJXtXUcqeHgSutlWt5UVWrF5xjRtZXQ8PVFKhc1LeMoQ3PXVJ67s685Qvmk/E2JKGvo7jcu0U1dcNq1DPQRN16nV1fSZtaip4eGqmSfArUNn21tOE6UMOEeKtXos5xq+np1JVbyvXU1Ulnj4zotcLC/Z0GeoPlCY7objda3bEdRNc1Y+pbU0UEkbpGMRjJEYrNVrmtaiZoiexM8yxnsKwnCFPiYUc4w2nq8tZznDJxv7iM3PjZyknlLDS3abtDUsXYxvGPb5JeL9WLX3KRjGPnWNjFcjWo1vg1ET2InjkZO1tKFlT8Dbx4sdXjXn+st69xVuJKVV5aWPUbTDp8xvBhpLI27M7O2mWjbUrSxLVNp1TJYkm1ddG5fHP4mPnsWxnW8O4a5y1l4b6Ws4yXFK+r0aapwlotFospdT5v+4KFRpkxhVw1MMt31qaotzLVJTdlhSFaZv1WJHqarcvFUciI7NfaVlsqzTUuJqpcfOXnjdOc59W7qIRvK8FFRlok4pYWMPfpuf1vXrLNDpzxpQXaS4suzJJ5aGO2yNlpIXRyU7PqMcxWaq5fbln8SnU2NZVKbpuGnGct7zxnvaec/8AB5Su61GUJwl9GPFX+Ho6/XqaGq5qqmaLRvLyYjEH+G/zsX/pznhn+bP1qj/Mdr/ow/P3+X3P8hx2cqMSbpUfrEv8S/8AIJEYB6LhH5QmPMD2KGz2q9NZQUyqtKyppIah1KqrmqxOkY5Wf0XLxKkqk5JJvdpnnx0Z/wC9RGMVHOOfXHNn6iHD2nfGuGaathpbrHUJV1a3Bz6+khqnx1S+CzsdK1yseqe9CMZOEYwjpxd3SulZ6+clL58pTlrxt/Xjdp1cxjMRaVcU4rorrSXW6dqp7pXtudYzs8TPS1LWaiPza1FT6PhkmSfApqKjGMVui219ct/b/wCE+PLMpZ1kkn9S3Gbt/wAofHlvu9xuO2IqqW4NhbVQ1dDBLBL6JqNid6JzNRHNREyciIvh45lbwktdd7cvW+ddDKfFWIroXF9XQ+n1la06dca2m+Xi6pd21tReHNfcIq+miqIalW/VV0T2q1NX3ZImWWSeBGEnCHEW7OenXp15z2Xzpcd78Y9XR9RRxDpexdiujvFJdbw6sp7tPDUVkboYkR74m6seWTU1EangjW5J8Cm4qSSfM3L1tYb69NNT1PEnJc6UfUnlLq11NPJHhXuX7Lq/4W/naAzVwRAAAAAAAAAAB7Jo600WjCmEqO011HWvmp1k+nTsY5rkc9zve5MvrZf0Oj7H4S2thZQtq0JNxzuw97b52uk1i92XVuK8qsJLDxvz0Y6DZPWJw3uV25UfmGa+WNh5E+yPxFh4kuPKj2v3D1icN7lduVH5g+WNh5E+yPxDxJceVHtfuHrE4b3K7cqPzB8sbDyJ9kfiHiS48qPa/cPWJw3uV25UfmD5Y2HkT7I/EPElx5Ue1+4esThvcrtyo/MHyxsPIn2R+IeJLjyo9r9w9YnDe5XblR+YPljYeRPsj8Q8SXHlR7X7h6xOG9yu3Kj8wfLGw8ifZH4h4kuPKj2v3D1icN7lduVH5g+WNh5E+yPxDxJceVHtfuHrE4b3K7cqPzB8sbDyJ9kfiHiS48qPa/cPWJw3uV25UfmD5Y2HkT7I/EPElx5Ue1+4/MvyisPpE9YqC5ukyXVa+ONEVfdmuuuXAjLhjY4fFpzz9S+I9WxLjOsl+33Hg9hkZDfLdJI5rI2VMbnOcuSIiOTNVU4nfRlO0qxistxl+5nWuDFanb7esK1aSjCNam228JJTi223oklq29x0pQaUsPU1DTQvrotaNjWLqzR5eCZKqLrJ/wAG9bI4eWlhs62tK1nXcqcIxeILGYxSePnrTToRnuEf9Gs9r7bvdpW+2bJQrVak0nX1SnNySeINZw9cNrPOT97OG9+Zzo+sy3KPYeaXHs4/Ga3yS3Hpqw+8Pux3s4b35nOj6xyj2Hmlx7OPxjkluPTVh94fdjvZw3vzOdH1jlHsPNLj2cfjHJLcemrD7w+7Hezhvfmc6PrHKPYeaXHs4/GOSW49NWH3h92O9nDe/M50fWOUew80uPZx+Mcktx6asPvD7sd7OG9+Zzo+sco9h5pcezj8Y5Jbj01YfeH3Y72cN78znR9Y5R7DzS49nH4xyS3Hpqw+8Pux3s4b35nOj6xyj2Hmlx7OPxjkluPTVh94fdjvZw3vzOdH1jlHsPNLj2cfjHJLcemrD7w+7Hezhvfmc6PrHKPYeaXHs4/GOSW49NWH3h92O9nDe/M50fWOUew80uPZx+Mcktx6asPvD7sd7OG9+Zzo+sco9h5pcezj8Y5Jbj01YfeH3Y72cN78znR9Y5R7DzS49nH4xyS3Hpqw+8Pux3s4b35nOj6xyj2Hmlx7OPxjkluPTVh94fdjvZw3vzOdH1jlHsPNLj2cfjHJLcemrD7w+7Hezhvfmc6PrHKPYeaXHs4/GOSW49NWH3h92O9nDe/M50fWOUew80uPZx+Mcktx6asPvD7sxeItKNilo2TU9XHLJSypUei9NHnJqoqo1PpL4r7PYatt7hdQ2zKyhb2taLpV6dRuUElxY5zuk9defC6ze+DPA2PBO32vc3e1bSr4azr0oxp1uNJzkotaOMc/Ra0y8tJI5qMKclNjfiCjkcr1SdquXNURjVy/rrA9yfNu0X7xy29QGRt2i/eOW3qAyNu0X7xy29QGRt2i/eOW3qAyNu0X7xy29QGRt2i/eOW3qAyNu0X7xy29QGRt2i/eOW3qAyV6+8009FLDEkqvkREze1ERMlRftX7AMmFB4ACfs0e9Q8H9IA7NHvUPB/SAOzR71Dwf0gDs0e9Q8H9IA7NHvUPB/SAOzR71Dwf0gDs0e9Q8H9IA7NHvUPB/SAOzR71Dwf0gDs0e9Q8H9IA7NHvUPB/SAOzR71Dwf0gDs0e9Q8H9IA7NHvUPB/SAOzR71Dwf0gDs0e9Q8H9IA7NHvUPB/SAOzR71Dwf0gDs0e9Q8H9IA7NHvUPB/SAOzR71Dwf0gDs0e9Q8H9IA7NHvUPB/SAOzR71Dwf0gDs0e9Q8H9IA7NHvUPB/SAOzR71Dwf0gDs0e9Q8H9IA7NHvUPB/SAOzR71Dwf0gDs0e9Q8H9IA7NHvUPB/SAOzR71Dwf0gDs0e9Q8H9IA7NHvUPB/SAOzR71Dwf0gDs0e9Q8H9IA7NHvUPB/SAOzR71Dwf0gDs0e9Q8H9IA7NHvUPB/SAOzR71Dwf0gDs0e9Q8H9IA7NHvUPB/SAOzR71Dwf0gDs0e9Q8H9IA7NHvUPB/SAOzR71Dwf0gHx1MiRveyaOXVTNUbrIqJnl70T7UAIQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAT036Gq/DT87QCAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAE9N+hqvw0/O0AgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABPTfoar8NPztAIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAT036Gq/DT87QCAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAE9N+hqvw0/O0AgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABPTfoar8NPztAIAAAf/2Q==) ###### Test alert trigger[​](#test-alert-trigger "Direct link to Test alert trigger") You can manually stop an FE node to test the alert. At this point, the heart-shaped symbol to the right of Frontends Status will change from green to yellow and then to red. **Green**: Indicates that during the last periodic check, the status of each instance of the metric item was normal, and no alert was triggered. The green status does not guarantee that the current node is in normal status. There may be a delay in status change after a node service anomaly, but typically, the delay is not in the order of minutes. ![MA-41](/assets/images/monitor41-d6555dd45cee2a87e315e538a7fff036.png) **Yellow**: Indicates that during the last periodic check, an instance of the metric item was found abnormal, but the abnormal state duration has not yet reached the "Duration" configured above. At this point, Grafana will not send an alert and will continue periodic checks until the abnormal state duration reaches the configured "Duration". During this period, if the status is restored, the symbol will change back to green. ![MA-42](/assets/images/monitor42-44ec0dd64f7e71b1f398710742990b94.jpeg) **Red**: When the abnormal state duration reaches the configured "Duration", the symbol changes to red, and Grafana will send an email alert. The symbol will remain red until the abnormal state is resolved, at which point it will change back to green. ![MA-43](/assets/images/monitor43-859a3b060d8b224da48e1cf53f593e18.jpeg) ###### Manually pause Alerts[​](#manually-pause-alerts "Direct link to Manually pause Alerts") Suppose the anomaly requires an extended period for resolution or the alerts are continuously triggered for some reasons other than anomaly. You can temporarily pause the evaluation of the alert rule to prevent Grafana from persistently sending alert emails. Navigate to the Alert tab corresponding to the metric item on the Dashboard and click the edit icon: ![MA-44](/assets/images/monitor44-f370c012a1066414412bf5024cfba667.png) In the **Alert Evaluation Behavior** section, toggle the **Pause Evaluation** switch to the ON position. ![MA-45](/assets/images/monitor45-2ef24b9bf18cc0429210e63aff8a9554.png) > **NOTE** > > After pausing the evaluation, you will receive an email notifying you that the service is restored. ###### Configure the alert rule for BE[​](#configure-the-alert-rule-for-be "Direct link to Configure the alert rule for BE") You can follow the above process to configure alert rules for BE. Editing the configuration for the metric item Backends Status: 1. In the **Set an alert rule name** section, configure the name as "\[PROD]Backends Status". 2. In the **Set a query and alert condition** section, set PromSQL to `(up{group="be"})`, and use the same settings as those in the FE alert rule for other items. 3. In the **Alert evaluation behavior** section, choose the **PROD** directory and Evaluation group **01** created earlier, and set the duration to 30 seconds. 4. In the **Add details for your alert rule** section, click **Add annotation**, select **Description**, and input the alert content, for example, "BE node in your StarRocks production cluster failed, please check! Stack information for BE failure will be printed in the BE log file **be.out**. You can identify the cause based on the logs". 5. In the **Notifications** section, configure **Labels** the same as the FE alert rule. If Labels are not configured, Grafana will use the Default policy and send alert emails to the "StarRocksOp" alert channel. ###### 3.4.2 Query alert rule[​](#342-query-alert-rule "Direct link to 3.4.2 Query alert rule") The metric item for query failures is **Query Error** under **Query Statistic**. Configure the alert rule for the metric item "Query Error" as follows: 1. In the **Set an alert rule name** section, configure the name as "\[PROD] Query Error". 2. In the **Set a query and alert condition** section, remove section **B**. Set **Input** in section **A** to **C**. In section **C**, use the default value for PromQL, which is `rate(starrocks_fe_query_err{job="StarRocks_Cluster01"}[1m])`, representing the number of failed queries per minute divided by 60s. This includes both failed queries and queries that exceeded the timeout limit. Then, in section **D**, configure the rule as `A IS ABOVE 0.05`. 3. In the **Alert evaluation behavior** section, choose the **PROD** directory and Evaluation group **01** created earlier, and set the duration to 30 seconds. 4. In the **Add details for your alert rule** section, click **Add annotation**, select **Description**, and input the alert content, for example, "High query failure rate, please check the resource usage or configure query timeout reasonably. If queries are failing due to timeouts, you can adjust the query timeout by setting the system variable `query_timeout`". 5. In the **Notifications** section, configure **Labels** the same as the FE alert rule. If Labels are not configured, Grafana will use the Default policy and send alert emails to the "StarRocksOp" alert channel. ###### 3.4.3 User operation failure alert rule[​](#343-user-operation-failure-alert-rule "Direct link to 3.4.3 User operation failure alert rule") This item monitors the rate of Schema Change operation failures, corresponding to the metric item **Schema Change** under **BE tasks**. It should be configured to alert when greater than 0. 1. In the **Set an alert rule name** section, configure the name as "\[PROD] Schema Change". 2. In the **Set a query and alert condition** section, remove section **A**. Set **Input** in section **C** to **B**. In section **B**, use the default value for PromQL, which is `irate(starrocks_be_engine_requests_total{job="StarRocks_Cluster01", type="create_rollup", status="failed"}[1m])`, representing the number of failed Schema Change tasks per minute divided by 60s. Then, in section **D**, configure the rule as `C IS ABOVE 0`. 3. In the **Alert evaluation behavior** section, choose the **PROD** directory and Evaluation group **01** created earlier, and set the duration to 30 seconds. 4. In the **Add details for your alert rule** section, click **Add annotation**, select **Description**, and input the alert content, for example, "Failed Schema Change tasks detected, please check promptly. You can increase the memory limit available for Schema Change by adjusting the BE configuration parameter `memory_limitation_per_thread_for_schema_change`, which is set to 2GB by default". 5. In the **Notifications** section, configure **Labels** the same as the FE alert rule. If Labels are not configured, Grafana will use the Default policy and send alert emails to the "StarRocksOp" alert channel. ###### 3.4.4 StarRocks operation failure alert rule[​](#344-starrocks-operation-failure-alert-rule "Direct link to 3.4.4 StarRocks operation failure alert rule") ###### BE Compaction Score[​](#be-compaction-score "Direct link to BE Compaction Score") This item corresponds to **BE Compaction Score** under **Cluster Overview**, and is used to monitor the compaction pressure on the cluster. 1. In the **Set an alert rule name** section, configure the name as "\[PROD] BE Compaction Score". 2. In the **Set a query and alert condition** section, configure the rule in section C as `B IS ABOVE 0`. You can use default values for other items. 3. In the **Alert evaluation behavior** section, choose the **PROD** directory and Evaluation group **01** created earlier, and set the duration to 30 seconds. 4. In the **Add details for your alert rule** section, click **Add annotation**, select **Description**, and input the alert content, for example, "High compaction pressure. Please check whether there are high-frequency or high concurrency loading tasks and reduce the loading frequency. If the cluster has sufficient CPU, memory, and I/O resources, consider adjusting the cluster compaction strategy". 5. In the **Notifications** section, configure **Labels** the same as the FE alert rule. If Labels are not configured, Grafana will use the Default policy and send alert emails to the "StarRocksOp" alert channel. ###### Clone[​](#clone "Direct link to Clone") This item corresponds to **Clone** in **BE tasks** and is mainly used to monitor replica balancing or replica repair operations within StarRocks, which usually should not fail. 1. In the **Set an alert rule name** section, configure the name as "\[PROD] Clone". 2. In the **Set a query and alert condition** section, remove section A. Set **Input** in section **C** to **B**. In section **B**, use the default value for PromQL, which is `irate(starrocks_be_engine_requests_total{job="StarRocks_Cluster01", type="clone", status="failed"}[1m])`, representing the number of failed Clone tasks per minute divided by 60s. Then, in section **D**, configure the rule as `C IS ABOVE 0`. 3. In the **Alert evaluation behavior** section, choose the **PROD** directory and Evaluation group **01** created earlier, and set the duration to 30 seconds. 4. In the **Add details for your alert rule** section, click **Add annotation**, select **Description**, and input the alert content, for example, "Detected a failure in the clone task. Please check the cluster BE status, disk status, and network status". 5. In the **Notifications** section, configure **Labels** the same as the FE alert rule. If Labels are not configured, Grafana will use the Default policy and send alert emails to the "StarRocksOp" alert channel. ###### 3.4.5 Service availability alert rule[​](#345-service-availability-alert-rule "Direct link to 3.4.5 Service availability alert rule") This item monitors the metadata log count in BDB, corresponding to the **Meta Log Count** monitoring item under **Cluster Overview**. 1. In the **Set an alert rule name** section, configure the name as "\[PROD] Meta Log Count". 2. In the **Set a query and alert condition** section, configure the rule in section **C** as `B IS ABOVE 100000`. You can use default values for other items. 3. In the **Alert evaluation behavior** section, choose the **PROD** directory and Evaluation group **01** created earlier, and set the duration to 30 seconds. 4. In the **Add details for your alert rule** section, click **Add annotation**, select **Description**, and input the alert content, for example, "Detected that the metadata count in FE BDB is significantly higher than the expected value, which can indicate a failed Checkpoint operation. Please check whether the Xmx heap memory configuration in the FE configuration file **fe.conf** is reasonable". 5. In the **Notifications** section, configure **Labels** the same as the FE alert rule. If Labels are not configured, Grafana will use the Default policy and send alert emails to the "StarRocksOp" alert channel. ###### 3.4.6 System overload alert rule[​](#346-system-overload-alert-rule "Direct link to 3.4.6 System overload alert rule") ###### BE CPU Idle[​](#be-cpu-idle "Direct link to BE CPU Idle") This item monitors the CPU idle rate on BE nodes. 1. In the **Set an alert rule name** section, configure the name as "\[PROD] BE CPU Idle". 2. In the **Set a query and alert condition** section, configure the rule in section C as `B IS BELOW 10`. You can use default values for other items. 3. In the **Alert evaluation behavior** section, choose the **PROD** directory and Evaluation group **01** created earlier, and set the duration to 30 seconds. 4. In the **Add details for your alert rule** section, click **Add annotation**, select **Description**, and input the alert content, for example, "Detected that BE CPU load is consistently high. It will impact other tasks in the cluster. Please check whether the cluster is abnormal or if there is a CPU resource bottleneck". 5. In the **Notifications** section, configure **Labels** the same as the FE alert rule. If Labels are not configured, Grafana will use the Default policy and send alert emails to the "StarRocksOp" alert channel. ###### BE Memory[​](#be-memory "Direct link to BE Memory") This item corresponds to **BE Mem** under **BE**, monitoring the memory usage on BE nodes. 1. In the **Set an alert rule name** section, configure the name as "\[PROD] BE Mem". 2. In the **Set a query and alert condition** section, configure PromSQL as `starrocks_be_process_mem_bytes{job="StarRocks_Cluster01"}/(*1024*1024*1024)`, where `` needs to be replaced with the current BE node's available memory limit, that is, the server's memory size multiplied by the value of the BE configuration item `mem_limit`. Example: `starrocks_be_process_mem_bytes{job="StarRocks_Cluster01"}/(49*1024*1024*1024)`. Then, in section **C**, configure the rule as `B IS ABOVE 0.9`. 3. In the **Alert evaluation behavior** section, choose the **PROD** directory and Evaluation group **01** created earlier, and set the duration to 30 seconds. 4. In the **Add details for your alert rule** section, click **Add annotation**, select **Description**, and input the alert content, for example, "Detected that BE memory usage is consistently high. To prevent query failure, please consider expanding memory size or adding BE nodes". 5. In the **Notifications** section, configure **Labels** the same as the FE alert rule. If Labels are not configured, Grafana will use the Default policy and send alert emails to the "StarRocksOp" alert channel. ###### Disks Avail Capacity[​](#disks-avail-capacity "Direct link to Disks Avail Capacity") This item corresponds to **Disk Usage** under **BE**, monitoring the remaining space ratio in the directory where the BE storage path is located. 1. In the **Set an alert rule name** section, configure the name as "\[PROD] Disks Avail Capacity". 2. In the **Set a query and alert condition** section, configure the rule in section **C** as `B`` IS BELOW 0.2`. You can use default values for other items. 3. In the **Alert evaluation behavior** section, choose the **PROD** directory and Evaluation group **01** created earlier, and set the duration to 30 seconds. 4. In the **Add details for your alert rule** section, click **Add annotation**, select **Description**, and input the alert content, for example, "Detected that BE disk available space is below 20%, please release disk space or expand the disk". 5. In the **Notifications** section, configure **Labels** the same as the FE alert rule. If Labels are not configured, Grafana will use the Default policy and send alert emails to the "StarRocksOp" alert channel. ###### FE JVM Heap Stat[​](#fe-jvm-heap-stat "Direct link to FE JVM Heap Stat") This item corresponds to **Cluster FE JVM Heap Stat** under **Overview**, monitoring the proportion of FE's JVM memory usage to FE heap memory limit. 1. In the **Set an alert rule name** section, configure the name as "\[PROD] FE JVM Heap Stat". 2. In the **Set a query and alert condition** section, configure the rule in section **C** as `B IS ABOVE 80`. You can use default values for other items. 3. In the **Alert evaluation behavior** section, choose the **PROD** directory and Evaluation group **01** created earlier, and set the duration to 30 seconds. 4. In the **Add details for your alert rule** section, click **Add annotation**, select **Description**, and input the alert content, for example, "Detected that FE heap memory usage is high, please adjust the heap memory limit in the FE configuration file **fe.conf**". 5. In the **Notifications** section, configure **Labels** the same as the FE alert rule. If Labels are not configured, Grafana will use the Default policy and send alert emails to the "StarRocksOp" alert channel. #### Appendix[​](#appendix "Direct link to Appendix") ##### Enable Service Detection for Prometheus[​](#enable-service-detection-for-prometheus "Direct link to Enable Service Detection for Prometheus") You can enable Service Detection for Prometheus so that it can automatically detect the services (nodes) after the cluster is scaled in or out. note The following section uses AWS as an example. 1. Grant the EC2 instance that hosts your Prometheus service the following permissions using IAM Policy: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ec2:DescribeInstances", "ec2:DescribeTags" ], "Resource": "*" } ] } ``` For detailed instructions of authentication to AWS resources, see [Authenticate to AWS resources](https://docs.starrocks.io/docs/integrations/authenticate_to_aws_resources.md). With these permissions, Prometheus is able to list the instances and their tags in the region. 2. Add the `ec2_sd_configs` and `relabel_configs` sections to **prometheus/prometheus.yml**. Example: ```yaml global: scrape_interval: 15s # Set the global scrape interval to 15s. The default is 1 min. evaluation_interval: 15s # Set the global rule evaluation interval to 15s. The default is 1 min. scrape_configs: - job_name: 'StarRocks_Cluster01' metrics_path: '/metrics' ec2_sd_configs: - region: us-west-2 port: 8030 filters: - name: tag:ClusterName values: ['test-stage-20251021'] - name: tag:ProcessType values: ['FE'] - region: us-west-2 port: 8040 filters: - name: tag:ClusterName values: ['test-stage-20251021'] - name: tag:ProcessType values: ['BE'] relabel_configs: - source_labels: [__meta_ec2_tag_ClusterName] regex: test-stage-20251021 target_label: cluster replacement: test-stage-20251021 - source_labels: [__meta_ec2_tag_ProcessType] regex: FE target_label: group replacement: fe - source_labels: [__meta_ec2_tag_ProcessType] regex: BE target_label: group replacement: be ``` #### Q\&A[​](#qa "Direct link to Q\&A") ##### Q: Why can't the Dashboard detect anomalies?[​](#q-why-cant-the-dashboard-detect-anomalies "Direct link to Q: Why can't the Dashboard detect anomalies?") A: Grafana Dashboard relies on the system time of the server it is hosted on to fetch values for monitoring items. If the Grafana Dashboard page remains unchanged after a cluster anomaly, you can check if the servers' system clock is synchronized and then perform cluster time calibration. ##### Q: How can I implement alert grading?[​](#q-how-can-i-implement-alert-grading "Direct link to Q: How can I implement alert grading?") A: Taking the Query Error item as an example, you can create two alert rules for it with different alert thresholds. For example: * **Risk Level**: Set the failure rate greater than 0.05, indicating a risk. Send the alert to the development team. * **Severity Level**: Set the failure rate greater than 0.20, indicating a severity. At this point, the alert notification will be sent to both the development and operations teams simultaneously. ##### Q: How can I retrieve more detailed metrics, including table-level metrics, materialized view metrics, and connection statistics with user labels?[​](#q-how-can-i-retrieve-more-detailed-metrics-including-table-level-metrics-materialized-view-metrics-and-connection-statistics-with-user-labels "Direct link to Q: How can I retrieve more detailed metrics, including table-level metrics, materialized view metrics, and connection statistics with user labels?") A: By default, the `/metrics` endpoint collects metrics in a minified mode to minimize performance impact. To retrieve detailed metrics, you need to add specific parameters to the request and provide Basic Authentication credentials for a user with ADMIN privileges. **Supported Parameters:** * `with_table_metrics=all`: Collects all table-level metrics. * `with_materialized_view_metrics=all`: Collects all materialized view metrics. * `with_user_connections=all`: Collects connection statistics categorized by user labels. **Authentication Requirement:** These parameters take effect only when the request includes valid Basic Authentication credentials for an ADMIN user. If the request is anonymous or the user lacks ADMIN privileges, these parameters are ignored, and only default metrics are returned. **Example Curl Command:** ```bash curl -u : \ "http://:/metrics?with_table_metrics=all&with_materialized_view_metrics=all&with_user_connections=all" ``` **Prometheus Configuration Example:** To enable detailed metric collection in Prometheus, configure `params` and `basic_auth` in your `prometheus.yml`: ```yaml scrape_configs: - job_name: 'StarRocks_Detailed_Metrics' metrics_path: '/metrics' params: with_table_metrics: ['all'] with_materialized_view_metrics: ['all'] with_user_connections: ['all'] basic_auth: username: '' password: '' static_configs: - targets: [':'] ``` note Collecting all table and materialized view metrics may increase the load on the FE node. Use these parameters with caution in large-scale environments. --- ### Process Profile (Proc Profile) The **Process Profile** (Proc Profile) feature provides a built-in mechanism to collect and visualize performance profiles for StarRocks Frontend (FE) and Backend (BE) processes. By generating flame graphs for CPU and memory allocation, it helps developers and administrators diagnose performance bottlenecks, high resource utilization, and complex runtime issues directly from the Web UI. #### Overview[​](#overview "Direct link to Overview") Process Profiling is a system-level diagnostic tool that captures the state of the StarRocks processes over a period of time. Unlike Query Profiles, which focus on individual SQL execution, Process Profiles provide a holistic view of what the processes are doing, including background tasks, metadata management, and internal synchronization. #### Page Examples[​](#page-examples "Direct link to Page Examples") The Proc Profile interface is integrated into the StarRocks Web UI under the **proc profiles** tab. ##### Profile List View[​](#profile-list-view "Direct link to Profile List View") The main page displays a list of collected profile files for the selected node. You can switch between FE and different BE nodes using the tabs at the top. ![Profile List Example](/assets/images/proc_profile_1-60bd9228b396079059a710aca842f293.png) ##### Flame Graph[​](#flame-graph "Direct link to Flame Graph") ![Flame Graph](/assets/images/proc_profile_2-33e9ad52afee3e204664f4eede245e48.png) A Flame Graph is a visualization tool used to show the resource consumption distribution of functions or code paths in a program. It represents the call stack using stacked rectangles (usually in a horizontal "flame" shape), where: * **Each box represents a function (or method) call.** * **The width of a box represents the amount of resources consumed by that function (such as CPU time, memory allocation frequency, or lock wait time). The wider the box, the more resources are consumed.** * **Vertical stacking represents the calling relationship, with the bottom layer being the entry function and child functions stacked on top.** Flame graphs help developers and operations personnel quickly identify code hotspots, performance bottlenecks, and call paths with the highest resource consumption, and are commonly used for performance tuning and troubleshooting. In StarRocks Process Profile, flame graphs are used to visually display the data distribution of CPU usage, memory allocation, and (in BE scenarios) lock contention, helping to locate the most time-consuming code segments or call paths. #### Use Scenarios[​](#use-scenarios "Direct link to Use Scenarios") * **CPU Hotspot Analysis**: Identify which code paths or functions are consuming the most CPU cycles. * **Memory Allocation Profiling**: Track where memory is being frequently allocated to find potential memory pressure sources. * **R\&D Troubleshooting**: Assist developers in analyzing complex bugs or performance regressions in production environments without needing external profiling tools. #### Feature Description[​](#feature-description "Direct link to Feature Description") ##### How to Use[​](#how-to-use "Direct link to How to Use") 1. **Access**: Open the StarRocks Web UI (default port 8030) and click on the **proc profiles** tab. 2. **Select Node**: Choose the **FE** tab or a specific **BE** node tab. 3. **Visualize**: Click the **View** link for any entry. * For **FE**, it extracts and displays a pre-generated HTML flame graph. * For **BE**, it may convert a raw `pprof` file into an SVG flame graph on the fly (the first view might take a few seconds as it performs the conversion). 4. **Interact**: Use the flame graph to zoom into specific call stacks, search for function names, or analyze stack depth. ##### Information Provided[​](#information-provided "Direct link to Information Provided") * **Type**: CPU, Memory (Allocation), or Contention (BE only). * **Timestamp**: When the profile collection was completed. * **File Size**: Size of the compressed profile data. * **Flame Graph**: A hierarchical visualization where the width of each box represents the relative resource consumption (CPU time, allocation frequency, or lock wait time). #### Configuration[​](#configuration "Direct link to Configuration") The profiling feature performs function-level sampling to generate flame graphs for visualization. ##### Frontend (FE) Configuration[​](#frontend-fe-configuration "Direct link to Frontend (FE) Configuration") FE profiling is managed by an internal daemon and uses **AsyncProfiler** for data collection. You can configure these in `fe.conf`. | Parameter | Default | Description | | --------------------------------------- | ------------------ | --------------------------------------------------------------- | | `proc_profile_cpu_enable` | `true` | Whether to enable automatic CPU profiling for FE. | | `proc_profile_mem_enable` | `true` | Whether to enable automatic memory allocation profiling for FE. | | `proc_profile_collect_time_s` | `120` | Duration (seconds) for each profile collection. | | `proc_profile_jstack_depth` | `128` | Maximum Java stack depth to collect. | | `proc_profile_file_retained_days` | `1` | How many days to retain profile files. | | `proc_profile_file_retained_size_bytes` | `2147483648` (2GB) | Maximum total size of retained profile files. | ##### Backend (BE) Configuration[​](#backend-be-configuration "Direct link to Backend (BE) Configuration") BE profiles are collected using the built-in **gperftools** and are typically collected via a background script or manual triggers. The collected data is then converted into flame graphs using **pprof**. ###### Configuration in `be.conf`[​](#configuration-in-beconf "Direct link to configuration-in-beconf") | Parameter | Default | Description | | ----------------------------- | ---------------------------------- | ---------------------------------------------------------------------------------------------- | | `brpc_port` | `8060` | Port used by collection scripts to fetch data from BE. | | `sys_log_dir` | `${STARROCKS_HOME}/log` | Base directory for storing collected profiles (stored in the `proc_profile` subdirectory). | | `flamegraph_tool_dir` | `${STARROCKS_HOME}/bin/flamegraph` | Path to conversion tools (**pprof**, `flamegraph.pl`). | | `COLLECT_BE_PROFILE_INTERVAL` | `60` | Collection interval in seconds when running the `collect_be_profile.sh` script in daemon mode. | ###### Manual BE Collection Options[​](#manual-be-collection-options "Direct link to Manual BE Collection Options") The `collect_be_profile.sh` script supports the following command-line options: | Option | Default | Description | | ------------------ | ------------------ | ----------------------------------------------------------- | | `--profiling-type` | `cpu` | Type of profile to collect: `cpu`, `contention`, or `both`. | | `--duration` | `10` | Duration (seconds) for each profile collection. | | `--interval` | `60` | Interval (seconds) between collections in daemon mode. | | `--cleanup-days` | `1` | Number of days to retain profile files. | | `--cleanup-size` | `2147483648` (2GB) | Maximum total size of retained profile files. | | `--daemon` | - | Run the collection script in daemon mode in the background. | ##### Manual BE Collection Example[​](#manual-be-collection-example "Direct link to Manual BE Collection Example") You can use the provided script to trigger or schedule BE collection: ```bash # Collect a 30-second CPU profile ./bin/collect_be_profile.sh --profiling-type cpu --duration 30 # Run in daemon mode to collect profiles every hour ./bin/collect_be_profile.sh --daemon --interval 3600 ``` --- ### Add labels on BEs Since v3.2.8, StarRocks supports adding labels on BEs. When creating a table or an asynchronous materialized view, you can specify the label of a certain group of BE nodes. This ensures that data replicas are distributed only on the BE nodes associated with that label. Data replicas will be evenly distributed among nodes with the same label, enhancing data high availability and resource isolation. #### Usage[​](#usage "Direct link to Usage") ##### Add labels on BEs[​](#add-labels-on-bes-1 "Direct link to Add labels on BEs") Suppose that one StarRocks cluster includes six BEs which are distributed evenly across three racks. You can add labels on BEs based on the racks where the BEs are located. ```sql ALTER SYSTEM MODIFY BACKEND "172.xx.xx.46:9050" SET ("labels.location" = "rack:rack1"); ALTER SYSTEM MODIFY BACKEND "172.xx.xx.47:9050" SET ("labels.location" = "rack:rack1"); ALTER SYSTEM MODIFY BACKEND "172.xx.xx.48:9050" SET ("labels.location" = "rack:rack2"); ALTER SYSTEM MODIFY BACKEND "172.xx.xx.49:9050" SET ("labels.location" = "rack:rack2"); ALTER SYSTEM MODIFY BACKEND "172.xx.xx.50:9050" SET ("labels.location" = "rack:rack3"); ALTER SYSTEM MODIFY BACKEND "172.xx.xx.51:9050" SET ("labels.location" = "rack:rack3"); ``` After adding labels, you can execute `SHOW BACKENDS;` and view the labels of BEs in the `Location` field of the returned result. If you need to modify the labels of BEs, you can execute `ALTER SYSTEM MODIFY BACKEND "172.xx.xx.48:9050" SET ("labels.location" = "rack:xxx");`. ##### Use labels to specify table data distribution on BE nodes[​](#use-labels-to-specify-table-data-distribution-on-be-nodes "Direct link to Use labels to specify table data distribution on BE nodes") If you need to specify the locations to which a table's data is distributed, for example, distributing a table's data across two racks, rack1 and rack2, you can add labels to the table. After labels are added, all the replicas of the same tablet in the table are distributed across labels in a Round-Robin approach. Moreover, if multiple replicas of the same tablet exist within the same label, these replicas will be distributed as evenly as possible across different BEs in that label. note * If the total number of BE nodes associated with the labels is fewer than the number of replicas, the system will preferentially ensure there are enough replicas. In this case, replicas may not be distributed as the label specified. * The label to be associated with a table must already exist. Otherwise, an error `Getting analyzing error. Detail message: Cannot find any backend with location: rack:xxx` will occur. ###### At table creation[​](#at-table-creation "Direct link to At table creation") You can use the property `"labels.location"` to distribute the table's data across rack 1 and rack 2 at table creation: ```sql CREATE TABLE example_table ( order_id bigint NOT NULL, dt date NOT NULL, user_id INT NOT NULL, good_id INT NOT NULL, cnt int NOT NULL, revenue int NOT NULL ) PROPERTIES ("labels.location" = "rack:rack1,rack:rack2"); ``` For newly created tables, the default value of the table property `labels.location` is `*`, indicating that replicas are evenly distributed across all labels. If the data distribution of a newly created table does not need to be aware of the geographical locations of servers in the cluster, you can manually set the table property `"labels.location" = ""`. ###### After table creation[​](#after-table-creation "Direct link to After table creation") If you need to modify the data distribution location of the table after table creation, for example, modify the location to rack 1, rack 2, and rack 3, you can execute the following statement: ```sql ALTER TABLE example_table SET ("labels.location" = "rack:rack1,rack:rack2,rack:rack3"); ``` note If you have upgraded StarRocks to version 3.2.8 or later, for historical tables created before the upgrade, data is not distributed based on labels by default. If you need to distribute data of a historical table based on labels, you can execute the following statement to add labels to the historical table: ```sql ALTER TABLE example_table1 SET ("labels.location" = "rack:rack1,rack:rack2"); ``` ##### Use labels to specify materialized view data distribution on BE nodes[​](#use-labels-to-specify-materialized-view-data-distribution-on-be-nodes "Direct link to Use labels to specify materialized view data distribution on BE nodes") If you need to specify the locations to which an asynchronous materialized view's data is distributed, for example, distributing data across two racks, rack1 and rack2, you can add labels to the materialized view. After labels are added, all the replicas of the same tablet in the materialized view are distributed across labels in a Round-Robin approach. Moreover, if multiple replicas of the same tablet exist within the same label, these replicas will be distributed as evenly as possible across different BEs in that label. note * If the total number of BE nodes associated with the labels is fewer than the number of replicas, the system will preferentially ensure there are enough replicas. In this case, replicas may not be distributed as the label specified. * The labels to be associated with the materialized view must already exist. Otherwise, an error `Getting analyzing error. Detail message: Cannot find any backend with location: rack:xxx` will occur. ###### At materialized view creation[​](#at-materialized-view-creation "Direct link to At materialized view creation") If you want to distribute the materialized view's data across rack 1 and rack 2 while creating it, you can execute the following statement: ```sql CREATE MATERIALIZED VIEW mv_example_mv DISTRIBUTED BY RANDOM PROPERTIES ( "labels.location" = "rack:rack1,rack:rack2") as select order_id, dt from example_table; ``` For newly created materialized view, the default value of the property `labels.location` is `*`, indicating that replicas are evenly distributed across all labels. If the data distribution of a newly created materialized view does not need to be aware of the geographical locations of servers in the cluster, you can manually set the property `"labels.location" = ""`. ###### After materialized view creation[​](#after-materialized-view-creation "Direct link to After materialized view creation") If you need to modify the data distribution location of the materialized view after it is created, for example, modify the location to rack 1, rack 2, and rack 3, you can execute the following statement: ```sql ALTER MATERIALIZED VIEW mv_example_mv SET ("labels.location" = "rack:rack1,rack:rack2,rack:rack3"); ``` note If you have upgraded StarRocks to version 3.2.8 or later, for existing materialized views created before the upgrade, data is not distributed based on labels by default. If you need to distribute data of an existing based on labels, you can execute the following statement to add labels to the materialized view: ```sql ALTER TABLE example_mv1 SET ("labels.location" = "rack:rack1,rack:rack2"); ``` --- ### Blacklist Management In some cases, administrators need to disable certain patterns of SQL to avoid SQL from triggering cluster crashes or unexpected high concurrent queries. The blacklist is only for SELECT statements, INSERT statements (from v3.1 onwards), and CTAS statements (from v3.4 onwards). StarRocks allows users to add, view, and delete SQL blacklists. #### Syntax[​](#syntax "Direct link to Syntax") Enable SQL blacklisting via `enable_sql_blacklist`. The default is False (off). ```sql admin set frontend config ("enable_sql_blacklist" = "true"); ``` The admin user who has ADMIN\_PRIV privileges can manage blacklists by executing the following commands: ```sql ADD SQLBLACKLIST ""; DELETE SQLBLACKLIST ; SHOW SQLBLACKLIST; ``` * When `enable_sql_blacklist` is true, every SQL query needs to be filtered by sqlblacklist. If it matches, the user will be informed that the SQL is in the blacklist. Otherwise, the SQL will be executed normally. The message may be as follows when the SQL is blacklisted: `ERROR 1064 (HY000): Access denied; sql 'select count (*) from test_all_type_select_2556' is in blacklist` #### Add blacklist[​](#add-blacklist "Direct link to Add blacklist") ```sql ADD SQLBLACKLIST ""; ``` **sql** is a regular expression for a certain type of SQL. tip Currently, StarRocks supports adding SELECT statements to the SQL Blacklist. Since SQL itself contains the common characters `(`, `)`, `*`, `.` that may be mixed up with the semantics of regular expressions, we need to distinguish those by using escape characters. Given that `(` and `)` are used too often in SQL, there is no need to use escape characters. Other special characters need to use the escape character `\` as a prefix. For example: * Prohibit `count(\*)`: ```sql ADD SQLBLACKLIST "select count(\\*) from .+"; ``` * Prohibit `count(distinct)`: ```sql ADD SQLBLACKLIST "select count(distinct .+) from .+"; ``` * Prohibit order by limit `x`, `y`, `1 <= x <=7`, `5 <=y <=7`: ```sql ADD SQLBLACKLIST "select id_int from test_all_type_select1 order by id_int limit [1-7], [5-7]"; ``` * Prohibit complex SQL: ```sql ADD SQLBLACKLIST "select id_int \\* 4, id_tinyint, id_varchar from test_all_type_nullable except select id_int, id_tinyint, id_varchar from test_basic except select (id_int \\* 9 \\- 8) \\/ 2, id_tinyint, id_varchar from test_all_type_nullable2 except select id_int, id_tinyint, id_varchar from test_basic_nullable"; ``` * Prohibit all INSERT INTO statements: ```sql ADD SQLBLACKLIST "(?i)^insert\\s+into\\s+.*"; ``` * Prohibit all INSERT INTO ... VALUES statements: ```sql ADD SQLBLACKLIST "(?i)^insert\\s+into\\s+.*values\\s*\\("; ``` * Prohibit all INSERT INTO ... VALUES statements except those against the system-defined view `_statistics_.column_statistics`: ```sql ADD SQLBLACKLIST "(?i)^insert\\s+into\\s+(?!column_statistics\\b).*values\\s*\\("; ``` #### View blacklist[​](#view-blacklist "Direct link to View blacklist") ```sql SHOW SQLBLACKLIST; ``` Result format: `Index | Forbidden SQL` For example: ```sql mysql> show sqlblacklist; +-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Index | Forbidden SQL | +-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 1 | select count\(\*\) from .+ | | 2 | select id_int \* 4, id_tinyint, id_varchar from test_all_type_nullable except select id_int, id_tinyint, id_varchar from test_basic except select \(id_int \* 9 \- 8\) \/ 2, id_tinyint, id_varchar from test_all_type_nullable2 except select id_int, id_tinyint, id_varchar from test_basic_nullable | | 3 | select id_int from test_all_type_select1 order by id_int limit [1-7], [5-7] | | 4 | select count\(distinct .+\) from .+ | +-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` The SQL shown in `Forbidden SQL` is escaped for all SQL semantic characters. #### Delete blacklist[​](#delete-blacklist "Direct link to Delete blacklist") ```sql DELETE SQLBLACKLIST ; ``` `` is a list of SQL IDs separated by comma (,). For example, delete the No.3 and No.4 SQLs in the above blacklist: ```sql delete sqlblacklist 3, 4; ``` Then, the remaining sqlblacklist is as follows: ```sql mysql> show sqlblacklist; +-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Index | Forbidden SQL | +-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 1 | select count\(\*\) from .+ | | 2 | select id_int \* 4, id_tinyint, id_varchar from test_all_type_nullable except select id_int, id_tinyint, id_varchar from test_basic except select \(id_int \* 9 \- 8\) \/ 2, id_tinyint, id_varchar from test_all_type_nullable2 except select id_int, id_tinyint, id_varchar from test_basic_nullable | +-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` --- ### File manager With file manager, you can create, view, and delete files, such as the files that are used to access external data sources: public key files, private key files, and certificate files. You can reference or access the created files by using commands. #### Basic concepts[​](#basic-concepts "Direct link to Basic concepts") **File**: refers to the file that is created and saved in StarRocks. After a file is created and stored in StarRocks, StarRocks assigns a unique ID to the file. You can find a file based on the database name, catalog, and file name. In a database, only an admin user can create and delete files, and all users who have permissions to access a database can use the files that belong to the database. #### Before you begin[​](#before-you-begin "Direct link to Before you begin") * Configure the following parameters for each FE. * `small_file_dir`: the path in which the uploaded files are stored. The default path is `small_files/`, which is in the runtime directory of the FE. You need to specify this parameter in the **fe.conf** file and then restart the FE to allow the change to take effect. * `max_small_file_size_bytes`: the maximum size of a single file. The default value of this parameter is 1 MB. If the size of a file exceeds the value of this parameter, the file cannot be created. You can specify this parameter by using the [ADMIN SET CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md) statement. * `max_small_file_number`: the maximum number of files that can be created within a cluster. The default value of this parameter is 100. If the number of files that you have created reaches the value of this parameter, you cannot continue creating files. You can specify this parameter by using the ADMIN SET CONFIG statement. > Note: Increasing the values of the two parameters causes an increase in the memory usage of the FE. Therefore, we recommend that you do not increase the values of the two parameters unless necessary. * Configure the following parameters for each BE. `small_file_dir`: the path in which the downloaded files are stored. The default path is `lib/small_files/`, which is in the runtime directory of the BE. You can specify this parameter in the **be.conf** file. #### Create a file[​](#create-a-file "Direct link to Create a file") You can execute the CREATE FILE statement to create a file. For more information, see [CREATE FILE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/file/CREATE_FILE.md). After a file is created, the file is uploaded and persisted in StarRocks. #### View a file[​](#view-a-file "Direct link to View a file") You can execute the SHOW FILE statement to view the information about a file stored in a database. For more information, see [SHOW FILE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/file/SHOW_FILE.md). #### Delete a file[​](#delete-a-file "Direct link to Delete a file") You can execute the DROP FILE statement to delete a file. For more information, see [DROP FILE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/file/DROP_FILE.md). #### How an FE and a BE use a file[​](#how-an-fe-and-a-be-use-a-file "Direct link to How an FE and a BE use a file") * **FE**: The SmallFileMgr class stores the data related to the file in the specified directory of the FE. Then the SmallFileMgr class returns a local file path for the FE to use the file. * **BE**: The BE calls the **/api/get\_small\_file API** (HTTP) to download the file to its specified directory and record the information of the file. When the BE requests to use the file, the BE checks whether the file has been downloaded and then verifies the file. If the file pass the verification, the path of the file is returned. If the file fails the verification, the file is deleted and re-downloaded from the FE. When a BE restarts, it preloads the downloaded file into its memory. --- ### Load Balancing When deploying multiple FE nodes, users can deploy a load balancing layer on top of the FEs to achieve high availability. The following are some high availability options: #### Code approach[​](#code-approach "Direct link to Code approach") One way is to implement code at the application layer to perform retry and load balancing. For example, if a connection is broken, it will automatically retry on other connections. This approach requires users to configure multiple FE node addresses. #### JDBC Connector[​](#jdbc-connector "Direct link to JDBC Connector") JDBC connector supports automatic retry: ```sql jdbc:mysql:loadbalance://[host1][:port],[host2][:port][,[host3][:port]]...[/[database]][?propertyName1=propertyValue1[&propertyName2=propertyValue2]...] ``` #### ProxySQL[​](#proxysql "Direct link to ProxySQL") ProxySQL is a MySQL proxy layer that supports read/write separation, query routing, SQL caching, dynamic load configuration, failover, and SQL filtering. StarRocks FE is responsible for receiving connection and query requests, and it’s horizontally scalable and highly available. However FE requires users to set up a proxy layer on top of it to achieve automatic load balancing. See the following steps for setup: ##### 1. Install relevant dependencies[​](#1-install-relevant-dependencies "Direct link to 1. Install relevant dependencies") ```shell yum install -y gnutls perl-DBD-MySQL perl-DBI perl-devel ``` ##### 2. Download the installation package[​](#2-download-the-installation-package "Direct link to 2. Download the installation package") ```shell wget https://github.com/sysown/proxysql/releases/download/v2.0.14/proxysql-2.0.14-1-centos7.x86_64.rpm ``` ##### 3. Decompress to the current directory[​](#3-decompress-to-the-current-directory "Direct link to 3. Decompress to the current directory") ```shell rpm2cpio proxysql-2.0.14-1-centos7.x86_64.rpm | cpio -ivdm ``` ##### 4. Modify the configuration file[​](#4-modify-the-configuration-file "Direct link to 4. Modify the configuration file") ```shell vim ./etc/proxysql.cnf ``` Direct to a directory that the user has privilege to access (absolute path): ```vim datadir="/var/lib/proxysql" errorlog="/var/lib/proxysql/proxysql.log" ``` ##### 5. Start[​](#5-start "Direct link to 5. Start") ```shell ./usr/bin/proxysql -c ./etc/proxysql.cnf --no-monitor ``` ##### 6. Log in[​](#6-log-in "Direct link to 6. Log in") ```shell mysql -u admin -padmin -h 127.0.0.1 -P6032 ``` ##### 7. Configure the global log[​](#7-configure-the-global-log "Direct link to 7. Configure the global log") ```shell SET mysql-eventslog_filename='proxysql_queries.log'; SET mysql-eventslog_default_log=1; SET mysql-eventslog_format=2; LOAD MYSQL VARIABLES TO RUNTIME; SAVE MYSQL VARIABLES TO DISK; ``` ##### 8. Insert into the leader node[​](#8-insert-into-the-leader-node "Direct link to 8. Insert into the leader node") ```sql insert into mysql_servers(hostgroup_id, hostname, port) values(1, '172.xx.xx.139', 9030); ``` ##### 9. Insert the observer nodes[​](#9-insert-the-observer-nodes "Direct link to 9. Insert the observer nodes") ```sql insert into mysql_servers(hostgroup_id, hostname, port) values(2, '172.xx.xx.139', 9030); insert into mysql_servers(hostgroup_id, hostname, port) values(2, '172.xx.xx.140', 9030); ``` ##### 10. Load the configuration[​](#10-load-the-configuration "Direct link to 10. Load the configuration") ```sql load mysql servers to runtime; save mysql servers to disk; ``` ##### 11. Configure the username and password[​](#11-configure-the-username-and-password "Direct link to 11. Configure the username and password") ```sql insert into mysql_users(username, password, active, default_hostgroup, backend, frontend) values('root', '*94BDCEBE19083CE2A1F959FD02F964C7AF4CFC29', 1, 1, 1, 1); ``` ##### 12. Load the configuration[​](#12-load-the-configuration "Direct link to 12. Load the configuration") ```sql load mysql users to runtime; save mysql users to disk; ``` ##### 13. Write to the proxy rules[​](#13-write-to-the-proxy-rules "Direct link to 13. Write to the proxy rules") ```sql insert into mysql_query_rules(rule_id, active, match_digest, destination_hostgroup, mirror_hostgroup, apply) values(1, 1, '.', 1, 2, 1); ``` ##### 14. Load the configuration[​](#14-load-the-configuration "Direct link to 14. Load the configuration") ```sql load mysql query rules to runtime; save mysql query rules to disk; ``` --- ### Memory Management This section briefly introduces memory classification and StarRocks’ methods of managing memory. #### Memory Classification[​](#memory-classification "Direct link to Memory Classification") Explanation: | Metric | Name | Description | | ------------- | ------------------------------- | ----------------------------------------------------------------------------------------------- | | `process` | Total memory used of BE | | | `query_pool` | Memory used by data querying | Consists of two parts: memory used by the execution layer and memory used by the storage layer. | | `load` | Memory used by data loading | Generally MemTable | | `table_meta` | Metadata memory | S Schema, Tablet metadata, RowSet metadata, Column metadata, ColumnReader, IndexReader | | `compaction` | Multi-version memory compaction | compaction that happens after data import is complete | | `snapshot` | Snapshot memory | Generally used for clone, little memory usage | | `column_pool` | Column pool memory | Request to release column cache for accelerated column | | `page_cache` | BE's own PageCache | The default is off, the user can turn it on by modifying the BE file | #### Memory-related configuration[​](#memory-related-configuration "Direct link to Memory-related configuration") * **BE Configuration** | Name | Default | Description | | ------------------------------------------------ | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | vector\_chunk\_size | 4096 | Number of chunk rows | | mem\_limit | 90% | BE process memory upper limit. You can set it as a percentage ("80%") or a physical limit ("100G"). The default hard limit is 90% of the server's memory size, and the soft limit is 80%. You need to configure this parameter if you want to deploy StarRocks with other memory-intensive services on a same server. | | disable\_storage\_page\_cache | false | The boolean value to control whether to disable PageCache. When PageCache is enabled, StarRocks caches the recently scanned data. PageCache can significantly improve the query performance when similar queries are repeated frequently. `true` indicates to disable PageCache. Use this item together with `storage_page_cache_limit`, you can accelerate query performance in scenarios with sufficient memory resources and much data scan. The default value of this item has been changed from `true` to `false` since StarRocks v2.4. | | write\_buffer\_size | 104857600 | The capacity limit of a single MemTable, exceeding which a disk swipe will be performed. | | load\_process\_max\_memory\_limit\_bytes | 107374182400 | The upper limit of memory resources that can be taken up by all load processes on a BE node. Its value is the smaller one between `mem_limit * load_process_max_memory_limit_percent / 100` and `load_process_max_memory_limit_bytes`. If this threshold is exceeded, a flush and backpressure will be triggered. | | load\_process\_max\_memory\_limit\_percent | 30 | The maximum percentage of memory resources that can be taken up by all load processes on a BE node. Its value is the smaller one between `mem_limit * load_process_max_memory_limit_percent / 100` and `load_process_max_memory_limit_bytes`. If this threshold is exceeded, a flush and backpressure will be triggered. | | default\_load\_mem\_limit | 2147483648 | If the memory limit on the receiving side is reached for a single import instance, a disk swipe will be triggered. This needs to be modified with the Session variable `load_mem_limit` to take effect. This parameter is mutable when the Event-based Compaction Framework is enabled. | | max\_compaction\_concurrency | -1 | The maximum concurrency of compactions (both Base Compaction and Cumulative Compaction). The value -1 indicates that no limit is imposed on the concurrency. | | cumulative\_compaction\_check\_interval\_seconds | 1 | Interval of compaction check | * **Session variables** | Name | Default | Description | | ----------------- | ------- | --------------------------------------------------------------------------------------- | | query\_mem\_limit | 0 | Memory limit of a query on each BE node | | load\_mem\_limit | 0 | Memory limit of a single import task. If the value is 0, `exec_mem_limit` will be taken | #### View memory usage[​](#view-memory-usage "Direct link to View memory usage") * **`mem_tracker`** ```bash //View the overall memory statistics // View fine-grained memory statistics ``` * **`tcmalloc`** ```bash ``` ```plain ------------------------------------------------ MALLOC: 777276768 ( 741.3 MiB) Bytes in use by application MALLOC: + 8851890176 ( 8441.8 MiB) Bytes in page heap freelist MALLOC: + 143722232 ( 137.1 MiB) Bytes in central cache freelist MALLOC: + 21869824 ( 20.9 MiB) Bytes in transfer cache freelist MALLOC: + 832509608 ( 793.9 MiB) Bytes in thread cache freelists MALLOC: + 58195968 ( 55.5 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 10685464576 (10190.5 MiB) Actual memory used (physical + swap) MALLOC: + 25231564800 (24062.7 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 35917029376 (34253.1 MiB) Virtual address space used MALLOC: MALLOC: 112388 Spans in use MALLOC: 335 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). Bytes released to the OS take up virtual address space but no physical memory. ``` The memory queried by this method is accurate. However, some memory in StarRocks is reserved but not in use. TcMalloc counts the memory that is reserved, not the memory used. Here `Bytes in use by application` refers to the memory currently in use. * **metrics** ```bash curl -XGET http://be_ip:be_http_port/metrics | grep 'mem' curl -XGET http://be_ip:be_http_port/metrics | grep 'column_pool' ``` The value of metrics is updated every 10 seconds. It is possible to monitor some of the memory statistics with older versions. --- ### Query Management #### Number of user connections[​](#number-of-user-connections "Direct link to Number of user connections") `Property` is set for user granularity. To set the maximum number of connections between Client and FE, use the following command. ```sql ALTER USER '' SET PROPERTIES ("key"="value", ...) ``` User properties include the resources assigned to the user. The properties set here are for the user, not `user_identity`. That is, if two users `jack'@'%` and `jack'@'192.%` are created by the `CREATE USER` statement, then the `ALTER USER SET PROPERTIES` statement can work on the user `jack`, not `jack'@'%` or `jack'@'192.%`. Example 1: ```sql -- For the user `jack`, change the maximum number of connections to 1000 ALTER USER 'jack' SET PROPERTIES ("max_user_connections" = "1000"); -- Check the connection limit for the root user SHOW PROPERTY FOR 'root'; ``` #### Query-related session variables[​](#query-related-session-variables "Direct link to Query-related session variables") The session variables can be set by 'key' = 'value', which can limit the concurrency, memory and other query parameters in the current session. For example: * parallel\_fragment\_exec\_instance\_num The parallelism of the query with a default value of 1. It indicates the number of fragment instances on each BE. You can set this to half the number of CPU cores of the BE to improve query performance. * query\_mem\_limit Memory limit of a query on each BE node, can be adjusted when a query reports insufficient memory. * load\_mem\_limit Memory limit for import, can be adjusted when an import job reports insufficient memory. Example 2: ```sql set parallel_fragment_exec_instance_num = 8; set query_mem_limit = 137438953472; ``` #### capacity quota of database storage[​](#capacity-quota-of-database-storage "Direct link to capacity quota of database storage") The capacity quota of database storage is unlimited by default. And you can change quota value by using `alter database`. ```sql ALTER DATABASE db_name SET DATA QUOTA quota; ``` The quota units are: B/K/KB/M/MB/G/GB/T/TB/P/PB Example 3: ```sql ALTER DATABASE example_db SET DATA QUOTA 10T; ``` #### Kill queries[​](#kill-queries "Direct link to Kill queries") To terminate a query on a particular connection with the following command: ```sql kill connection_id; ``` The `connection_id` can be seen by `show processlist;` or `select connection_id();`. ```plain show processlist; +------+------------+---------------------+-----------------+---------------+---------+------+-------+------+ | Id | User | Host | Cluster | Db | Command | Time | State | Info | +------+------------+---------------------+-----------------+---------------+---------+------+-------+------+ | 1 | starrocksmgr | 172.26.34.147:56208 | default_cluster | starrocks_monitor | Sleep | 8 | | | | 129 | root | 172.26.92.139:54818 | default_cluster | | Query | 0 | | | | 114 | test | 172.26.34.147:57974 | default_cluster | ssb_100g | Query | 3 | | | | 3 | starrocksmgr | 172.26.34.147:57268 | default_cluster | starrocks_monitor | Sleep | 8 | | | | 100 | root | 172.26.34.147:58472 | default_cluster | ssb_100 | Sleep | 637 | | | | 117 | starrocksmgr | 172.26.34.147:33790 | default_cluster | starrocks_monitor | Sleep | 8 | | | | 6 | starrocksmgr | 172.26.34.147:57632 | default_cluster | starrocks_monitor | Sleep | 8 | | | | 119 | starrocksmgr | 172.26.34.147:33804 | default_cluster | starrocks_monitor | Sleep | 8 | | | | 111 | root | 172.26.92.139:55472 | default_cluster | | Sleep | 2758 | | | +------+------------+---------------------+-----------------+---------------+---------+------+-------+------+ 9 rows in set (0.00 sec) mysql> select connection_id(); +-----------------+ | CONNECTION_ID() | +-----------------+ | 98 | +-----------------+ mysql> kill 114; Query OK, 0 rows affected (0.02 sec) ``` --- ### Query queues This topic describes how to manage query queues in StarRocks. From v2.5, StarRocks supports query queues. With query queues enabled, StarRocks automatically queues the incoming queries when the concurrency threshold or resource limit is reached, thereby avoiding the overload deteriorating. Pending queries wait in a queue until there is enough compute resources available to begin execution. The Query Queue feature has two versions: * [**Query Queue v1**](#query-queue-v1): Triggers queuing based on query concurrency, BE memory usage, and BE CPU usage. The original query queue configurations and behaviors in this topic belong to v1. From v3.1.4 onwards, v1 supports setting query queues on the resource group level. * [**Query Queue v2**](#query-queue-v2): Supported from v3.3 onwards. v2 estimates the BE resources consumed by each query, represents BE resources as logical slots, and queues and schedules queries based on the number of slots each query needs. #### Query Queue v1[​](#query-queue-v1 "Direct link to Query Queue v1") Query Queue v1 supports setting thresholds on CPU usage, memory usage, and query concurrency to trigger query queues. **Roadmap**: | Version | Global query queue | Resource group-level query queue | Collective concurrency management | Dynamic concurrency adjustment | | ------- | ------------------ | -------------------------------- | --------------------------------- | ------------------------------ | | v2.5 | ✅ | ❌ | ❌ | ❌ | | v3.1.4 | ✅ | ✅ | ✅ | ✅ | ##### Enable Query Queue v1[​](#enable-query-queue-v1 "Direct link to Enable Query Queue v1") Query queues are disabled by default. You can enable global or resource group-level query queues for INSERT loading, SELECT queries, and statistics queries by setting corresponding global session variables. ###### Enable global query queues[​](#enable-global-query-queues "Direct link to Enable global query queues") * Enable query queues for loading tasks: ```sql SET GLOBAL enable_query_queue_load = true; ``` * Enable query queues for SELECT queries: ```sql SET GLOBAL enable_query_queue_select = true; ``` * Enable query queues for statistics queries: ```sql SET GLOBAL enable_query_queue_statistic = true; ``` ###### Enable resource group-level query queues[​](#enable-resource-group-level-query-queues "Direct link to Enable resource group-level query queues") From v3.1.4 onwards, StarRocks supports setting query queues on the resource group level. To enable the resource group-level query queues, you also need to set `enable_group_level_query_queue` in addition to the global session variables mentioned above. ```sql SET GLOBAL enable_group_level_query_queue = true; ``` ##### Specify resource thresholds[​](#specify-resource-thresholds "Direct link to Specify resource thresholds") ###### Specify resource thresholds for global query queues[​](#specify-resource-thresholds-for-global-query-queues "Direct link to Specify resource thresholds for global query queues") You can set the thresholds that trigger query queues via the following global session variables: | **Variable** | **Default** | **Description** | | ---------------------------------------- | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | query\_queue\_concurrency\_limit | 0 | The upper limit of concurrent queries on a BE. It takes effect only after being set greater than `0`. Setting it to `0` indicates no limit is imposed. | | query\_queue\_mem\_used\_pct\_limit | 0 | The upper limit of memory usage percentage on a BE. It takes effect only after being set greater than `0`. Setting it to `0` indicates no limit is imposed. Range: \[0, 1] | | query\_queue\_cpu\_used\_permille\_limit | 0 | The upper limit of CPU usage permille (CPU usage \* 1000) on a BE. It takes effect only after being set greater than `0`. Setting it to `0` indicates no limit is imposed. Range: \[0, 1000] | note * After Query Queue v2 is enabled, `query_queue_mem_used_pct_limit` and `query_queue_cpu_used_permille_limit` are no longer supported for queue triggering. * By default, BE reports resource usage to FE at one-second intervals. You can change this interval by setting the BE configuration item `report_resource_usage_interval_ms`. ###### Specify resource thresholds for resource group-level query queues[​](#specify-resource-thresholds-for-resource-group-level-query-queues "Direct link to Specify resource thresholds for resource group-level query queues") From v3.1.4 onwards, you can set individual concurrency limits (`concurrency_limit`) and CPU core limits (`max_cpu_cores`) when creating a resource group. When a query is initiated, if any of the resource consumptions exceed the resource threshold at either the global or resource group level, the query will be placed in queue until all resource consumptions are within the threshold. | **Variable** | **Default** | **Description** | | ------------------ | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | concurrency\_limit | 0 | The concurrency limit for the resource group on a single BE node. It takes effect only when it is set to greater than `0`. | | max\_cpu\_cores | 0 | The CPU core limit for this resource group on a single BE node. It takes effect only when it is set to greater than `0`. Range: \[0, `avg_be_cpu_cores`], where `avg_be_cpu_cores` represents the average number of CPU cores across all BE nodes. | You can use SHOW USAGE RESOURCE GROUPS to view the resource usage information for each resource group on each BE node, as described in [View Resource Group Usage Information](https://docs.starrocks.io/docs/administration/management/resource_management/resource_group.md#view-resource-group-usage-information). ###### Manage query concurrency[​](#manage-query-concurrency "Direct link to Manage query concurrency") When the number of running queries (`num_running_queries`) exceeds the global or resource group's `concurrency_limit`, incoming queries are placed in the queue. The way to obtain `num_running_queries` differs between versions < v3.1.4 and ≥ v3.1.4. * In versions < v3.1.4, `num_running_queries` is reported by BEs at the interval specified in `report_resource_usage_interval_ms`. Therefore, there might be some delay in the identification of changes in `num_running_queries`. For example, if the `num_running_queries` reported by BEs at the moment does not exceed the global or resource group's `concurrency_limit`, but incoming queries arrive and exceed the `concurrency_limit` before the next report, these incoming queries will be executed without waiting in the queue. * In versions ≥ v3.1.4, all running queries are collectively managed by the Leader FE. Each Follower FE notifies the Leader FE when initiating or finishing a query, allowing the StarRocks to handle scenarios where there is a sudden increase in queries exceeding the `concurrency_limit`. ##### Configure Query Queue v1[​](#configure-query-queue-v1 "Direct link to Configure Query Queue v1") You can set the capacity of a query queue and the maximum timeout of queries in queues via the following global session variables: | **Variable** | **Default** | **Description** | | -------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | query\_queue\_max\_queued\_queries | 1024 | The upper limit of queries in a queue. When this threshold is reached, incoming queries are rejected. It takes effect only after being set greater than `0`. | | query\_queue\_pending\_timeout\_second | 300 | The maximum timeout of a pending query in a queue. When this threshold is reached, the corresponding query is rejected. Unit: second. | ##### Configure dynamic adjustment of query concurrency[​](#configure-dynamic-adjustment-of-query-concurrency "Direct link to Configure dynamic adjustment of query concurrency") Starting from version v3.1.4, for queries managed by the query queue and run by the Pipeline Engine, StarRocks can dynamically adjust the query concurrency `pipeline_dop` for incoming queries based on the current number of running queries `num_running_queries`, the number of fragments `num_fragments`, and the query concurrency `pipeline_dop`. This allows you to dynamically control query concurrency while minimizing scheduling overhead, ensuring optimal BE resource utilization. For more information about fragments and query concurrency `pipeline_dop`, see [Query Management - Adjusting Query Concurrency](https://docs.starrocks.io/docs/administration/management/resource_management/Query_management.md). For each query under a query queue, StarRocks maintains a concept of drivers, which represent the concurrent fragments of a query on a single BE. Its logical value `num_drivers`, which represents the total concurrency of all fragments of that query on a single BE, is equal to `num_fragments * pipeline_dop`. When a new query arrives, StarRocks adjusts the query concurrency `pipeline_dop` based on the following rules: * The more the number of running drivers `num_drivers` exceeds the low water limit of concurrent drivers `query_queue_driver_low_water`, the lower the query concurrency `pipeline_dop` is adjusted to. * StarRocks restrains the number of running drivers `num_drivers` below the high water limit of concurrent drivers for queries `query_queue_driver_high_water`. You can configure the dynamic adjustment of query concurrency `pipeline_dop` using the following global session variables: | **Variable** | **Default** | **Description** | | --------------------------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | query\_queue\_driver\_high\_water | -1 | The high water limit of concurrent drivers for a query. It takes effect only when it is set to a non-negative value. When set to `0`, it is equivalent to `avg_be_cpu_cores * 16`, where `avg_be_cpu_cores` represents the average number of CPU cores across all BE nodes. When set to a value greater than `0`, that value is used directly. | | query\_queue\_driver\_low\_water | -1 | The lower limit of concurrent drivers for queries. It takes effect only when it is set to a non-negative value. When set to `0`, it is equivalent to `avg_be_cpu_cores * 8`. When set to a value greater than `0`, that value is used directly. | #### Query Queue v2[​](#query-queue-v2 "Direct link to Query Queue v2") From v3.3 onwards, StarRocks supports Query Queue v2. In Query Queue v2, query queues are no longer triggered based on fixed thresholds for query concurrency, BE memory usage, or BE CPU usage. Instead, it estimates the BE resources required by each query and queues and schedules queries based on logical slots. If there are not enough available slots, the query waits in the queue until enough slots are released. ##### Configure Query Queue v2[​](#configure-query-queue-v2 "Direct link to Configure Query Queue v2") Query Queue v2 is enabled and tuned through FE configuration items. Changes to `enable_query_queue_v2` require restarting FE nodes to take effect. | Configuration item | Default | Description | | -------------------------------------- | ------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `enable_query_queue_v2` | `false` (v3.3 to v4.0)
`true` (from v4.1 onwards) | Whether to enable Query Queue v2. When this item is set to `true`, StarRocks uses the v2 slot-based query scheduling mechanism. | | `query_queue_v2_concurrency_level` | `4` | The logical concurrency level used by Query Queue v2 to calculate the total number of cluster slots. A larger value allows the system to admit more queries. This is a relative tuning parameter. | | `query_queue_slots_estimator_strategy` | `PBE` | The slot estimation strategy used for queue-based queries. Valid values: `PBE` (parallelism-based, the default), `MBE` (memory-cost-based), and `CBE` (CPU-cost-based). PBE estimates a query's slots from scan parallelism, capped by the worker count: for OLAP tables it uses the number of scan ranges left after pruning, so only very small queries fall below the worker count; a connector/external scan is treated as a full-parallelism scan (the worker count) rather than a single-slot query. MBE estimates slots from the query's memory cost divided by `query_queue_v2_mem_bytes_per_slot`. CBE estimates slots from the plan CPU cost divided by `query_queue_v2_cpu_costs_per_slot`. MBE and CBE per-query slots are additionally capped by `number_of_workers * max(1, pipeline_dop / 2)`. The legacy values `MAX` and `MIN` are still accepted for forward compatibility and are treated as the default estimator; any other value is rejected by configuration validation. | | `query_queue_v2_schedule_strategy` | `SWRR` | The scheduling policy used by Query Queue V2 to order pending queries. Supported values (case-insensitive) are `SWRR` (Smooth Weighted Round Robin) — the default, suitable for mixed/hybrid workloads that need fair weighted sharing — and `SJF` (Short Job First + Aging) — prioritizes short jobs while using aging to avoid starvation. The value is parsed with case-insensitive enum lookup; an unrecognized value is logged as an error and the default policy is used. This configuration only affects behavior when Query Queue V2 is enabled and interacts with V2 sizing settings such as `query_queue_v2_concurrency_level`. | | `query_queue_v2_mem_bytes_per_slot` | `0` | Per-slot memory target used by the memory-cost-based estimator (MBE). When `query_queue_slots_estimator_strategy` is `MBE`, the total slots are derived from the warehouse memory budget, and a query's slots are estimated from its total memory cost divided by this value, capped by `number_of_workers * max(1, pipeline_dop / 2)`. If it is non-positive, Query Queue V2 uses the average worker memory per core. | | `query_queue_v2_cpu_costs_per_slot` | `1000000000` | Per-slot CPU cost threshold used by the CPU-cost-based estimator (CBE) to estimate how many slots a query needs from its plan CPU cost. The scheduler computes slots as `ceil(plan_cpu_costs / query_queue_v2_cpu_costs_per_slot)` and clamps the result to the range `[1, min(totalSlots, number_of_workers * max(1, pipeline_dop / 2))]`. A non-positive value is normalized to `1`. Increasing this value reduces slots allocated per query (favoring fewer, larger-slot queries); decreasing it increases slots per query. | | query\_queue\_concurrency\_limit | 0 | The upper limit of concurrent queries on a BE. It takes effect only after being set greater than `0`. Setting it to `0` indicates no limit is imposed. | note `query_queue_mem_used_pct_limit` and `query_queue_cpu_used_permille_limit` apply only to Query Queue v1. After Query Queue v2 is enabled, these parameters no longer take effect. ##### Resource slots[​](#resource-slots "Direct link to Resource slots") Query Queue v2 represents BE resources as logical slots: * **Total cluster slots**: StarRocks sets a logical total number of slots for the entire cluster. This total is positively correlated with the number of BEs and BE CPU cores, and is also affected by `query_queue_v2_concurrency_level`. * **Slots required by a query**: StarRocks estimates the number of slots required by each query. The estimation is based on factors such as statistics, query complexity, the number of fragments, estimated input and output data volumes of complex operators, and DOP. ##### Queuing logic[​](#queuing-logic "Direct link to Queuing logic") When the number of slots required by a query exceeds the current number of remaining slots, the query waits in the queue. Query Queue v2 preferentially satisfies queries that require fewer slots, allowing small queries to obtain resources first and avoiding head-of-line blocking where a large query at the head of the queue blocks later small queries. The entire queuing logic is completed on FE, including setting the total number of cluster slots, estimating the number of slots required by a query, and deciding which query's slot requirement to satisfy first. Query Queue v2 does not schedule based on the actual resource usage of BEs. ##### Choose an estimation strategy[​](#choose-an-estimation-strategy "Direct link to Choose an estimation strategy") ###### PBE[​](#pbe "Direct link to PBE") Parallel-based estimation (PBE) strategy is best for: * Normal reporting queries * Mixed point lookups and large queries * Users who do not want to understand cost model details * DBAs who want stable, simple, and explainable queueing behavior first Expected behaviors with PBE include: * Point lookups or queries that scan little data after pruning use fewer slots * Queries that scan larger ranges use more slots * Small queries are more likely to get execution resources during peak hours The following example sets PBE as the strategy: ```sql ADMIN SET FRONTEND CONFIG ("query_queue_slots_estimator_strategy" = "PBE"); ``` ###### MBE[​](#mbe "Direct link to MBE") Memory cost-based estimation (MBE) strategy is suitable for dealing with memory pressure, such as large joins, large aggregations, or high-cardinality aggregations. The following example sets MBE as the strategy, and allocates 2 GB of memory to each slot: ```sql ADMIN SET FRONTEND CONFIG ("query_queue_slots_estimator_strategy" = "MBE"); ADMIN SET FRONTEND CONFIG ("query_queue_v2_mem_bytes_per_slot" = "2147483648"); ``` MBE divides the query’s total memory cost by this value to get query slots, and divides the warehouse memory budget by this value to get total slots. Tune the MBE strategy in the following directions: **Symptom: Memory still gets saturated easily** * **Adjustment**: Decrease `query_queue_v2_concurrency_level` * **Effect**: Directly lowers the MBE total memory budget **Symptom: Queue is too long but BE memory still has room** * **Adjustment**: Increase `query_queue_v2_concurrency_level` * **Effect**: Directly raises the MBE total memory budget **Symptom: `max_slots` is very small and integer rounding is visible** * **Adjustment**: Decrease `query_queue_v2_mem_bytes_per_slot` * **Effect**: Uses a finer memory slot granularity and reduces coarse rounding error ###### CBE[​](#cbe "Direct link to CBE") CPU cost-based estimation (CBE) strategy is suitable for dealing with memory pressure, such as compute-heavy SQL, complex expressions, or heavy CPU work after scanning. The following example sets CBE as the strategy, and set the CPU cost threshold to `1000000000` for each slot: ```sql ADMIN SET FRONTEND CONFIG ("query_queue_slots_estimator_strategy" = "CBE"); ADMIN SET FRONTEND CONFIG ("query_queue_v2_cpu_costs_per_slot" = "1000000000"); ``` **Symptom: CPU is often saturated** * **Adjustment**: Decrease `query_queue_v2_cpu_costs_per_slot` * **Effect**: The same CPU cost for more slots, making concurrency more conservative **Symptom: Queries queue noticeably but CPU still has room** * **Adjustment**: Increase `query_queue_v2_cpu_costs_per_slot` * **Effect**: The same CPU cost for fewer slots, making concurrency looser ##### Tune concurrency capacity[​](#tune-concurrency-capacity "Direct link to Tune concurrency capacity") If you only want to increase or decrease overall concurrency, do not switch between PBE, MBE, and CBE first. Tune total slot capacity first: ```sql ADMIN SET FRONTEND CONFIG ("query_queue_v2_concurrency_level" = ""); ``` Recommended process: 1. Start with default value `4`. 2. Observe `remain_slots`, `max_slots`, `query_pending_length`, CPU, memory, and query latency. 3. If there is resource headroom but queries queue noticeably, gradually increase `query_queue_v2_concurrency_level`. 4. If resources are often saturated or queries interfere with each other heavily, gradually decrease `query_queue_v2_concurrency_level`. 5. Make small changes each time, such as 10% to 25%, and observe one business peak period before making another change. **Tuning Priority**: Use `query_queue_v2_concurrency_level` to tune overall capacity first. Consider switching to MBE or PBE only after that. Do not change multiple parameters at the same time at the beginning, because it becomes hard to tell which parameter caused the effect. ###### Fallback Concurrency Cap[​](#fallback-concurrency-cap "Direct link to Fallback Concurrency Cap") `query_queue_concurrency_limit` is a fallback concurrency cap and applies to PBE, MBE, and CBE. Query Queue V2 first estimates the slots required by a query with the current estimator and checks whether total slots are available. After that, it checks whether the current number of running queries has reached `query_queue_concurrency_limit`. The default value `0` means unlimited. Set it only when you need an absolute cap on the number of concurrently running queries: ```sql ALTER WAREHOUSE default_warehouse SET ("query_queue_concurrency_limit" = "8"); ``` Use `query_queue_v2_concurrency_level` to tune resource capacity first. Use `query_queue_concurrency_limit` only when you need to explicitly limit the number of queries run at the same time. #### Monitor query queues[​](#monitor-query-queues "Direct link to Monitor query queues") You can view information related to query queues using the following methods. ##### SHOW PROC[​](#show-proc "Direct link to SHOW PROC") You can check the number of running queries, and memory and CPU usages in BE nodes using [SHOW PROC](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md): ```plain mysql> SHOW PROC '/backends'\G *************************** 1. row *************************** ... NumRunningQueries: 0 MemUsedPct: 0.79 % CpuUsedPct: 0.0 % ``` ##### SHOW PROCESSLIST[​](#show-processlist "Direct link to SHOW PROCESSLIST") You can check if a query is in a queue (when `IsPending` is `true`) using [SHOW PROCESSLIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROCESSLIST.md): ```plain mysql> SHOW PROCESSLIST; +------+------+---------------------+-------+---------+---------------------+------+-------+-------------------+-----------+ | Id | User | Host | Db | Command | ConnectionStartTime | Time | State | Info | IsPending | +------+------+---------------------+-------+---------+---------------------+------+-------+-------------------+-----------+ | 2 | root | xxx.xx.xxx.xx:xxxxx | | Query | 2022-11-24 18:08:29 | 0 | OK | SHOW PROCESSLIST | false | +------+------+---------------------+-------+---------+---------------------+------+-------+-------------------+-----------+ ``` ##### FE audit log[​](#fe-audit-log "Direct link to FE audit log") You can check the FE audit log file **fe.audit.log**. The field `PendingTimeMs` indicates the time a query spent waiting in a queue, and its unit is milliseconds. ##### Monitoring metrics[​](#monitoring-metrics "Direct link to Monitoring metrics") You can obtain metrics of query queues in StarRocks using the [Monitor and Alert](https://docs.starrocks.io/docs/administration/management/monitoring/Monitor_and_Alert.md) feature. The following FE metrics are derived from the statistical data of each FE node. | Metric | Unit | Type | Description | | ----------------------------------------------------- | ----- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | starrocks\_fe\_query\_queue\_pending | Count | Instantaneous | The current number of queries in the queue. | | starrocks\_fe\_query\_queue\_total | Count | Instantaneous | The total number of queries historically queued (including those currently running). | | starrocks\_fe\_query\_queue\_timeout | Count | Instantaneous | The total number of queries that have timed out while in the queue. | | starrocks\_fe\_resource\_group\_query\_queue\_total | Count | Instantaneous | The total number of queries historically queued in this resource group (including those currently running). The `name` label indicates the name of the resource group. This metric is supported from v3.1.4 onwards. | | starrocks\_fe\_resource\_group\_query\_queue\_pending | Count | Instantaneous | The number of queries currently in the queue for this resource group. The `name` label indicates the name of the resource group. This metric is supported from v3.1.4 onwards. | | starrocks\_fe\_resource\_group\_query\_queue\_timeout | Count | Instantaneous | The number of queries that have timed out while in the queue for this resource group. The `name` label indicates the name of the resource group. This metric is supported from v3.1.4 onwards. | ##### SHOW RUNNING QUERIES[​](#show-running-queries "Direct link to SHOW RUNNING QUERIES") From v3.1.4 onwards, StarRocks supports the SQL statement `SHOW RUNNING QUERIES`, which is used to display queue information for each query. The meanings of each field are as follows: * `QueryId`: The ID of the query. * `ResourceGroupId`: The ID of the resource group that the query hit. When there is no hit on a user-defined resource group, it will be displayed as "-". * `StartTime`: The start time of the query. * `PendingTimeout`: The time when the PENDING query will time out in the queue. * `QueryTimeout`: The time when the query times out. * `State`: The queue state of the query, where "PENDING" indicates it is in the queue, and "RUNNING" indicates it is currently executing. * `Slots`: The logical resource quantity requested by the query. In Query Queue v1, this value is usually `1`. In Query Queue v2, this value is the estimated number of slots for the query. * `Frontend`: The FE node that initiated the query. * `FeStartTime`: The start time of the FE node that initiated the query. Example: ```plain MySQL [(none)]> SHOW RUNNING QUERIES; +--------------------------------------+-----------------+---------------------+---------------------+---------------------+-----------+-------+---------------------------------+---------------------+ | QueryId | ResourceGroupId | StartTime | PendingTimeout | QueryTimeout | State | Slots | Frontend | FeStartTime | +--------------------------------------+-----------------+---------------------+---------------------+---------------------+-----------+-------+---------------------------------+---------------------+ | a46f68c6-3b49-11ee-8b43-00163e10863a | - | 2023-08-15 16:56:37 | 2023-08-15 17:01:37 | 2023-08-15 17:01:37 | RUNNING | 1 | 127.00.00.01_9010_1692069711535 | 2023-08-15 16:37:03 | | a6935989-3b49-11ee-935a-00163e13bca3 | 12003 | 2023-08-15 16:56:40 | 2023-08-15 17:01:40 | 2023-08-15 17:01:40 | RUNNING | 1 | 127.00.00.02_9010_1692069658426 | 2023-08-15 16:37:03 | | a7b5e137-3b49-11ee-8b43-00163e10863a | 12003 | 2023-08-15 16:56:42 | 2023-08-15 17:01:42 | 2023-08-15 17:01:42 | PENDING | 1 | 127.00.00.03_9010_1692069711535 | 2023-08-15 16:37:03 | +--------------------------------------+-----------------+---------------------+---------------------+---------------------+-----------+-------+---------------------------------+---------------------+ ``` --- ### Manage replica Manage data replicas in your StarRocks cluster. This topic includes two sections - [shared-nothing](#shared-nothing) and [shared-data](#shared-data). #### Shared-nothing[​](#shared-nothing "Direct link to Shared-nothing") ##### Overview[​](#overview "Direct link to Overview") For native tables in shared-nothing clusters, StarRocks adopts a multi-replica strategy to guarantee the high availability of data. When you create a table, you must specify the replica count of the table using the table property `replication_num` (Default value: `3`). When a loading transaction starts, data is simultaneously loaded into the specified number of replicas. The transaction is returned with success only after the data is stored in the majority of replicas. For detailed information, see [Write quorum](#write-quorum). Nonetheless, StarRocks allows you to specify a lower write quorum for a table to achieve better loading performance. StarRocks stores multiple replicas across different BE nodes. For example, if you want to store three replicas for a table, you must deploy at least three BE nodes in your StarRocks cluster. If any of the replicas fail, StarRocks clones a healthy replica, partially or wholly, from another BE node to repair the failed replica. By using the Multi-Version Concurrency Control (MVCC) technique, StarRocks accelerates the repairing of the replica by duplicating the physical copies of these multi-version data. ##### Loading data into a multi-replica table[​](#loading-data-into-a-multi-replica-table "Direct link to Loading data into a multi-replica table") ![Replica-1](/assets/images/replica-1-39f5580148f6b7a0ee79929f5ae45c7a.png) The routine of a loading transaction is as follows: 1. The client submits a loading request to FE. 2. FE chooses a BE node as the Coordinator BE node of this loading transaction, and generates an execution plan for the transaction. 3. The Coordinator BE node reads the data to be loaded from the client. 4. The Coordinator BE node dispatches the data to all the replicas of tablets. > **NOTE** > > A tablet is a logical slice of a table. A table has multiple tablets, and each tablet has replication\_num replicas. The number of tablets in a table is determined by the `bucket_size` property of the table. 5. After the data is loaded and stored in all the tablets, FE makes the loaded data visible. 6. FE returns loading success to the client. Such a routine guarantees service availability even under extreme scenarios. ##### Write quorum[​](#write-quorum "Direct link to Write quorum") Loading data into a multi-replica table can be very time-consuming. If you want to improve the loading performance and you can tolerate relatively lower data availability, you can set a lower write quorum for tables. A write quorum refers to the minimum number of replicas that need to acknowledge a write operation before it is considered successful. You can specify write quorum by adding the property `write_quorum` when you [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md), or add this property to an existing table using [ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md). This property is supported from v2.5. `write_quorum` supports the following values: * `MAJORITY`: Default value. When the majority of data replicas return loading success, StarRocks returns loading task success. Otherwise, StarRocks returns loading task failed. * `ONE`: When one of the data replicas returns loading success, StarRocks returns loading task success. Otherwise, StarRocks returns loading task failed. * `ALL`: When all of the data replicas return loading success, StarRocks returns loading task success. Otherwise, StarRocks returns loading task failed. ##### Automatic replica repair[​](#automatic-replica-repair "Direct link to Automatic replica repair") Replicas can fail because certain BE nodes crash or some loading tasks fail. StarRocks automatically repairs these failed replicas. Every `tablet_sched_checker_interval_seconds`, default 20 seconds, Tablet Checker in FE scans all tablet replicas of all tables in your StarRocks cluster, and judges if a replica is healthy by checking the version number of the currently visible data and the health status of the BE node. If the visible version of a replica lags behind those of the other replicas, StarRocks performs an incremental clone to repair the failed replica. If a BE node fails to receive heartbeats or is dropped from the cluster, or the replica is too lagged to be repaired by an incremental clone, StarRocks performs a full clone to repair the lost replica. After detecting tablet replicas that need repair, FE generates a tablet scheduling task, and adds the task to the scheduling task queue. Tablet Scheduler in the FE receives the scheduling task from the queue, creates clone tasks for each failed replica in accordance with the clone type they need, and assigns the tasks to the executor BE nodes. A clone task is essentially copying data from a source BE node (which has a healthy replica), and loading the data into the destination BE node (which has a failed replica). For a replica with a lagged data version, FE assigns an incremental clone task to the BE executor that stores the failed replica, and informs the executor BE node from which peer BE node it can find a healthy replica and clone the new data. If a replica is lost, FE chooses a surviving BE node as the executor BE node, creates an empty replica in the BE node, and assigns a full clone task to the BE node. For each clone task, regardless of its type, the executor BE node duplicates the physical data files from a healthy replica, and then updates its metadata accordingly. After the clone task is completed, the executor BE node reports task success to Tablet Scheduler in FE. After removing the redundant tablet replicas, FE updates its metadata, marking the completion of the replica repair. ![Replica-2](/assets/images/replica-2-146e16e99c04cdec3405f0dcb7fdc262.png) During tablet repair, StarRocks can still execute queries. StarRocks can load data into the table as long as the number of healthy replicas satisfies `write_quorum`. ##### Repair replica manually[​](#repair-replica-manually "Direct link to Repair replica manually") The manual replica repair consists of two steps: 1. Check the replica status. 2. Set the replica priority level. ###### Check replica status[​](#check-replica-status "Direct link to Check replica status") Follow these steps to check the replica status of tablets to identify the unhealthy (failed) tablets. 1. **Check the status of all tablets in the cluster.** ```sql SHOW PROC '/statistic'; ``` Example: ```plain mysql> SHOW PROC '/statistic'; +----------+-----------------------------+----------+--------------+----------+-----------+------------+--------------------+-----------------------+ | DbId | DbName | TableNum | PartitionNum | IndexNum | TabletNum | ReplicaNum | UnhealthyTabletNum | InconsistentTabletNum | +----------+-----------------------------+----------+--------------+----------+-----------+------------+--------------------+-----------------------+ | 35153636 | default_cluster:DF_Newrisk | 3 | 3 | 3 | 96 | 288 | 0 | 0 | | 48297972 | default_cluster:PaperData | 0 | 0 | 0 | 0 | 0 | 0 | 0 | | 5909381 | default_cluster:UM_TEST | 7 | 7 | 10 | 320 | 960 | 1 | 0 | | Total | 240 | 10 | 10 | 13 | 416 | 1248 | 1 | 0 | +----------+-----------------------------+----------+--------------+----------+-----------+------------+--------------------+-----------------------+ ``` * `UnhealthyTabletNum`: indicates the number of unhealthy tablets in the corresponding database. * `InconsistentTabletNum`: indicates the number of tablets whose replicas are inconsistent. If the value of `UnhealthyTabletNum` or `InconsistentTabletNum` is not `0` in a specific database, you can check the unhealthy tablets in the database with its `DbId`. ```sql SHOW PROC '/statistic/' ``` Example: ```plain mysql> SHOW PROC '/statistic/5909381'; +------------------+---------------------+ | UnhealthyTablets | InconsistentTablets | +------------------+---------------------+ | [40467980] | [] | +------------------+---------------------+ ``` The ID of the unhealthy tablet is returned in the field `UnhealthyTablets`. 2. **Check the tablet status in a specific table or partition.** You can use the WHERE clause in [ADMIN SHOW REPLICA STATUS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SHOW_REPLICA_STATUS.md) to filter the tablets with a certain `STATUS`. ```sql ADMIN SHOW REPLICA STATUS FROM [PARTITION ([, , ...])] [WHERE STATUS = {'OK'|'DEAD'|'VERSION_ERROR'|'SCHEMA_ERROR'|'MISSING'}] ``` Example: ```plain mysql> ADMIN SHOW REPLICA STATUS FROM tbl PARTITION (p1, p2) WHERE STATUS = "OK"; +----------+-----------+-----------+---------+-------------------+--------------------+------------------+------------+------------+-------+--------+--------+ | TabletId | ReplicaId | BackendId | Version | LastFailedVersion | LastSuccessVersion | CommittedVersion | SchemaHash | VersionNum | IsBad | State | Status | +----------+-----------+-----------+---------+-------------------+--------------------+------------------+------------+------------+-------+--------+--------+ | 29502429 | 29502432 | 10006 | 2 | -1 | 2 | 1 | -1 | 2 | false | NORMAL | OK | | 29502429 | 36885996 | 10002 | 2 | -1 | -1 | 1 | -1 | 2 | false | NORMAL | OK | | 29502429 | 48100551 | 10007 | 2 | -1 | -1 | 1 | -1 | 2 | false | NORMAL | OK | | 29502433 | 29502434 | 10001 | 2 | -1 | 2 | 1 | -1 | 2 | false | NORMAL | OK | | 29502433 | 44900737 | 10004 | 2 | -1 | -1 | 1 | -1 | 2 | false | NORMAL | OK | | 29502433 | 48369135 | 10006 | 2 | -1 | -1 | 1 | -1 | 2 | false | NORMAL | OK | +----------+-----------+-----------+---------+-------------------+--------------------+------------------+------------+------------+-------+--------+--------+ ``` If the field `IsBad` is `true`, this tablet is corrupted. For detailed information provided in the field `Status`, see [ADMIN SHOW REPLICA STATUS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SHOW_REPLICA_STATUS.md). You can further explore the details of tablets in the table using [SHOW TABLET](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_TABLET.md). ```sql SHOW TABLET FROM ``` Example: ```plain mysql> SHOW TABLET FROM tbl1; +----------+-----------+-----------+------------+---------+-------------+-------------------+-----------------------+------------------+----------------------+---------------+----------+----------+--------+-------------------------+--------------+----------------------+--------------+----------------------+----------------------+----------------------+ | TabletId | ReplicaId | BackendId | SchemaHash | Version | VersionHash | LstSuccessVersion | LstSuccessVersionHash | LstFailedVersion | LstFailedVersionHash | LstFailedTime | DataSize | RowCount | State | LstConsistencyCheckTime | CheckVersion | CheckVersionHash | VersionCount | PathHash | MetaUrl | CompactionStatus | +----------+-----------+-----------+------------+---------+-------------+-------------------+-----------------------+------------------+----------------------+---------------+----------+----------+--------+-------------------------+--------------+----------------------+--------------+----------------------+----------------------+----------------------+ | 29502429 | 29502432 | 10006 | 1421156361 | 2 | 0 | 2 | 0 | -1 | 0 | N/A | 784 | 0 | NORMAL | N/A | -1 | -1 | 2 | -5822326203532286804 | url | url | | 29502429 | 36885996 | 10002 | 1421156361 | 2 | 0 | -1 | 0 | -1 | 0 | N/A | 784 | 0 | NORMAL | N/A | -1 | -1 | 2 | -1441285706148429853 | url | url | | 29502429 | 48100551 | 10007 | 1421156361 | 2 | 0 | -1 | 0 | -1 | 0 | N/A | 784 | 0 | NORMAL | N/A | -1 | -1 | 2 | -4784691547051455525 | url | url | +----------+-----------+-----------+------------+---------+-------------+-------------------+-----------------------+------------------+----------------------+---------------+----------+----------+--------+-------------------------+--------------+----------------------+--------------+----------------------+----------------------+----------------------+ ``` The returned results show the size, row count, version, and URL of the tablets. The field `State` returned by SHOW TABLET indicates the task state of the tablet, including `CLONE`, `SCHEMA_CHANGE`, and `ROLLUP`. You can also check the replica distribution of a specific table or partition to see if these replicas are distributed evenly using [ADMIN SHOW REPLICA DISTRIBUTION](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SHOW_REPLICA_DISTRIBUTION.md). ```sql ADMIN SHOW REPLICA DISTRIBUTION FROM ``` Example: ```plain mysql> ADMIN SHOW REPLICA DISTRIBUTION FROM tbl1; +-----------+------------+-------+---------+ | BackendId | ReplicaNum | Graph | Percent | +-----------+------------+-------+---------+ | 10000 | 7 | | 7.29 % | | 10001 | 9 | | 9.38 % | | 10002 | 7 | | 7.29 % | | 10003 | 7 | | 7.29 % | | 10004 | 9 | | 9.38 % | | 10005 | 11 | > | 11.46 % | | 10006 | 18 | > | 18.75 % | | 10007 | 15 | > | 15.62 % | | 10008 | 13 | > | 13.54 % | +-----------+------------+-------+---------+ ``` The returned results show the number of tablet replicas on each BE node, and their corresponding percentages. 3. **Check the** **replica** **status of a specific tablet.** With the `TabletId` of the unhealthy tablets you obtained in the preceding procedures, you can examine the replica statues of them. ```sql SHOW TABLET ``` Example: ```plain mysql> SHOW TABLET 29502553; +------------------------+-----------+---------------+-----------+----------+----------+-------------+----------+--------+---------------------------------------------------------------------------+ | DbName | TableName | PartitionName | IndexName | DbId | TableId | PartitionId | IndexId | IsSync | DetailCmd | +------------------------+-----------+---------------+-----------+----------+----------+-------------+----------+--------+---------------------------------------------------------------------------+ | default_cluster:test | test | test | test | 29502391 | 29502428 | 29502427 | 29502428 | true | SHOW PROC '/dbs/29502391/29502428/partitions/29502427/29502428/29502553'; | +------------------------+-----------+---------------+-----------+----------+----------+-------------+----------+--------+---------------------------------------------------------------------------+ ``` The returned results show detailed information about the database, table, partition, and index (Rollup) of the tablet. You can copy the SQL statement in the field `DetailCmd` to further examine the replica status of the tablet. Example: ```plain mysql> SHOW PROC '/dbs/29502391/29502428/partitions/29502427/29502428/29502553'; +-----------+-----------+---------+-------------+-------------------+-----------------------+------------------+----------------------+---------------+------------+----------+----------+--------+-------+--------------+----------------------+----------+------------------+ | ReplicaId | BackendId | Version | VersionHash | LstSuccessVersion | LstSuccessVersionHash | LstFailedVersion | LstFailedVersionHash | LstFailedTime | SchemaHash | DataSize | RowCount | State | IsBad | VersionCount | PathHash | MetaUrl | CompactionStatus | +-----------+-----------+---------+-------------+-------------------+-----------------------+------------------+----------------------+---------------+------------+----------+----------+--------+-------+--------------+----------------------+----------+------------------+ | 43734060 | 10004 | 2 | 0 | -1 | 0 | -1 | 0 | N/A | -1 | 784 | 0 | NORMAL | false | 2 | -8566523878520798656 | url | url | | 29502555 | 10002 | 2 | 0 | 2 | 0 | -1 | 0 | N/A | -1 | 784 | 0 | NORMAL | false | 2 | 1885826196444191611 | url | url | | 39279319 | 10007 | 2 | 0 | -1 | 0 | -1 | 0 | N/A | -1 | 784 | 0 | NORMAL | false | 2 | 1656508631294397870 | url | url | +-----------+-----------+---------+-------------+-------------------+-----------------------+------------------+----------------------+---------------+------------+----------+----------+--------+-------+--------------+----------------------+----------+------------------+ ``` The returned results show all the replicas of the tablet. ###### Set replica priority level[​](#set-replica-priority-level "Direct link to Set replica priority level") Tablet Scheduler automatically assigns a different priority level to each different type of clone task. If you want the tablets from a certain table or certain partitions to be repaired at the earliest opportunity, you can manually assign a `VERY_HIGH` priority level to them using [ADMIN REPAIR TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_REPAIR.md). ```sql ADMIN REPAIR TABLE [PARTITION ([, , ...])] ``` > **NOTE** > > * Executing this SQL statement only submits a hint to modify the priority level of the tablets to be repaired. It does not guarantee that these tablets can be successfully repaired. > * Tablet Scheduler might still assign different priority levels to these tablets after you execute this SQL statement. > * When the Leader FE node is changed or restarted, the hint this SQL statement submitted expires. You can cancel this operation using [ADMIN CANCEL REPAIR TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_CANCEL_REPAIR.md). ```sql ADMIN CANCEL REPAIR TABLE [PARTITION ([, , ...])] ``` ##### Replica balancing[​](#replica-balancing "Direct link to Replica balancing") StarRocks automatically balances the tablets across BE nodes. To move a tablet from a high-load node to a low-load node, StarRocks first creates a replica of the tablet in the low-load node, and then drops the corresponding replica on the high-load node. If different types of storage mediums are used in the cluster, StarRocks categorizes all the BE nodes in accordance with the storage medium types. StarRocks moves the tablet across the BE nodes of the same storage medium type whenever possible. Replicas of the same tablet are stored on different BE nodes. ###### BE load[​](#be-load "Direct link to BE load") StarRocks shows the load statistics of each BE node in the cluster using `ClusterLoadStatistics` (CLS). Tablet Scheduler triggers the replica balancing based on `ClusterLoadStatistics`. StarRocks evaluates the **disk utilization** and the **replica count** of each BE node and calculates a `loadScore` accordingly. The higher the `loadScore` of a BE node, the higher the load the node has. Tablet Scheduler updates `ClusterLoadStatistics` every one minute. `capacityCoefficient` and `replicaNumCoefficient`are the weighting factors for the disk utilization and the replica count respectively. The sum of `capacityCoefficient` and `replicaNumCoefficient` is one. `capacityCoefficient` is dynamically adjusted according to the actual disk usage. When the overall disk utilization of a BE node is below 50%, the `capacityCoefficient` value is 0.5. When the disk utilization is above 75%, the value is 1. You can configure this limit via the FE configuration item `capacity_used_percent_high_water`. If the utilization is between 50% and 75%, `capacityCoefficient` increases smoothly based on this formula: ```sql capacityCoefficient= 2 * Disk utilization - 0.5 ``` `capacityCoefficient` ensures that when the disk usage is exceedingly high, the `loadScore` of this BE node gets higher, forcing the system to reduce the load on this BE node at the earliest opportunity. ###### Balancing policy[​](#balancing-policy "Direct link to Balancing policy") Each time Tablet Scheduler schedules tablets, it selects a certain number of healthy tablets as the candidate tablets to be balanced through Load Balancer. Next time when scheduling tablets, Tablet Scheduler balances these healthy tablets. ###### View System Balance Status[​](#view-system-balance-status "Direct link to View System Balance Status") You can view the current overall balance status of the system and the details of different balance types. * **View the current overall balance status of the system:** ```sql SHOW PROC '/cluster_balance/balance_stat'; ``` Example: ```plain +---------------+--------------------------------+----------+----------------+----------------+ | StorageMedium | BalanceType | Balanced | PendingTablets | RunningTablets | +---------------+--------------------------------+----------+----------------+----------------+ | HDD | inter-node disk usage | true | 0 | 0 | | HDD | inter-node tablet distribution | true | 0 | 0 | | HDD | intra-node disk usage | true | 0 | 0 | | HDD | intra-node tablet distribution | true | 0 | 0 | | HDD | colocation group | true | 0 | 0 | | HDD | label-aware location | true | 0 | 0 | +---------------+--------------------------------+----------+----------------+----------------+ ``` * `StorageMedium`: Storage medium. * `BalanceType`: Type of balance. * `Balanced`: Whether the balanced state is achieved. * `PendingTablets`: Number of tablets with task status Pending. * `RunningTablets`: Number of tablets with task status Running. * **View the balance of disk utilization by node:** ```sql SHOW PROC '/cluster_balance/cluster_load_stat'; ``` Example: ```plain +---------------+----------------------------------------------------------------------------------------------------------------------+ | StorageMedium | ClusterDiskBalanceStat | +---------------+----------------------------------------------------------------------------------------------------------------------+ | HDD | {"balanced":false,"maxBeId":1,"minBeId":2,"maxUsedPercent":0.9,"minUsedPercent":0.1,"type":"INTER_NODE_DISK_USAGE"} | | SSD | {"balanced":true} | +---------------+----------------------------------------------------------------------------------------------------------------------+ ``` * `StorageMedium`: Storage medium. * `ClusterDiskBalanceStat`: Balance status across nodes based on disk usage. If not balanced, displays the maximum and minimum disk utilization and the corresponding BEs. * **View the balance of disk usage within node:** ```sql SHOW PROC '/cluster_balance/cluster_load_stat/HDD'; ``` Example: ```plain +-------+-----------------+-----------+--------------+--------------+-------------+------------+----------+-----------+-------+-------+---------------------------------------------------------------------------------------------------------------------------------------------+ | BeId | Cluster | Available | UsedCapacity | Capacity | UsedPercent | ReplicaNum | CapCoeff | ReplCoeff | Score | Class | BackendDiskBalanceStat | +-------+-----------------+-----------+--------------+--------------+-------------+------------+----------+-----------+-------+-------+---------------------------------------------------------------------------------------------------------------------------------------------+ | 10004 | default_cluster | true | 651509602 | 243695955810 | 0.267 | 339 | 0.5 | 0.5 | 1.0 | MID | {"maxUsedPercent":0.9,"minUsedPercent":0.1,"beId":1,"maxPath":"/disk1","minPath":"/disk2","type":"INTRA_NODE_DISK_USAGE","balanced":false} | | 10005 | default_cluster | true | 651509602 | 243695955810 | 0.267 | 339 | 0.5 | 0.5 | 1.0 | MID | {"balanced":true} | +-------+-----------------+-----------+--------------+--------------+-------------+------------+----------+-----------+-------+-------+---------------------------------------------------------------------------------------------------------------------------------------------+ ``` * `BeId`: ID of the BE node. * `BackendDiskBalanceStat`: Balance status between disks within the node based on disk utilization. If not balanced, displays the maximum and minimum disk usage and the corresponding disk paths. * **View balanced distribution by tablet:** ```sql SHOW PROC '/dbs/ssb/lineorder/partitions/lineorder'; ``` Example: ```plain +---------+-----------+--------+--------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+ | IndexId | IndexName | State | LastConsistencyCheckTime | TabletBalanceStat | +---------+-----------+--------+--------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+ | 11129 | lineorder | NORMAL | NULL | {"maxTabletNum":23,"minTabletNum":21,"maxBeId":10012,"minBeId":10013,"type":"INTER_NODE_TABLET_DISTRIBUTION","balanced":false} | | 11230 | lineorder | NORMAL | NULL | {"maxTabletNum":23,"minTabletNum":21,"beId":10012,"maxPath":"/disk1","minPath":"/disk2","type":"INTRA_NODE_TABLET_DISTRIBUTION","balanced":false} | | 10432 | lineorder | NORMAL | NULL | {"tabletId":10435,"currentBes":[10002,10004],"expectedBes":[10003,10004],"type":"COLOCATION_GROUP","balanced":false} | | 10436 | lineorder | NORMAL | NULL | {"tabletId":10438,"currentBes":[10005,10006],"expectedLocations":{"rack":["rack1","rack2"]},"type":"LABEL_AWARE_LOCATION","balanced":false} | +---------+-----------+--------+--------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------+ ``` * `IndexId`: ID of the Materialized Index within the partition. * `TabletBalanceStat`: Balance status of tablet distribution between nodes or within nodes. If not balanced, displays the details of the imbalance, including Colocation Group, Label-aware Location. * **View partitions with imbalanced tablet distribution:** ```sql SELECT DB_NAME, TABLE_NAME, PARTITION_NAME, TABLET_BALANCED FROM information_schema.partitions_meta WHERE TABLET_BALANCED = 0; ``` Example: ```plain +--------------+---------------+----------------+-----------------+ | DB_NAME | TABLE_NAME | PARTITION_NAME | TABLET_BALANCED | +--------------+---------------+----------------+-----------------+ | ssb | lineorder | lineorder | 0 | +--------------+---------------+----------------+-----------------+ ``` * `TABLET_BALANCED`: whether the tablet distribution is balanced. ###### Check tablet scheduling tasks[​](#check-tablet-scheduling-tasks "Direct link to Check tablet scheduling tasks") You can check tablet scheduling tasks that are pending, running, and finished. * **Check pending tablet scheduling tasks** ```sql SHOW PROC '/cluster_balance/pending_tablets'; ``` Example: ```plain +----------+--------+-----------------+---------+----------+----------+-------+---------+--------+----------+---------+---------------------+---------------------+---------------------+----------+------+-------------+---------------+---------------------+------------+---------------------+--------+---------------------+-------------------------------+ | TabletId | Type | Status | State | OrigPrio | DynmPrio | SrcBe | SrcPath | DestBe | DestPath | Timeout | Create | LstSched | LstVisit | Finished | Rate | FailedSched | FailedRunning | LstAdjPrio | VisibleVer | VisibleVerHash | CmtVer | CmtVerHash | ErrMsg | +----------+--------+-----------------+---------+----------+----------+-------+---------+--------+----------+---------+---------------------+---------------------+---------------------+----------+------+-------------+---------------+---------------------+------------+---------------------+--------+---------------------+-------------------------------+ | 4203036 | REPAIR | REPLICA_MISSING | PENDING | HIGH | LOW | -1 | -1 | -1 | -1 | 0 | 2019-02-21 15:00:20 | 2019-02-24 11:18:41 | 2019-02-24 11:18:41 | N/A | N/A | 2 | 0 | 2019-02-21 15:00:43 | 1 | 0 | 2 | 0 | unable to find source replica | +----------+--------+-----------------+---------+----------+----------+-------+---------+--------+----------+---------+---------------------+---------------------+---------------------+----------+------+-------------+---------------+---------------------+------------+---------------------+--------+---------------------+-------------------------------+ ``` * `TabletId`: the ID of the tablet pending to be scheduled. A scheduled task is only for one tablet. * `Type`: the task type. Valid values: REPAIR and BALANCE. * `Status`: the current status of the tablet, such as REPLICA\_MISSING. * `State`: the state of the scheduling task. Valid values: PENDING, RUNNING, FINISHED, CANCELLED, TIMEOUT, and UNEXPECTED. * `OrigPrio`: the original priority of the task. * `DynmPrio`: the current priority of the task after the dynamic adjustment. * `SrcBe`: the ID of the source BE node. * `SrcPath`: the hash value of the path to the source BE node. * `DestBe`: the ID of the destination BE node. * `DestPath`: the hash value of the path to the destination BE node. * `Timeout`: the timeout of the task when the task is scheduled successfully. Unit: second. * `Create`: the time when the task was created. * `LstSched`: the time when the task was scheduled most recently. * `LstVisit`: the time when the task was visited most recently. To visit the task here means to schedule the task or to report its execution. * `Finished`: the time when the task is finished. * `Rate`: the rate at which the data is cloned. * `FailedSched`: the number of task scheduling failures. * `FailedRunning`: the number of task execution failures. * `LstAdjPrio`: the time when the task priority was adjusted most recently. * `CmtVer`, `CmtVerHash`, `VisibleVer`, and `VisibleVerHash`: the version information used to execute the clone task. * `ErrMsg`: the error message that occurs when the task is scheduled and running. * **Check running tablet scheduling tasks** ```sql SHOW PROC '/cluster_balance/running_tablets'; ``` The returned results are identical to those of the pending tasks. * **Check finished tablet scheduling tasks** ```sql SHOW PROC '/cluster_balance/history_tablets'; ``` The returned results are identical to those of the pending tasks. If the `State` of the task is `FINISHED`, the task is completed successfully. If not, please check the `ErrMsg` field for the cause of the task failure. ##### Resource control[​](#resource-control "Direct link to Resource control") Because StarRocks repairs and balances tablets by cloning tablets from one BE node to another, the I/O load of a BE node can increase dramatically if the node executes such tasks too frequently in a short time. To avoid this situation, StarRocks sets a concurrency limit on clone tasks for each BE node. The minimum unit of resource control is a disk, which is a data storage path (`storage_root_path`) you have specified in the BE configuration file. By default, StarRocks allocates two slots for each disk to process tablet repair tasks. A clone task occupies one slot on the source BE node and one on the destination BE node. If all the slots on a BE node are occupied, StarRocks stops scheduling tasks to the node. You can increase the number of slots on a BE node by increasing the value of the FE dynamic parameter `tablet_sched_slot_num_per_path`. StarRocks allocates two slots specifically for tablet balancing tasks to avoid the situation that a high-load BE node fails to release the disk space by balancing tablets because tablet repair tasks constantly occupy the slots. #### Shared-data[​](#shared-data "Direct link to Shared-data") From v4.1 onwards, StarRocks supports repairing the data replica of cloud-native tables in shared-data clusters. ##### Overview[​](#overview-1 "Direct link to Overview") In a shared-data architecture, data is stored in single-replica mode on remote storage systems such as object storage or HDFS in order to reduce storage costs. Unlike traditional shared-nothing architectures, this design cannot rely on multiple replicas to automatically recover data when files are lost. As a result, if the effective metadata version maintained by FE references metadata or data files that no longer exist in remote storage, data ingestion and query operations will fail with "File not found" errors, potentially rendering the service unavailable. Such file loss may occur under the following circumstances: * **Accidental deletion**: Object storage files are mistakenly removed due to operational errors. * **Consistency issues**: In extreme cases, the storage system experiences delayed consistency or metadata loss. * **Software defects**: System bugs cause files to be cleaned up prematurely. Traditional snapshot-based restore mechanisms are often time-consuming and costly, making them unsuitable for fast recovery in production environments. To address this issue, StarRocks provides a low-cost, second-level recovery mechanism. By scanning historical metadata versions, the system identifies the most recent healthy version in which all required files are present, and rolls back the Tablet metadata to that version. This approach sacrifices a small amount of recent data in exchange for rapid restoration of table availability. ###### Mechanism[​](#mechanism "Direct link to Mechanism") This feature is an extension of the ADMIN REPAIR TABLE statement for cloud-native tables in shared-data clusters. It operates through the following mechanisms: 1. **Automated Detection** The FE coordinates Compute Nodes (CNs) to probe historical Tablet metadata versions in reverse chronological order and in batches. 2. **Deterministic Path Derivation** Metadata file paths are derived deterministically, allowing direct probing without performing expensive and inefficient object storage List Objects operations. 3. **Multi-Strategy Recovery Decisions** Two recovery strategies are supported—Strict Consistency and Maximum Availability—to accommodate different business requirements. 4. **Metadata Reset Capability** When metadata is completely unavailable, the system can create empty Tablets (Empty Tablet Recovery) to prevent a small number of corrupted Tablets from blocking the availability of the entire table. ###### Key Benefits[​](#key-benefits "Direct link to Key Benefits") * **Ultra-fast recovery** Only metadata (KB/MB scale) is modified. No data movement is required, enabling second-level recovery even for PB-scale tables. * **Low operational cost** No additional replicas or expensive object storage API calls are needed. ##### Check Tablet Status[​](#check-tablet-status "Direct link to Check Tablet Status") Before executing a repair, you can use the [ADMIN SHOW TABLET STATUS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SHOW_TABLET_STATUS.md) statement to check the file integrity of each Tablet in a cloud-native table or cloud-native materialized view, and identify whether metadata or data files are missing. **Only cloud-native tables and cloud-native materialized views in shared-data clusters support this statement.** ###### Syntax[​](#syntax "Direct link to Syntax") ```sql ADMIN SHOW TABLET STATUS FROM [.] [PARTITION ( [, , ...])] [WHERE STATUS [=|!=] {'NORMAL'|'MISSING_META'|'MISSING_DATA'}] [PROPERTIES ("max_missing_data_files_to_show" = "")] ``` **Tablet status values:** | Status | Description | | ------------- | --------------------------------------- | | NORMAL | Metadata and data files are all intact. | | MISSING\_META | Metadata file is missing. | | MISSING\_DATA | Data file(s) are missing. | **Return columns:** | Column | Description | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | TabletId | Tablet ID. | | PartitionId | Physical partition ID of the partition the Tablet belongs to. | | Version | Current visible version of the Tablet. | | Status | Tablet status. See status values above. | | MissingDataFileCount | Number of missing data files. Populated when `Status` is `MISSING_DATA`. | | MissingDataFiles | List of missing data file paths. Limited by `max_missing_data_files_to_show` (default: 5). Populated when `Status` is `MISSING_DATA`. | ###### Example: Check abnormal tablets in a partition[​](#example-check-abnormal-tablets-in-a-partition "Direct link to Example: Check abnormal tablets in a partition") ```sql ADMIN SHOW TABLET STATUS FROM my_cloud_table PARTITION (p20250101) WHERE STATUS != "NORMAL"; ``` ##### Usage[​](#usage "Direct link to Usage") ###### Syntax[​](#syntax-1 "Direct link to Syntax") Use the ADMIN REPAIR TABLE statement with PROPERTIES to control recovery behavior. ```sql ADMIN REPAIR TABLE [PARTITION (, ...)] PROPERTIES ( 'enforce_consistent_version' = 'true', 'allow_empty_tablet_recovery' = 'false' ); ``` **Properties** | Property | Type | Default | Description | | ------------------------------ | ------- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | enforce\_consistent\_version | Boolean | True | Whether to enforce all tablets in a partition to roll back to a consistent version. If this item is set to `true`, the system will search for a consistent version in the history that is valid for all tablets to perform the rollback, ensuring data version alignment across the partition. If it is set to `false`, each tablet in the partition is allowed to rollback to its latest available valid version. The versions of different tablets may be inconsistent, but this maximizes data preservation. | | allow\_empty\_tablet\_recovery | Boolean | False | Whether to allow recovery by creating empty tablets. This item takes effect only when `enforce_consistent_version` is `false`. If this item is set to `true`, when metadata is missing for all versions of some tablets but valid metadata exists for at least one tablet, the system attempts to create empty tablets to fill the missing versions. If metadata for all versions of all tablets is lost, recovery is impossible. | | dry\_run | Boolean | False | Whether to return the repair plan without actually executing the repair. If set to `true`, the system evaluates the recoverability of each partition and returns the repair plan without performing any actual metadata rollback. Useful for previewing the repair effect before execution. See [Preview the Repair Plan (Dry Run)](#preview-the-repair-plan-dry-run). | ##### Examples[​](#examples "Direct link to Examples") ###### Example 1: Strict Consistency Recovery (Recommended for Strong Consistency)[​](#example-1-strict-consistency-recovery-recommended-for-strong-consistency "Direct link to Example 1: Strict Consistency Recovery (Recommended for Strong Consistency)") Restore all Tablets in partition `p20250101` to the most recent uniform and complete version. ```sql ADMIN REPAIR TABLE my_cloud_table PARTITION (p20250101); ``` ###### Example 2: Maximum Availability Recovery[​](#example-2-maximum-availability-recovery "Direct link to Example 2: Maximum Availability Recovery") Restore each Tablet in partition p20250101 to its own most recent valid version, allowing version divergence. ```sql ADMIN REPAIR TABLE my_cloud_table PARTITION (p20250101) PROPERTIES ( 'enforce_consistent_version' = 'false' ); ``` ###### Example 3: Allow Empty Tablet Recovery[​](#example-3-allow-empty-tablet-recovery "Direct link to Example 3: Allow Empty Tablet Recovery") Allow Tablets with completely missing metadata to be recovered as empty Tablets. ```sql ADMIN REPAIR TABLE my_cloud_table PARTITION (p20250101) PROPERTIES ( 'enforce_consistent_version' = 'false', 'allow_empty_tablet_recovery' = 'true' ); ``` ##### Preview the Repair Plan (Dry Run)[​](#preview-the-repair-plan-dry-run "Direct link to Preview the Repair Plan (Dry Run)") Before executing the repair, you can use `dry_run` mode to preview the repair plan. This lets you check the recoverability of each partition and the version each tablet would roll back to, without actually modifying any metadata. **`dry_run` mode is only supported for cloud-native tables in shared-data clusters.** ###### Syntax[​](#syntax-2 "Direct link to Syntax") ```sql ADMIN REPAIR TABLE [PARTITION (, ...)] PROPERTIES ( 'dry_run' = 'true' ); ``` ###### Return Columns[​](#return-columns "Direct link to Return Columns") | Column | Description | | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | PartitionId | Partition ID. | | VisibleVersion | Current visible version. | | RepairStatus | Repair status: `NORMAL` (all tablets healthy, no repair needed), `RECOVERABLE` (missing files but recoverable), `UNRECOVERABLE` (missing files and unrecoverable), `UNKNOWN` (exception during probing). | | TabletRecoverInfo | JSON array listing the version each tablet will roll back to. Only populated when `RepairStatus` is `RECOVERABLE`. | | ErrorMsg | Error message. Only populated when `RepairStatus` is `UNRECOVERABLE` or `UNKNOWN`. | ###### Example: Preview the repair plan for partition p1[​](#example-preview-the-repair-plan-for-partition-p1 "Direct link to Example: Preview the repair plan for partition p1") ```sql ADMIN REPAIR TABLE my_cloud_table PARTITION (p1) PROPERTIES ("dry_run" = "true"); ``` Example output: ```plain +-------------+----------------+--------------+-----------------------------------------------------------------------+---------+ | PartitionId | VisibleVersion | RepairStatus | TabletRecoverInfo | ErrorMsg| +-------------+----------------+--------------+-----------------------------------------------------------------------+---------+ | 10001 | 100 | RECOVERABLE | [{"tabletId":20001,"version":98},{"tabletId":20002,"version":98}] | | | 10002 | 100 | NORMAL | [] | | +-------------+----------------+--------------+-----------------------------------------------------------------------+---------+ ``` After confirming the repair plan, remove `dry_run` (or set it to `false`) to execute the actual repair. ##### Limitations and Recommendations[​](#limitations-and-recommendations "Direct link to Limitations and Recommendations") * Repairing replicas is supported for shared-data clusters from v3.5 onwards. * Currently, setting `PROPERTIES` is applicable only to shared-data tables. * Take the following points into consideration when performing materialized view recovery: * Asynchronous materialized views require a manual refresh after recovery. * Synchronous materialized views must use Strict Consistency Recovery to ensure consistency between base tables and rollup data. * Recovery is best-effort. A small amount of data loss is expected, depending on the available historical versions and the selected recovery strategy. --- ### Resource group This topic describes the resource group feature of StarRocks. ![resource group](/assets/images/resource_group-48d40767073c72d56e922e11a48e213c.png) With this feature, you could simultaneously run several workloads in a single cluster, including short query, ad-hoc query, ETL jobs, to save extra cost of deploying multiple clusters. From technical perspective, the execution engine would schedule concurrent workloads according to users' specification and isolate the interference among them. The roadmap of Resource Group: * Since v2.2, StarRocks supports limiting resource consumption for queries and implementing isolation and efficient use of resources among tenants in the same cluster. * In StarRocks v2.3, you can further restrict the resource consumption for big queries, and prevent the cluster resources from getting exhausted by oversized query requests, to guarantee the system stability. * StarRocks v2.5 supports limiting computation resource consumption for data loading (INSERT). * From v3.3.5 onwards, StarRocks supports imposing hard limits on CPU resources. | | Internal Table | External Table | Big Query Restriction | INSERT INTO | Broker Load | Routine Load, Stream Load, Schema Change | CPU Hard Limit | | --------------- | -------------- | -------------- | --------------------- | ----------- | ----------- | ---------------------------------------- | -------------- | | 2.2 | √ | × | × | × | × | × | x | | 2.3 | √ | √ | √ | × | × | × | x | | 2.5 | √ | √ | √ | √ | × | × | x | | 3.1 & 3.2 | √ | √ | √ | √ | √ | × | x | | 3.3.5 and later | √ | √ | √ | √ | √ | × | √ | #### Terms[​](#terms "Direct link to Terms") This section describes the terms that you must understand before you use the resource group feature. ##### resource group[​](#resource-group-1 "Direct link to resource group") Each resource group is a set of computing resources from a specific BE. You can divide each BE of your cluster into multiple resource groups. When a query is assigned to a resource group, StarRocks allocates CPU and memory resources to the resource group based on the resource quotas that you specified for the resource group. You can specify CPU and memory resource quotas for a resource group on a BE by using the following parameters: | Parameter | Description | Value Range | Default | | ------------------------------ | ---------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | ------------------ | | cpu\_weight | The CPU scheduling weight of this resource group on a BE node. | (0, `avg_be_cpu_cores`] (takes effect when greater than 0) | 0 | | cpu\_weight\_percent | The CPU scheduling weight percentage of this resource group on a BE node. Supported from v4.1. | \[0, 100] (takes effect when greater than 0) | 0 | | exclusive\_cpu\_cores | CPU hard isolation parameter for this resource group. | (0, `min_be_cpu_cores - 1`] (takes effect when greater than 0) | 0 | | exclusive\_cpu\_percent | CPU hard isolation percentage for this resource group. Supported from v4.1. | \[0, 100] (takes effect when greater than 0) | 0 | | mem\_limit | The percentage of memory available for queries by this resource group on the current BE node. | (0, 1] (required) | - | | mem\_pool | Groups resource groups to share a memory limit. | String | default\_mem\_pool | | spill\_mem\_limit\_threshold | Memory usage threshold that triggers spilling to disk. | (0, 1] | 1.0 | | concurrency\_limit | Maximum number of concurrent queries in this resource group. | Integer (takes effect when greater than 0) | 0 | | big\_query\_cpu\_second\_limit | Maximum CPU time (in seconds) for big query tasks on each BE node. | Integer (takes effect when greater than 0) | 0 | | big\_query\_scan\_rows\_limit | Maximum number of rows big query tasks can scan on each BE node. | Integer (takes effect when greater than 0) | 0 | | big\_query\_mem\_limit | Maximum memory big query tasks can use on each BE node. | Integer (takes effect when greater than 0) | 0 | ###### CPU resource parameters[​](#cpu-resource-parameters "Direct link to CPU resource parameters") ###### `cpu_weight` and `cpu_weight_percent`[​](#cpu_weight-and-cpu_weight_percent "Direct link to cpu_weight-and-cpu_weight_percent") These parameters specify the CPU scheduling weight of a resource group on a single BE node, determining the relative share of CPU time allocated to tasks from this group. Before v3.3.5, they were referred to as `cpu_core_limit`. You can use one of the following parameters to set the CPU scheduling weight: * `cpu_weight`: Directly sets the CPU scheduling weight. Its value range is (0, `avg_be_cpu_cores`], where `avg_be_cpu_cores` is the average number of CPU cores across all BE nodes. The parameter is effective only when it is set to greater than 0. * `cpu_weight_percent`: Supported from v4.1. Sets the CPU scheduling weight as a percentage. Its value range is \[0, 100]. The parameter is effective only when it is set to greater than 0. If `min_be_cpu_cores * cpu_weight_percent / 100 < 1`, the system returns an error, where `min_be_cpu_cores` is the minimum number of CPU cores across all BE nodes. At runtime, each BE converts `cpu_weight_percent` to the actual `cpu_weight` based on the number of CPU cores on that BE (`be_cpu_cores`): `cpu_weight = be_cpu_cores * cpu_weight_percent / 100`. Only one of `cpu_weight`, `cpu_weight_percent`, `exclusive_cpu_cores`, and `exclusive_cpu_percent` can be greater than 0. > **NOTE** > > For example, suppose three resource groups, rg1, rg2, and rg3, have cpu\_weight values of 2, 6, and 8, respectively. On a fully loaded BE node, these groups would receive 12.5%, 37.5%, and 50% of the CPU time. If the node is not fully loaded and rg1 and rg2 are under load while rg3 is idle, rg1 and rg2 would receive 25% and 75% of the CPU time, respectively. ###### `exclusive_cpu_cores` and `exclusive_cpu_percent`[​](#exclusive_cpu_cores-and-exclusive_cpu_percent "Direct link to exclusive_cpu_cores-and-exclusive_cpu_percent") These parameters define CPU hard isolation for a resource group. It has two implications: * **Exclusive**: Reserves a specified number of CPU cores exclusively for this resource group, making them unavailable to other groups, even when idle. * **Quota**: Limits the resource group to only using these reserved CPU cores, preventing it from using available CPU resources from other groups. You can use one of the following parameters to set CPU hard isolation: * `exclusive_cpu_cores`: Directly sets the number of reserved CPU cores. Its value range is (0, `min_be_cpu_cores - 1`]. The parameter is effective only when it is set to greater than 0. * `exclusive_cpu_percent`: Supported from v4.1. Sets the number of reserved CPU cores as a percentage. Its value range is \[0, 100]. The parameter is effective only when it is set to greater than 0. If `min_be_cpu_cores * exclusive_cpu_percent / 100 < 1`, the system returns an error. At runtime, each BE converts `exclusive_cpu_percent` to the actual `exclusive_cpu_cores` based on the number of CPU cores on that BE (`be_cpu_cores`): `exclusive_cpu_cores = be_cpu_cores * exclusive_cpu_percent / 100`. * Resource groups with `exclusive_cpu_cores` or `exclusive_cpu_percent` greater than 0 are called Exclusive Resource Groups, and the CPU cores allocated to them are called Exclusive Cores. Other groups are called Shared Resource Groups and run on Shared Cores. * The total number of `exclusive_cpu_cores` across all Exclusive Resource Groups cannot exceed `min_be_cpu_cores - 1`. If `exclusive_cpu_percent` is used, the system first converts it to CPU cores based on `min_be_cpu_cores * exclusive_cpu_percent / 100` and then calculates the total. The upper limit is set to leave at least one Shared Core available. The relationship between `exclusive_cpu_cores`, `exclusive_cpu_percent`, `cpu_weight`, and `cpu_weight_percent`: Only one of `cpu_weight`, `cpu_weight_percent`, `exclusive_cpu_cores`, and `exclusive_cpu_percent` can be active at a time. Exclusive Resource Groups operate on their own reserved Exclusive Cores without requiring a share of CPU time through `cpu_weight` or `cpu_weight_percent`. You can configure whether Shared Resource Groups can borrow Exclusive Cores from Exclusive Resource Groups using the BE configuration `enable_resource_group_cpu_borrowing`. When set to `true` (default), Shared Resouce Groups can borrow CPU resources when Exclusive Resouce Groups are idle. To modify this configuration dynamically, use the following command: ```sql UPDATE information_schema.be_configs SET VALUE = "false" WHERE NAME = "enable_resource_group_cpu_borrowing"; ``` ###### Memory resource parameters[​](#memory-resource-parameters "Direct link to Memory resource parameters") ###### `mem_limit`[​](#mem_limit "Direct link to mem_limit") Specifies the percentage of memory (query pool) available to the resource group on the current BE node. The value range is (0,1]. ###### `mem_pool`[​](#mem_pool "Direct link to mem_pool") Since v4.0, specifies a shared memory pool identifier. Resource groups with the same mem\_pool identifier draw from a shared memory pool, collectively limited by the `mem_limit`. If not specified, the resource group is assigned to `default_mem_pool`, and its memory usage is limited solely by its own `mem_limit`. All resource groups sharing the same `mem_pool` must be configured with an identical `mem_limit`. To limit two resource groups to consume 50% of memory collectively, it could be defined in the following way: ```sql CREATE RESOURCE GROUP rg1 TO (db='db1') WITH ( 'mem_limit' = '50%', 'mem_pool' = 'shared_pool' ); CREATE RESOURCE GROUP rg2 TO (db='db1') WITH ( 'mem_limit' = '50%', 'mem_pool' = 'shared_pool' ); ``` ###### `spill_mem_limit_threshold`[​](#spill_mem_limit_threshold "Direct link to spill_mem_limit_threshold") Defines the memory usage threshold that triggers spilling to disk. The value range is (0,1], with the default being 1 (inactive). Introduced in v3.1.7. * When automatic spilling is enabled (`spill_mode` set to `auto`), but resource groups are disabled, the system will spill intermediate results to disk when a query’s memory usage exceeds 80% of `query_mem_limit`. * When resource groups are enabled, spilling will occur if: * The total memory usage of all queries in the group exceeds `current BE memory limit * mem_limit * spill_mem_limit_threshold`, or * The memory usage of the current query exceeds 80% of `query_mem_limit`. ###### Query concurrency parameters[​](#query-concurrency-parameters "Direct link to Query concurrency parameters") ###### `concurrency_limit`[​](#concurrency_limit "Direct link to concurrency_limit") Defines the maximum number of concurrent queries in the resource group to prevent system overload. Effective only when greater than 0, with a default value of 0. ###### Big query resource parameters[​](#big-query-resource-parameters "Direct link to Big query resource parameters") You can configure resource limits specifically for large queries using the following parameters: ###### `big_query_cpu_second_limit`[​](#big_query_cpu_second_limit "Direct link to big_query_cpu_second_limit") Specifies the maximum CPU time (in seconds) that large query tasks can use on each BE node, summing the actual CPU time used by parallel tasks. Effective only when greater than 0, with a default value of 0. ###### `big_query_scan_rows_limit`[​](#big_query_scan_rows_limit "Direct link to big_query_scan_rows_limit") Sets a limit on the number of rows large query tasks can scan on each BE node. Effective only when greater than 0, with a default value of 0. ###### `big_query_mem_limit`[​](#big_query_mem_limit "Direct link to big_query_mem_limit") Defines the maximum memory large query tasks can use on each BE node, in bytes. Effective only when greater than 0, with a default value of 0. > **NOTE** > > When a query running in a resource group exceeds the above big query limit, the query will be terminated with an error. You can also view error messages in the `ErrorCode` column of the FE node **fe.audit.log**. ###### Type (Deprecated Since v3.3.5)[​](#type-deprecated-since-v335 "Direct link to Type (Deprecated Since v3.3.5)") Before v3.3.5, StarRocks allowed setting the `type` of a resource group to `short_query`. However, the parameter `type` has been deprecated and replaced by `exclusive_cpu_cores`. For any existing resource groups of this type, the system will automatically convert them to an Exclusive resource group where the `exclusive_cpu_cores` value equals the `cpu_weight` after upgrading to v3.3.5. ###### System-defined resource groups[​](#system-defined-resource-groups "Direct link to System-defined resource groups") There are two system-defined resource groups in each StarRocks instance: `default_wg` and `default_mv_wg`. You can modify the configuration of system-defined resource groups using the ALTER RESOURCE GROUP command, but you cannot define classifiers for them or delete system-defined resource groups. ###### default\_wg[​](#default_wg "Direct link to default_wg") `default_wg` will be assigned to regular queries that are under the management of resource groups but don't match any classifier. The default resource limits of `default_wg` are as follows: * `cpu_weight`: The number of CPU cores of the BE. * `mem_limit`: 100%. * `concurrency_limit`: 0. * `big_query_cpu_second_limit`: 0. * `big_query_scan_rows_limit`: 0. * `big_query_mem_limit`: 0. * `spill_mem_limit_threshold`: 1. ###### default\_mv\_wg[​](#default_mv_wg "Direct link to default_mv_wg") `default_mv_wg` will be assigned to asynchronous materialized view refresh tasks if no resource group is allocated to the corresponding materialized view in the property `resource_group` during materialized view creation. The default resource limits of `default_mv_wg` are as follows: * `cpu_weight`: 1. * `mem_limit`: 80%. * `concurrency_limit`: 0. * `spill_mem_limit_threshold`: 80%. ##### classifier[​](#classifier "Direct link to classifier") Each classifier holds one or more conditions that can be matched to the properties of queries. StarRocks identifies the classifier that best matches each query based on the match conditions and assigns resources for running the query. Classifiers support the following conditions: * `user`: the name of the user. * `role`: the role of the user. * `query_type`: the type of the query. `SELECT` and `INSERT` (from v2.5) are supported. When INSERT INTO or BROKER LOAD tasks hit a resource group with `query_type` as `insert`, the BE node reserves the specified CPU resources for the tasks. * `source_ip`: the CIDR block from which the query is initiated. * `db`: the database which the query accesses. It can be specified by strings separated by commas `,`. * `plan_cpu_cost_range`: The estimated CPU cost range of the query. The format is `(DOUBLE, DOUBLE]`. The default value is NULL, indicating no such restriction. The `PlanCpuCost` column in `fe.audit.log` represents the system's estimate of the CPU cost for the query. This parameter is supported from v3.1.4 onwards. * `plan_mem_cost_range`: The system-estimated memory cost range of a query. The format is `(DOUBLE, DOUBLE]`. The default value is NULL, indicating no such restriction. The `PlanMemCost` column in `fe.audit.log` represents the system's estimate of the memory cost for the query. This parameter is supported from v3.1.4 onwards. A classifier matches a query only when one or all conditions of the classifier match the information about the query. If multiple classifiers match a query, StarRocks calculates the degree of matching between the query and each classifier and identifies the classifier with the highest degree of matching. > **NOTE** > > You can view the resource group to which a query belongs in the `ResourceGroup` column of the FE node **fe.audit.log** or by running `EXPLAIN VERBOSE `, as described in [View the resource group of a query](#view-the-resource-group-of-a-query). StarRocks calculates the degree of matching between a query and a classifier by using the following rules: * If the classifier has the same value of `user` as the query, the degree of matching of the classifier increases by 1. * If the classifier has the same value of `role` as the query, the degree of matching of the classifier increases by 1. * If the classifier has the same value of `query_type` as the query, the degree of matching of the classifier increases by 1 plus the number obtained from the following calculation: 1/Number of `query_type` fields in the classifier. * If the classifier has the same value of `source_ip` as the query, the degree of matching of the classifier increases by 1 plus the number obtained from the following calculation: (32 - `cidr_prefix`)/64. * If the classifier has the same value of `db` as the query, the degree of matching of the classifier increases by 10. * If the query's CPU cost falls within the `plan_cpu_cost_range`, the degree of matching of the classifier increases by 1. * If the query's memory cost falls within the `plan_mem_cost_range`, the degree of matching of the classifier increases by 1. If multiple classifiers match a query, the classifier with a larger number of conditions has a higher degree of matching. ```plain -- Classifier B has more conditions than Classifier A. Therefore, Classifier B has a higher degree of matching than Classifier A. classifier A (user='Alice') classifier B (user='Alice', source_ip = '192.168.1.0/24') ``` If multiple matching classifiers have the same number of conditions, the classifier whose conditions are described more accurately has a higher degree of matching. ```plain -- The CIDR block that is specified in Classifier B is smaller in range than Classifier A. Therefore, Classifier B has a higher degree of matching than Classifier A. classifier A (user='Alice', source_ip = '192.168.1.0/16') classifier B (user='Alice', source_ip = '192.168.1.0/24') -- Classifier C has fewer query types specified in it than Classifier D. Therefore, Classifier C has a higher degree of matching than Classifier D. classifier C (user='Alice', query_type in ('select')) classifier D (user='Alice', query_type in ('insert','select')) ``` If multiple classifiers have the same degree of matching, one of the classifiers will be randomly selected. ```plain -- If a query simultaneously queries both db1 and db2 and the classifiers E and F have the -- highest degree of matching among the hit classifiers, one of E and F will be randomly selected. classifier E (db='db1') classifier F (db='db2') ``` #### Isolate computing resources[​](#isolate-computing-resources "Direct link to Isolate computing resources") You can isolate computing resources among queries by configuring resource groups and classifiers. ##### Enable resource groups[​](#enable-resource-groups "Direct link to Enable resource groups") To use resource group, you must enable Pipeline Engine for your StarRocks cluster: ```sql -- Enable Pipeline Engine in the current session. SET enable_pipeline_engine = true; -- Enable Pipeline Engine globally. SET GLOBAL enable_pipeline_engine = true; ``` > **NOTE** > > From v3.1.0 onwards, Resource Group is enabled by default, and the session variable `enable_resource_group` is deprecated. ##### Create resource groups and classifiers[​](#create-resource-groups-and-classifiers "Direct link to Create resource groups and classifiers") Execute the following statement to create a resource group, associate the resource group with a classifier, and allocate computing resources to the resource group: ```sql CREATE RESOURCE GROUP TO ( user='string', role='string', query_type in ('select'), source_ip='cidr' ) --Create a classifier. If you create more than one classifier, separate the classifiers with commas (`,`). WITH ( "{ cpu_weight | exclusive_cpu_cores }" = "INT", "mem_limit" = "m%", "concurrency_limit" = "INT", "type" = "str" --The type of the resource group. Set the value to normal. ); ``` Example: ```sql CREATE RESOURCE GROUP rg1 TO (user='rg1_user1', role='rg1_role1', query_type in ('select'), source_ip='192.168.x.x/24'), (user='rg1_user2', query_type in ('select'), source_ip='192.168.x.x/24'), (user='rg1_user3', source_ip='192.168.x.x/24'), (user='rg1_user4'), (db='db1') WITH ( 'exclusive_cpu_cores' = '10', 'mem_limit' = '20%', 'big_query_cpu_second_limit' = '100', 'big_query_scan_rows_limit' = '100000', 'big_query_mem_limit' = '1073741824' ); ``` ##### Specify resource group (Optional)[​](#specify-resource-group-optional "Direct link to Specify resource group (Optional)") You can specify resource group for the current session directly, including `default_wg` and `default_mv_wg`. ```sql SET resource_group = 'group_name'; ``` ##### View resource groups and classifiers[​](#view-resource-groups-and-classifiers "Direct link to View resource groups and classifiers") Execute the following statement to query all resource groups and classifiers: ```sql SHOW RESOURCE GROUPS ALL; ``` Execute the following statement to query the resource groups and classifiers of the logged-in user: ```sql SHOW RESOURCE GROUPS; ``` Execute the following statement to query a specified resource group and its classifiers: ```sql SHOW RESOURCE GROUP group_name; ``` Example: ```plain mysql> SHOW RESOURCE GROUPS ALL; +---------------+-------+------------+---------------------+-----------+----------------------------+---------------------------+---------------------+-------------------+---------------------------+----------------------------------------+ | name | id | cpu_weight | exclusive_cpu_cores | mem_limit | big_query_cpu_second_limit | big_query_scan_rows_limit | big_query_mem_limit | concurrency_limit | spill_mem_limit_threshold | classifiers | +---------------+-------+------------+---------------------+-----------+----------------------------+---------------------------+---------------------+-------------------+---------------------------+----------------------------------------+ | default_mv_wg | 3 | 1 | 0 | 80.0% | 0 | 0 | 0 | null | 80% | (id=0, weight=0.0) | | default_wg | 2 | 1 | 0 | 100.0% | 0 | 0 | 0 | null | 100% | (id=0, weight=0.0) | | rge1 | 15015 | 0 | 6 | 90.0% | 0 | 0 | 0 | null | 100% | (id=15016, weight=1.0, user=rg1_user) | | rgs1 | 15017 | 8 | 0 | 90.0% | 0 | 0 | 0 | null | 100% | (id=15018, weight=1.0, user=rgs1_user) | | rgs2 | 15019 | 8 | 0 | 90.0% | 0 | 0 | 0 | null | 100% | (id=15020, weight=1.0, user=rgs2_user) | +---------------+-------+------------+---------------------+-----------+----------------------------+---------------------------+---------------------+-------------------+---------------------------+----------------------------------------+ ``` > **NOTE** > > In the preceding example, `weight` indicates the degree of matching. To query all fields of a resource group, including deprecated fields. By adding the keyword `VERBOSE` to the three commands mentioned above, you can view all fields of the resource group, including deprecated ones, such as `type` and `max_cpu_cores`. ```sql SHOW VERBOSE RESOURCE GROUPS ALL; SHOW VERBOSE RESOURCE GROUPS; SHOW VERBOSE RESOURCE GROUP group_name; ``` ##### Manage resource groups and classifiers[​](#manage-resource-groups-and-classifiers "Direct link to Manage resource groups and classifiers") You can modify the resource quotas for each resource group. You can also add or delete classifiers from resource groups. Execute the following statement to modify the resource quotas for an existing resource group: ```sql ALTER RESOURCE GROUP group_name WITH ( 'cpu_core_limit' = 'INT', 'mem_limit' = 'm%' ); ``` Execute the following statement to delete a resource group: ```sql DROP RESOURCE GROUP group_name; ``` Execute the following statement to add a classifier to a resource group: ```sql ALTER RESOURCE GROUP ADD (user='string', role='string', query_type in ('select'), source_ip='cidr'); ``` Execute the following statement to delete a classifier from a resource group: ```sql ALTER RESOURCE GROUP DROP (CLASSIFIER_ID_1, CLASSIFIER_ID_2, ...); ``` Execute the following statement to delete all classifiers of a resource group: ```sql ALTER RESOURCE GROUP DROP ALL; ``` #### Observe resource groups[​](#observe-resource-groups "Direct link to Observe resource groups") ##### View the resource group of a query[​](#view-the-resource-group-of-a-query "Direct link to View the resource group of a query") For queries that have not yet been executed, you can view the resource group matched by the query from the `RESOURCE GROUP` field returned by `EXPLAIN VERBOSE `. While a query is running, you can check which resource group the query has hit from the `ResourceGroup` field returned by `SHOW PROC '/current_queries'` and `SHOW PROC '/global_current_queries'`. After a query has completed, you can view the resource group that the query matched by checking the `ResourceGroup` field in the **fe.audit.log** file on the FE node. * If the query is not under the management of resource groups, the column value is an empty string `""`. * If the query is under the management of resource groups but doesn't match any classifier, it will be assigned to the default resource group `default_wg`. ##### Monitoring resource groups[​](#monitoring-resource-groups "Direct link to Monitoring resource groups") You can set up [monitoring and alerting](https://docs.starrocks.io/docs/administration/management/monitoring/Monitor_and_Alert.md) for your resource groups. Resource group-related FE and BE metrics are as follows. All the metrics below have a `name` label indicating their corresponding resource group. ##### FE metrics[​](#fe-metrics "Direct link to FE metrics") The following FE metrics only provide statistics within the current FE node: | Metric | Unit | Type | Description | | ----------------------------------------------------- | ----- | ------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | starrocks\_fe\_query\_resource\_group | Count | Instantaneous | The number of queries historically run in this resource group (including those currently running). | | starrocks\_fe\_query\_resource\_group\_latency | ms | Instantaneous | The query latency percentile for this resource group. The label `type` indicates specific percentiles, including `mean`, `75_quantile`, `95_quantile`, `98_quantile`, `99_quantile`, `999_quantile`. | | starrocks\_fe\_query\_resource\_group\_err | Count | Instantaneous | The number of queries in this resource group that encountered an error. | | starrocks\_fe\_resource\_group\_query\_queue\_total | Count | Instantaneous | The total number of queries historically queued in this resource group (including those currently running). This metric is supported from v3.1.4 onwards. It is valid only when query queues are enabled. | | starrocks\_fe\_resource\_group\_query\_queue\_pending | Count | Instantaneous | The number of queries currently in the queue of this resource group. This metric is supported from v3.1.4 onwards. It is valid only when query queues are enabled. | | starrocks\_fe\_resource\_group\_query\_queue\_timeout | Count | Instantaneous | The number of queries in this resource group that have timed out while in the queue. This metric is supported from v3.1.4 onwards. It is valid only when query queues are enabled. | ##### BE metrics[​](#be-metrics "Direct link to BE metrics") | Metric | Unit | Type | Description | | --------------------------------------------- | ---------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | resource\_group\_running\_queries | Count | Instantaneous | The number of queries currently running in this resource group. | | resource\_group\_total\_queries | Count | Instantaneous | The number of queries historically run in this resource group (including those currently running). | | resource\_group\_bigquery\_count | Count | Instantaneous | The number of queries in this resource group that triggered the big query limit. | | resource\_group\_concurrency\_overflow\_count | Count | Instantaneous | The number of queries in this resource group that triggered the `concurrency_limit` limit. | | resource\_group\_mem\_limit\_bytes | Bytes | Instantaneous | The memory limit for this resource group. | | resource\_group\_mem\_inuse\_bytes | Bytes | Instantaneous | The memory currently in use by this resource group. | | resource\_group\_cpu\_limit\_ratio | Percentage | Instantaneous | The ratio of this resource group's `cpu_core_limit` to the total `cpu_core_limit` across all resource groups. | | resource\_group\_inuse\_cpu\_cores | Count | Average | The estimated number of CPU cores in use by this resource group. This value is an approximate estimate. It represents the average value calculated based on the statistics from two consecutive metric collections. This metric is supported from v3.1.4 onwards. | | resource\_group\_cpu\_use\_ratio | Percentage | Average | **Deprecated** The ratio of the Pipeline thread time slices used by this resource group to the total Pipeline thread time slices used by all resource groups. It represents the average value calculated based on the statistics from two consecutive metric collections. | | resource\_group\_connector\_scan\_use\_ratio | Percentage | Average | **Deprecated** The ratio of the external table Scan thread time slices used by this resource group to the total Pipeline thread time slices used by all resource groups. It represents the average value calculated based on the statistics from two consecutive metric collections. | | resource\_group\_scan\_use\_ratio | Percentage | Average | **Deprecated** The ratio of the internal table Scan thread time slices used by this resource group to the total Pipeline thread time slices used by all resource groups. It represents the average value calculated based on the statistics from two consecutive metric collections. | ##### View resource group usage information[​](#view-resource-group-usage-information "Direct link to View resource group usage information") From v3.1.4 onwards, StarRocks supports the SQL statement [SHOW USAGE RESOURCE GROUPS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/resource_group/SHOW_USAGE_RESOURCE_GROUPS.md), which is used to display usage information for each resource group across BEs. The descriptions of each field are as follows: * `Name`: The name of the resource group. * `Id`: The ID of the resource group. * `Backend`: The BE's IP or FQDN. * `BEInUseCpuCores`: The number of CPU cores currently in use by this resource group on this BE. This value is an approximate estimate. * `BEInUseMemBytes`: The number of memory bytes currently in use by this resource group on this BE. * `BERunningQueries`: The number of queries from this resource group that are still running on this BE. Please note: * BEs periodically report this resource usage information to the Leader FE at the interval specified in `report_resource_usage_interval_ms`, which is by default set to 1 second. * The results will only show rows where at least one of `BEInUseCpuCores`/`BEInUseMemBytes`/`BERunningQueries` is a positive number. In other words, the information is displayed only when a resource group is actively using some resources on a BE. Example: ```plain MySQL [(none)]> SHOW USAGE RESOURCE GROUPS; +------------+----+-----------+-----------------+-----------------+------------------+ | Name | Id | Backend | BEInUseCpuCores | BEInUseMemBytes | BERunningQueries | +------------+----+-----------+-----------------+-----------------+------------------+ | default_wg | 0 | 127.0.0.1 | 0.100 | 1 | 5 | +------------+----+-----------+-----------------+-----------------+------------------+ | default_wg | 0 | 127.0.0.2 | 0.200 | 2 | 6 | +------------+----+-----------+-----------------+-----------------+------------------+ | wg1 | 0 | 127.0.0.1 | 0.300 | 3 | 7 | +------------+----+-----------+-----------------+-----------------+------------------+ | wg2 | 0 | 127.0.0.1 | 0.400 | 4 | 8 | +------------+----+-----------+-----------------+-----------------+------------------+ ``` ##### View thread information for Exclusive and Shared resource groups[​](#view-thread-information-for-exclusive-and-shared-resource-groups "Direct link to View thread information for Exclusive and Shared resource groups") Query execution mainly involves three thread pools: `pip_exec`, `pip_scan`, and `pip_con_scan`. * Exclusive resource groups run in their dedicated thread pools and are bound to the Exclusive CPU cores allocated to them. * Shared resource groups run in shared thread pools and are bound to the remaining Shared CPU cores. The threads in these three pools follow the naming convention `{ pip_exec | pip_scan | pip_con_scan }_{ com | }`, where `com` refers to the shared thread pool, and `` refers to the ID of the Exclusive resource group. You can view the CPU information bound to each BE thread through the system-defined view `information_schema.be_threads`. The fields `BE_ID`, `NAME`, and `BOUND_CPUS` represent the BE's ID, the name of the thread, and the number of CPU cores bound to that thread, respectively. ```sql select * from information_schema.be_threads where name like '%pip_exec%'; select * from information_schema.be_threads where name like '%pip_scan%'; select * from information_schema.be_threads where name like '%pip_con_scan%'; ``` Example: ```sql select BE_ID, NAME, FINISHED_TASKS, BOUND_CPUS from information_schema.be_threads where name like '%pip_exec_com%' and be_id = 10223; +-------+--------------+----------------+------------+ | BE_ID | NAME | FINISHED_TASKS | BOUND_CPUS | +-------+--------------+----------------+------------+ | 10223 | pip_exec_com | 2091295 | 10 | | 10223 | pip_exec_com | 2088025 | 10 | | 10223 | pip_exec_com | 1637603 | 6 | | 10223 | pip_exec_com | 1641260 | 6 | | 10223 | pip_exec_com | 1634197 | 6 | | 10223 | pip_exec_com | 1633804 | 6 | | 10223 | pip_exec_com | 1638184 | 6 | | 10223 | pip_exec_com | 1636374 | 6 | | 10223 | pip_exec_com | 2095951 | 10 | | 10223 | pip_exec_com | 2095248 | 10 | | 10223 | pip_exec_com | 2098745 | 10 | | 10223 | pip_exec_com | 2085338 | 10 | | 10223 | pip_exec_com | 2101221 | 10 | | 10223 | pip_exec_com | 2093901 | 10 | | 10223 | pip_exec_com | 2092364 | 10 | | 10223 | pip_exec_com | 2091366 | 10 | +-------+--------------+----------------+------------+ ``` --- ### Spill to disk Beta feature [Advice on use of Beta features](https://docs.starrocks.io/docs/introduction/maturity.md) This topic describes how to spill intermediate computation results of large operators to local disks and object storage. #### Overview[​](#overview "Direct link to Overview") For database systems that rely on in-memory computing for query execution, like StarRocks, they can consume substantial memory resources when processing queries with aggregate, sort, and join operators on a big dataset. When memory limits are reached, these queries are forcibly terminated due to out-of-memory (OOM). However, there are still chances that you want certain memory-intensive tasks to be completed stably and performance is not your top priority, for example, building a materialized view, or performing a lightweight ETL with INSERT INTO SELECT. These tasks can easily exhaust your memory resources and thereby block other queries running in your cluster. Usually, to address this issue, you can only fine-tune these tasks individually, and rely on your resource isolation strategy to control the query concurrency. This could be particularly inconvenient and likely to fail under some extreme scenarios. From StarRocks v3.0.1, StarRocks supports spilling the intermediate results of some memory-intensive operators to disks. With this feature, you can trade a tolerable drop in performance for a significant reduction in memory usage, thereby improving system availability. Currently, StarRocks' spilling feature supports the following operators: * Aggregate operators * Sort operators * Hash join (LEFT JOIN, RIGHT JOIN, FULL JOIN, OUTER JOIN, SEMI JOIN, and INNER JOIN) operators * CTE operators (Supported from v3.3.4 onwards) #### Enable intermediate result spilling[​](#enable-intermediate-result-spilling "Direct link to Enable intermediate result spilling") Follow these steps to enable intermediate result spilling: 1. Specify the local spill directory `spill_local_storage_dir`, which stores the spilled intermediate result on the local disk, in the BE configuration file **be.conf** or the CN configuration file **cn.conf**, and restart the cluster to allow the modification to take effect. ```properties spill_local_storage_dir=/[;/] ``` > **NOTE** > > * You can specify multiple `spill_local_storage_dir` by separating them with semicolons (`;`). > * In a production environment, we strongly recommend you use different disks for data storage and spilling. When intermediate results are spilled to disk, there could be a significant increase in both writing load and disk usage. If the same disk is used, this surge can impact other queries or tasks running in the cluster. 2. Execute the following statement to enable intermediate result spilling: ```sql SET enable_spill = true; ``` 3. Configure the mode of intermediate result spilling using the session variable `spill_mode`: ```sql SET spill_mode = { "auto" | "force" }; ``` > **NOTE** > > Each time a query with spilling completes, StarRocks automatically clears the spilled data the query produces. If BE crashes before clearing the data, StarRocks clears it when the BE is restarted. | **Variable** | **Default** | **Description** | | ------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | enable\_spill | false | Whether to enable intermediate result spilling. If it is set to `true`, StarRocks spills the intermediate results to disk to reduce the memory usage when processing aggregate, sort, or join operators in queries. | | spill\_mode | auto | The execution mode of intermediate result spilling. Valid values:- `auto`: Spilling is automatically triggered when the memory usage threshold is reached.
- `force`: StarRocks forcibly executes spilling for all relevant operators, regardless of memory usage.This variable takes effect only when the variable `enable_spill` is set to `true`. | #### \[Preview] Spill intermediate result to object storage[​](#preview-spill-intermediate-result-to-object-storage "Direct link to [Preview] Spill intermediate result to object storage") From v3.3.0 onwards, StarRocks supports spilling intermediate results to object storage. tip Before enabling spilling to object storage, you must create a storage volume to define the object storage you want to use. For detailed instruction on creating a storage volume, see [CREATE STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md). After you have enabled spilling in the previous step, you can further set these system variables to allow the intermediate results to be spilled to object storage: ```sql SET enable_spill_to_remote_storage = true; -- Replace with the name of the storage volume which you want to use. SET spill_storage_volume = ''; ``` After spilling to object storage has been enabled, the intermediate results of queries that triggered spilling will be first stored in the local disks of the BE or CN nodes, and then the object storage if the capacity limit of the local disks is reached. Please note that, if the storage volume you specified for `spill_storage_volume` does not exist, spilling to object storage will not be enabled. #### Limitations[​](#limitations "Direct link to Limitations") * Not all OOM issues can be resolved by spilling. For example, StarRocks cannot release the memory used for expression evaluation. * Usually, queries with spilling involved indicate a tenfold increase in query latency. We recommend you extend the query timeout for these queries by setting the session variable `query_timeout`. * There is a significant performance drop in spilling to object storage compared to spilling to local disks. * `spill_local_storage_dir` of each BE or CN node is shared among all queries running on the node. Currently, StarRocks does not support setting a size limit of spilled data to local disks individually for each query. Therefore, concurrent queries involved spilling may impact one another. --- ### Scale in and out This topic describes how to scale in and out the node of StarRocks. #### Scale FE in and out[​](#scale-fe-in-and-out "Direct link to Scale FE in and out") StarRocks has two types of FE nodes: Follower and Observer. Followers are involved in election voting and writing. Observers are only used to synchronize logs and extend read performance. > * The number of follower FEs (including leader) must be odd, and it is recommended to deploy 3 of them to form a High Availability (HA) mode. > * When the FE is in high availability deployment (1 leader, 2 followers), it is recommended to add Observer FEs for better read performance. ##### Scale FE out[​](#scale-fe-out "Direct link to Scale FE out") After deploying the FE node and starting the service, run the following command to scale FE out. ```sql alter system add follower "fe_host:edit_log_port"; alter system add observer "fe_host:edit_log_port"; ``` ##### Scale FE in[​](#scale-fe-in "Direct link to Scale FE in") FE scale-in is similar to the scale-out. Run the following command to scale FE in. ```sql alter system drop follower "fe_host:edit_log_port"; alter system drop observer "fe_host:edit_log_port"; ``` After the expansion and contraction, you can view the node information by running `show proc '/frontends';`. #### Scale BE in and out[​](#scale-be-in-and-out "Direct link to Scale BE in and out") StarRocks will automatically perform load-balancing after BE's are scaled in or out without affecting the overall performance. When you add a new BE node, the system's Tablet Scheduler will detect the new node and its low load. It will then start moving tablets from high-load BE nodes to the new, low-load BE node to ensure an even distribution of data and load across the entire cluster. The balancing process is based on a loadScore calculated for each BE, which considers both disk utilization and replica count. The system aims to move tablets from nodes with a higher loadScore to nodes with a lower loadScore. You can check the FE configuration parameter `tablet_sched_disable_balance` to ensure that automatic balancing is not disabled (the parameter is false by default, which means that tablet balancing is enabled by default). More details are in the [manage replica docs](https://docs.starrocks.io/docs/administration/management/resource_management/Replica.md). ##### Scale BE out[​](#scale-be-out "Direct link to Scale BE out") Run the following command to scale BE out. ```sql alter system add backend 'be_host:be_heartbeat_service_port'; ``` Run the following command to check the BE status. ```sql show proc '/backends'; ``` ##### Scale BE in[​](#scale-be-in "Direct link to Scale BE in") There are two ways to scale in a BE node – `DROP` and `DECOMMISSION`. `DROP` will delete the BE node immediately, and the lost duplicates will be made up by FE scheduling. `DECOMMISSION` will make sure the duplicates are made up first, and then drop the BE node. `DECOMMISSION` is a bit more friendly and is recommended for BE scale-in. The commands of both methods are similar: * `alter system decommission backend "be_host:be_heartbeat_service_port";` * `alter system drop backend "be_host:be_heartbeat_service_port";` Drop backend is a dangerous operation, so you need to confirm it twice before executing it * `alter system drop backend "be_host:be_heartbeat_service_port";` #### Scale CN in and out[​](#scale-cn-in-and-out "Direct link to Scale CN in and out") ##### Scale CN out[​](#scale-cn-out "Direct link to Scale CN out") Run the following command to scale CN out. ```sql ALTER SYSTEM ADD COMPUTE NODE "cn_host:cn_heartbeat_service_port"; ``` Run the following command to check the CN status. ```sql SHOW PROC '/compute_nodes'; ``` ##### Scale CN in[​](#scale-cn-in "Direct link to Scale CN in") CN scale-in is similar to the scale-out. Run the following command to scale CN in. ```sql ALTER SYSTEM DROP COMPUTE NODE "cn_host:cn_heartbeat_service_port"; ``` You can view the node information by running `SHOW PROC '/compute_nodes';`. --- ### Configure a time zone This topic describes how to configure a time zone and the impacts of time zone settings. #### Configure a session-level time zone or a global time zone[​](#configure-a-session-level-time-zone-or-a-global-time-zone "Direct link to Configure a session-level time zone or a global time zone") You can configure a session-level time zone or a global time zone for your StarRocks cluster using the `time_zone` parameter. * To configure a session-level time zone, execute the command `SET time_zone = 'xxx';`. You can configure different time zones for different sessions. The time zone setting becomes invalid if you disconnect with FEs. * To configure a global time zone, execute the command `SET global time_zone = 'xxx';`. The time zone setting is persisted in FEs and is valid even if you disconnect with FEs. > **Note** > > Before you load data into StarRocks, modify the global time zone of your StarRocks cluster to the same value of the `system_time_zone` parameter. Otherwise, after data loading, data of the DATE type are incorrect. The `system_time_zone` parameter refers to the time zone of the machines that are used to host FEs. When the machines are started, the time zone of the machines is recorded as the value of this parameter. You cannot manually configure this parameter. ##### Time zone format[​](#time-zone-format "Direct link to Time zone format") The value of the `time_zone` parameter is not case-sensitive. The value of the parameter can be in one of the following formats. | **Format** | **Example** | | -------------- | ---------------------------------------------------------------------------------- | | UTC offset | `SET time_zone = '+10:00';` `SET global time_zone = '-6:00';` | | Time zone name | `SET time_zone = 'Asia/Shanghai';` `SET global time_zone = 'America/Los_Angeles';` | For more information about time zone format, see [List of tz database time zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones). > **Note** > > Time zone abbreviations are not supported except for CST. If you set the value of `time_zone` to `CST`, StarRocks converts `CST` into `Asia/Shanghai`. ##### Default time zone[​](#default-time-zone "Direct link to Default time zone") The default value of the `time_zone` parameter is `Asia/Shanghai`. #### View time zone settings[​](#view-time-zone-settings "Direct link to View time zone settings") To view the time zone setting, run the following command. ```plaintext SHOW VARIABLES LIKE '%time_zone%'; ``` #### Impacts of time zone settings[​](#impacts-of-time-zone-settings "Direct link to Impacts of time zone settings") * Time zone settings affect the time values returned by the SHOW LOAD and SHOW BACKENDS statements. However, the settings do not affect the value specified in the `LESS THAN` clause when the partitioning columns specified in CREATE TABLE statement are of the DATE or DATETIME type. The settings also do not affect data of the DATE and DATETIME types. * Time zone settings affect the display and storage of the following functions: * **from\_unixtime**: returns a date and time of your specified time zone based on a specified UTC timestamp. For example, if the global time zone of your StarRocks cluster is `Asia/Shanghai`, `select FROM_UNIXTIME(0);` returns `1970-01-01 08:00:00`. * **unix\_timestamp**: returns a UTC timestamp based on the date and time of your specified time zone. For example, if the global time zone of your StarRocks cluster is `Asia/Shanghai`, `select UNIX_TIMESTAMP('1970-01-01 08:00:00');` returns `0`. * **curtime**: returns the current time of your specified time zone. For example, if the current time of a specified time zone is 16:34:05. `select CURTIME();` returns `16:34:05`. * **now**: returns the current date and time of your specified time zone. For example, if the current date and time of a specified time zone is 2021-02-11 16:34:13, `select NOW();` returns `2021-02-11 16:34:13`. * **convert\_tz**: converts the date and time from one time zone to another. For example, `select CONVERT_TZ('2021-08-01 11:11:11', 'Asia/Shanghai', 'America/Los_Angeles');` returns `2021-07-31 20:11:11`. --- ### Metadata Recovery This topic describes how to recover the metadata in your StarRocks clusters when FE nodes encounter different exceptions. Generally, you may have to resort to metadata recovery when and only when one of the following issues occurs: * [FE fails to restart](#fe-fails-to-restart) * [FE fails to provide services](#fe-fails-to-provide-services) * [Recover metadata on a new FE node with the metadata backup](#recover-metadata-on-a-new-fe-node-with-metadata-backup). Check the issue you encountered, follow the solution provided in the corresponding section, and perform any recommended actions. #### FE fails to restart[​](#fe-fails-to-restart "Direct link to FE fails to restart") FE nodes may fail to restart if the metadata is damaged or incompatible with the cluster after rollback. ##### Metadata incompatibility after rollback[​](#metadata-incompatibility-after-rollback "Direct link to Metadata incompatibility after rollback") When you downgrade your StarRocks cluster, FEs may fail to restart if the metadata is incompatible with that before the downgrade. You can identify this issue if you encounter the following exception while downgrading the cluster: ```plain UNKNOWN Operation Type xxx ``` You can follow these steps to recover the metadata and start FE: 1. Stop all FE nodes. 2. Back up the metadata directories `meta_dir` of all FE nodes. 3. Add the configuration `metadata_ignore_unknown_operation_type = true` to the configuration files **fe.conf** of all FE nodes. 4. Start all FE nodes, and check whether your data and metadata are intact. 5. If both your data and metadata are intact, execute the following statement to create an image file for your metadata: ```sql ALTER SYSTEM CREATE IMAGE; ``` 6. After the new image file is transmitted to the directory **meta/image** of all FE nodes, you can remove the configuration `metadata_ignore_unknown_operation_type= true` from all FE configuration files and restart the FE nodes. ##### Metadata damage[​](#metadata-damage "Direct link to Metadata damage") Both damage to BDBJE metadata and StarRocks metadata will cause failures to restart. ###### BDBJE metadata damage[​](#bdbje-metadata-damage "Direct link to BDBJE metadata damage") ###### VLSN Bug[​](#vlsn-bug "Direct link to VLSN Bug") You can identify the VLSN bug based on the following error message: ```plain recoveryTracker should overlap or follow on disk last VLSN of 6,684,650 recoveryFirst= 6,684,652 UNEXPECTED_STATE_FATAL: Unexpected internal state, unable to continue. Environment is invalid and must be closed. ``` You can follow these steps to fix this issue: 1. Clear the metadata directory `meta_dir` of the FE node that throws this exception. 2. Restart the FE node using the Leader FE node as the helper. ```bash # Replace with the IP address (priority_networks) # of the Leader FE node, and replace (Default: 9010) with # the Leader FE node's edit_log_port. ./fe/bin/start_fe.sh --helper : --daemon ``` tip * This bug has been fixed in StarRocks v3.1. You can avoid this issue by upgrading your cluster to v3.1 or later. * This solution does not apply if more than half of the FE nodes have encountered this issue. You must follow the instructions provided in [Measure of the last resort](#7-measure-of-the-last-resort) to fix the issue. ###### RollbackException[​](#rollbackexception "Direct link to RollbackException") You can identify this issue based on the following error message: ```plain must rollback 1 total commits(1 of which were durable) to the earliest point indicated by transaction id=-14752149 time=2022-01-12 14:36:28.21 vlsn=28,069,415 lsn=0x1174/0x16e durable=false in order to rejoin the replication group. All existing ReplicatedEnvironment handles must be closed and reinstantiated. Log files were truncated to file 0x4467, offset 0x269, vlsn 28,069,413 HARD_RECOVERY: Rolled back past transaction commit or abort. Must run recovery by re-opening Environment handles Environment is invalid and must be closed. ``` This issue occurs when the Leader FE node writes BDBJE metadata but fails to synchronize it to Follower FE nodes before it hangs. After being restarted, the original Leader becomes a Follower, corrupting the metadata. To solve this issue, you only need to restart this node again to wipe out the dirty metadata. ###### ReplicaWriteException[​](#replicawriteexception "Direct link to ReplicaWriteException") You can identify this issue based on the keyword `removeReplicaDb` from the FE log **fe.log**. ```plain Caused by: com.sleepycat.je.rep.ReplicaWriteException: (JE 18.3.16) Problem closing transaction 25000090. The current state is:REPLICA. The node transitioned to this state at:Fri Feb 23 01:31:00 UTC 2024 Problem seen replaying entry NameLN_TX/14 vlsn=1,902,818,939 isReplicated="1" txn=-953505106 dbop=REMOVE Originally thrown by HA thread: REPLICA 10.233.132.23_9010_1684154162022(6) at com.sleepycat.je.rep.txn.ReadonlyTxn.disallowReplicaWrite(ReadonlyTxn.java:114) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.dbi.DbTree.checkReplicaWrite(DbTree.java:880) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.dbi.DbTree.doCreateDb(DbTree.java:579) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.dbi.DbTree.createInternalDb(DbTree.java:507) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.cleaner.ExtinctionScanner.openDb(ExtinctionScanner.java:357) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.cleaner.ExtinctionScanner.prepareForDbExtinction(ExtinctionScanner.java:1703) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.dbi.DbTree.doRemoveDb(DbTree.java:1208) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.dbi.DbTree.removeReplicaDb(DbTree.java:1261) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.impl.node.Replay.applyNameLN(Replay.java:996) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.impl.node.Replay.replayEntry(Replay.java:722) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.impl.node.Replica$ReplayThread.run(Replica.java:1225) ~[starrocks-bdb-je-18.3.16.jar:?] ``` This issue occurs when the BDBJE version of the failed FE node (v18.3.\*) mismatches that of the Leader FE node (v7.3.7). You can follow these steps to fix this issue: 1. Drop the failed Follower or Observer node. ```sql -- To drop a Follower node, replace with the IP address (priority_networks) -- of the Follower node, and replace (Default: 9010) with -- the Follower node's edit_log_port. ALTER SYSTEM DROP FOLLOWER ":"; -- To drop an Observer node, replace with the IP address (priority_networks) -- of the Observer node, and replace (Default: 9010) with -- the Observer node's edit_log_port. ALTER SYSTEM DROP OBSERVER ":"; ``` 2. Add the failed node back to the cluster. ```sql -- Add the Follower node: ALTER SYSTEM ADD FOLLOWER ":"; -- Add the Observer node: ALTER SYSTEM ADD OBSERVER ":"; ``` 3. Clear the metadata directory `meta_dir` of the failed node. 4. Restart the failed node using the Leader FE node as the helper. ```bash # Replace with the IP address (priority_networks) # of the Leader FE node, and replace (Default: 9010) with # the Leader FE node's edit_log_port. ./fe/bin/start_fe.sh --helper : --daemon ``` 5. After the failed node recovers to a healthy status, you need to upgrade the BDBJE packages in your cluster to **starrocks-bdb-je-18.3.16.jar** (or upgrade your StarRocks cluster to v3.0 or later), following the order of Followers first and then the Leader. ###### InsufficientLogException[​](#insufficientlogexception "Direct link to InsufficientLogException") You can identify this issue based on the following error message: ```plain xxx INSUFFICIENT_LOG: Log files at this node are obsolete. Environment is invalid and must be closed. ``` This issue occurs when the Follower node requires full metadata synchronization. It may occur when one of the following situations happens: * The metadata on the Follower node lags behind that of the Leader node, which has already done a CheckPoint of metadata within itself. The Follower node cannot perform incremental updates on its metadata, thus full metadata synchronization is required. * The original Leader node writes and checkpoints its metadata, but fails to synchronize it to Follower FE nodes before it hangs. After being restarted, it becomes a Follower node. With dirty metadata checkpointed, the Follower node cannot perform incremental deletion of its metadata, thus full metadata synchronization is required. Please note that this exception will be thrown when a new Follower node is added to the cluster. In this case, you do not need to take any action. If this exception is thrown for an existing Follower node or the Leader node, you only need to restart the node. ###### HANDSHAKE\_ERROR: Error during the handshake between two nodes[​](#handshake_error-error-during-the-handshake-between-two-nodes "Direct link to HANDSHAKE_ERROR: Error during the handshake between two nodes") You can identify this issue based on the following error message: ```plain 2023-11-13 21:51:55,271 WARN (replayer|82) [BDBJournalCursor.wrapDatabaseException():97] failed to get DB names for 1 times!Got EnvironmentFailureExce com.sleepycat.je.EnvironmentFailureException: (JE 18.3.16) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 18.3.16) 10.26.5.115_9010_1697071897979(1):/data1/meta/bdb A replica with the name: 10.26.5.115_9010_1697071897979(1) is already active with the Feeder:null HANDSHAKE_ERROR: Error during the handshake between two nodes. Some validity or compatibility check failed, preventing further communication between the nodes. Environment is invalid and must be closed. at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:230) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1835) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.dbi.EnvironmentImpl.checkOpen(EnvironmentImpl.java:1844) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.Environment.checkOpen(Environment.java:2697) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.Environment.getDatabaseNames(Environment.java:2455) ~[starrocks-bdb-je-18.3.16.jar:?] at com.starrocks.journal.bdbje.BDBEnvironment.getDatabaseNamesWithPrefix(BDBEnvironment.java:478) ~[starrocks-fe.jar:?] at com.starrocks.journal.bdbje.BDBJournalCursor.refresh(BDBJournalCursor.java:177) ~[starrocks-fe.jar:?] at com.starrocks.server.GlobalStateMgr$5.runOneCycle(GlobalStateMgr.java:2148) ~[starrocks-fe.jar:?] at com.starrocks.common.util.Daemon.run(Daemon.java:115) ~[starrocks-fe.jar:?] at com.starrocks.server.GlobalStateMgr$5.run(GlobalStateMgr.java:2216) ~[starrocks-fe.jar:?] Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 18.3.16) 10.26.5.115_9010_1697071897979(1):/data1/meta/bdb A replica with the name: 10.26.5.115_9010_1697071897979(1) is already active with the Feeder:null HANDSHAKE_ERROR: Error during the handshake between two nodes. Some validity or compatibility check failed, preventing further communication between the nodes. Environment is invalid and must be closed. Originally thrown by HA thread: UNKNOWN 10.26.5.115_9010_1697071897979(1) Originally thrown by HA thread: UNKNOWN 10.26.5.115_9010_1697071897979(1) at com.sleepycat.je.rep.stream.ReplicaFeederHandshake.negotiateProtocol(ReplicaFeederHandshake.java:198) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.stream.ReplicaFeederHandshake.execute(ReplicaFeederHandshake.java:250) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.impl.node.Replica.initReplicaLoop(Replica.java:709) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.impl.node.Replica.runReplicaLoopInternal(Replica.java:485) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.impl.node.Replica.runReplicaLoop(Replica.java:412) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.impl.node.RepNode.run(RepNode.java:1869) ~[starrocks-bdb-je-18.3.16.jar:?] ``` This issue occurs when the original Leader node hangs and becomes alive again while the surviving Follower nodes are trying to elect a new Leader node. The Follower nodes will try to establish a new connection with the original Leader node. However, the Leader node will reject the connection request because the old connection still exists. Once the request is rejected, the Follower node will set the environment as invalid and throw this exception. To solve this issue, you can either increase the JVM heap size or use the G1 GC algorithm. ###### DatabaseNotFoundException[​](#databasenotfoundexception "Direct link to DatabaseNotFoundException") You can identify this issue based on the following error message: ```plain 2024-01-05 12:47:21,087 INFO (main|1) [BDBEnvironment.ensureHelperInLocal():340] skip check local environment because helper node and local node are identical. 2024-01-05 12:47:21,339 ERROR (MASTER 172.17.0.1_9112_1704430041062(-1)|1) [StarRocksFE.start():186] StarRocksFE start failed com.sleepycat.je.DatabaseNotFoundException: (JE 18.3.16) _jeRepGroupDB at com.sleepycat.je.rep.impl.RepImpl.openGroupDb(RepImpl.java:1974) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.impl.RepImpl.getGroupDb(RepImpl.java:1912) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.impl.RepGroupDB.reinitFirstNode(RepGroupDB.java:1439) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.impl.node.RepNode.reinitSelfElect(RepNode.java:1686) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.impl.node.RepNode.startup(RepNode.java:874) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.impl.node.RepNode.joinGroup(RepNode.java:2153) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.impl.RepImpl.joinGroup(RepImpl.java:618) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.ReplicatedEnvironment.joinGroup(ReplicatedEnvironment.java:558) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.ReplicatedEnvironment.(ReplicatedEnvironment.java:619) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.ReplicatedEnvironment.(ReplicatedEnvironment.java:464) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.ReplicatedEnvironment.(ReplicatedEnvironment.java:538) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.util.DbResetRepGroup.reset(DbResetRepGroup.java:262) ~[starrocks-bdb-je-18.3.16.jar:?] at com.starrocks.journal.bdbje.BDBEnvironment.initConfigs(BDBEnvironment.java:188) ~[starrocks-fe.jar:?] at com.starrocks.journal.bdbje.BDBEnvironment.setup(BDBEnvironment.java:174) ~[starrocks-fe.jar:?] at com.starrocks.journal.bdbje.BDBEnvironment.initBDBEnvironment(BDBEnvironment.java:153) ~[starrocks-fe.jar:?] at com.starrocks.journal.JournalFactory.create(JournalFactory.java:31) ~[starrocks-fe.jar:?] at com.starrocks.server.GlobalStateMgr.initJournal(GlobalStateMgr.java:1201) ~[starrocks-fe.jar:?] at com.starrocks.server.GlobalStateMgr.initialize(GlobalStateMgr.java:1150) ~[starrocks-fe.jar:?] at com.starrocks.StarRocksFE.start(StarRocksFE.java:129) ~[starrocks-fe.jar:?] at com.starrocks.StarRocksFE.main(StarRocksFE.java:83) ~[starrocks-fe.jar:?] ``` This issue occurs when you add the configuration `metadata_failure_recovery = true` in the FE configuration file **fe.conf**. To solve this issue, you need to remove the configuration and restart the node. ###### StarRocks metadata damage[​](#starrocks-metadata-damage "Direct link to StarRocks metadata damage") You can identify the StarRocks metadata damage issue based on one of the following error messages: ```plain failed to load journal type xxx ``` Or ```plain catch exception when replaying ``` warning Before proceeding to recover the metadata by following the solution provided below, you are strongly advised to seek assistance from the technical experts in the StarRocks community, because this solution may lead to **data loss**. You can follow these steps to fix this issue: ###### Ignore Error Journal ID (Preferred)[​](#ignore-error-journal-id-preferred "Direct link to Ignore Error Journal ID (Preferred)") 1. Shut down all FE nodes. 2. Back up the metadata directories of all FE nodes. 3. Locate the erroneous journal ID in the logs. `xxx` in the following log represents the erroneous journal ID. ```plain got interrupt exception or inconsistent exception when replay journal xxx, will exit ``` 4. Add the following configuration to all **fe.conf** files and start the FE nodes. ```plain metadata_journal_skip_bad_journal_ids=xxx ``` 5. If the startup still again, identify the new failed journal ID through Step 3, add it to the **fe.conf**, and then restart the nodes with the previous configurations unchanged. ```plain metadata_journal_skip_bad_journal_ids=xxx,yyy ``` 6. If the system still fails to start after the above steps, or if there are too many failed journal IDs, proceed to Recovery Mode. ###### Recovery Mode[​](#recovery-mode "Direct link to Recovery Mode") 1. Stop all FE nodes. 2. Back up the metadata directories `meta_dir` of all FE nodes. 3. Add the configuration `metadata_enable_recovery_mode = true` to the configuration files **fe.conf** of all FE nodes. Note that data loading is forbidden in this mode. 4. Start all FE nodes, and query tables in the cluster to check whether your data is intact. You must wait until metadata recovery is completed if the following error is returned when you query these tables: ```plain ERROR 1064 (HY000): capture_consistent_versions error: version already been compacted. ``` You can execute the following statement from the Leader FE node to view the progress of metadata recovery: ```sql SHOW PROC '/meta_recovery'; ``` This statement will show the partitions that failed to be recovered. You can follow the advice returned to recover the partitions. If nothing is returned, it indicates the recovery is successful. 5. If both your data and metadata are intact, execute the following statement to create an image file for your metadata: ```sql ALTER SYSTEM CREATE IMAGE; ``` 6. After the new image file is transmitted to the directory **meta/image** of all FE nodes, you can remove the configuration `metadata_enable_recovery_mode = true` from all FE configuration files and restart the FE nodes. #### FE fails to provide services[​](#fe-fails-to-provide-services "Direct link to FE fails to provide services") FE will not provide services when Follower FE nodes fail to perform Leader election. When this issue occurs, you may find the following log record repeating: ```plain wait globalStateMgr to be ready. FE type: INIT. is ready: false ``` A variety of exceptions may cause this issue. You are strongly advised to troubleshoot the issue step by step following the sections below. Applying inappropriate solutions will deteriorate the problem and probably lead to data loss. ##### 1. The majority of Follower nodes are not running[​](#1-the-majority-of-follower-nodes-are-not-running "Direct link to 1. The majority of Follower nodes are not running") If the majority of the Follower nodes are not running, the FE group will not provide services. Here, 'majority' indicates `1 + (Follower node count/2)`. Please note that the Leader FE node itself is a Follower, but Observer nodes are not Followers. * You can identify the role of each FE node from the **fe/meta/image/ROLE** file: ```bash cat fe/meta/image/ROLE #Fri Jan 19 20:03:14 CST 2024 role=FOLLOWER hostType=IP name=172.26.92.154_9312_1705568349984 ``` * You can view the total number of Follower nodes from the BDBJE log: ```bash grep "Current group size" fe/meta/bdb/je.info.0 # The example output indicates there are three Follower nodes in the cluster. 2024-01-24 08:21:44.754 UTC INFO [172.26.92.139_29917_1698226672727] Current group size: 3 ``` To solve this issue, you need to start all Follower nodes in the cluster. If they cannot be restarted, please refer to [The measure of last resort](#10-measure-of-the-last-resort). ##### 2. Node IP is changed[​](#2-node-ip-is-changed "Direct link to 2. Node IP is changed") If the `priority_networks` of the node is not configured, the FE node will randomly select an available IP address once it is restarted. If the IP address recorded in the BDBJE metadata is different from that used to start the node, the FE will not provide services. * You can view the IP address recorded in the BDBJE metadata from the **fe/meta/image/ROLE** file: ```bash cat fe/meta/image/ROLE #Fri Jan 19 20:03:14 CST 2024 role=FOLLOWER hostType=IP name=172.26.92.154_9312_1705568349984 ``` The value `172.26.92.154` before the first underscore is the IP address recorded in the BDBJE metadata. * You can view the IP address used to start the node from the FE log: ```bash grep "IP:" fe/log/fe.log 2024-02-06 14:33:58,211 INFO (main|1) [FrontendOptions.initAddrUseIp():249] Use IP init local addr, IP: /172.17.0.1 2024-02-06 14:34:27,689 INFO (main|1) [FrontendOptions.initAddrUseIp():249] Use IP init local addr, IP: /172.17.0.1 ``` To solve this issue, you need to set the `priority_networks` of the node in the FE configuration file **fe.conf** to the IP address recorded in **fe/meta/image/ROLE**, and restart the node. ##### 3. System clock among nodes is not synchronized[​](#3-system-clock-among-nodes-is-not-synchronized "Direct link to 3. System clock among nodes is not synchronized") You can identify this issue based on the following error message from **fe.out**, **fe.log** or **fe/meta//bdb/je.info.0**: ```plain com.sleepycat.je.EnvironmentFailureException: (JE 7.3.7) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 7.3.7) 172.26.92.139_29917_1631006307557(2180):xxx Clock delta: 11020 ms. between Feeder: 172.26.92.154_29917_1641969377236 and this Replica exceeds max permissible delta: 5000 ms. HANDSHAKE_ERROR: Error during the handshake between two nodes. Some validity or compatibility check failed, preventing further communication between the nodes. Environment is invalid and must be closed. fetchRoot of 0x1278/0x1fcbb8 state=0 expires=never ``` You must synchronize the system clock among all nodes. ##### 4. Available disk space is insufficient[​](#4-available-disk-space-is-insufficient "Direct link to 4. Available disk space is insufficient") After you upgrade StarRocks to v3.0 or later, or upgrade BDBJE to v18 or later, the node may fail to restart when the available space of the disk that stores `meta_dir` is less than 5 GB. You can view the BDBJE version from the **.jar** package under the directory **fe/lib**. To solve this issue, you can scale up the disk, or allocate a dedicated disk with larger capacity for the FE metadata. ##### 5. `edit_log_port` is changed[​](#5-edit_log_port-is-changed "Direct link to 5-edit_log_port-is-changed") If the `edit_log_port` recorded in the BDBJE metadata is different from that configured in **fe.conf**, the FE will not provide services. You can view the `edit_log_port` recorded in the BDBJE metadata from the **fe/meta/image/ROLE** file: ```bash cat fe/meta/image/ROLE #Fri Jan 19 20:03:14 CST 2024 role=FOLLOWER hostType=IP name=172.26.92.154_9312_1705568349984 ``` The value `9312` before the second underscore is the `edit_log_port` recorded in the BDBJE metadata. To solve this issue, you need to set the `edit_log_port` of the node in the FE configuration file **fe.conf** to the `edit_log_port` recorded in **fe/meta/image/ROLE**, and restart the node. ##### 6. JVM heap size is insufficient[​](#6-jvm-heap-size-is-insufficient "Direct link to 6. JVM heap size is insufficient") You can view the JVM memory usage using the `jstat` command: ```plain jstat -gcutil pid 1000 1000 S0 S1 E O M CCS YGC YGCT FGC FGCT GCT 0.00 100.00 27.78 95.45 97.77 94.45 24 0.226 1 0.065 0.291 0.00 100.00 44.44 95.45 97.77 94.45 24 0.226 1 0.065 0.291 0.00 100.00 55.56 95.45 97.77 94.45 24 0.226 1 0.065 0.291 0.00 100.00 72.22 95.45 97.77 94.45 24 0.226 1 0.065 0.291 0.00 100.00 88.89 95.45 97.77 94.45 24 0.226 1 0.065 0.291 0.00 100.00 5.26 98.88 97.80 94.45 25 0.231 1 0.065 0.297 0.00 100.00 21.05 98.88 97.80 94.45 25 0.231 1 0.065 0.297 0.00 100.00 31.58 98.88 97.80 94.45 25 0.231 1 0.065 0.297 0.00 100.00 47.37 98.88 97.80 94.45 25 0.231 1 0.065 0.297 0.00 100.00 63.16 98.88 97.80 94.45 25 0.231 1 0.065 0.297 0.00 100.00 73.68 98.88 97.80 94.45 25 0.231 1 0.065 0.297 ``` If the percentages shown in field `O` remain high, it indicates that the JVM heap size is insufficient. To solve this issue, you must increase the JVM heap size. ##### 7. Latch timeout. com.sleepycat.je.log.LogbufferPool\_FullLatch[​](#7-latch-timeout-comsleepycatjeloglogbufferpool_fulllatch "Direct link to 7. Latch timeout. com.sleepycat.je.log.LogbufferPool_FullLatch") You can identify this issue based on the following error message: ```plain Environment invalid because of previous exception: xxx Latch timeout. com.sleepycat.je.log.LogbufferPool_FullLatch xxx' at com.sleepycat.je.EnvironmentFailureException.unexpectedState(EnvironmentFailureException.java:459) at com.sleepycat.je.latch.LatchSupport.handleTimeout(LatchSupport.java:211) at com.sleepycat.je.latch.LatchWithStatsImpl.acquireExclusive(LatchWithStatsImpl.java:87) at com.sleepycat.je.log.LogBufferPool.bumpCurrent(LogBufferPool.java:527) at com.sleepycat.je.log.LogManager.flushInternal(LogManager.java:1373) at com.sleepycat.je.log.LogManager.flushNoSync(LogManager.java:1337) at com.sleepycat.je.log.LogFlusher$FlushTask.run(LogFlusher.java:232) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) ``` This issue occurs when there is excessive pressure on the local disk of the FE node. To solve this issue, you can allocate a dedicated disk for the FE metadata, or replace the disk with a high-performance one. ##### 8. InsufficientReplicasException[​](#8-insufficientreplicasexception "Direct link to 8. InsufficientReplicasException") You can identify this issue based on the following error message: ```plain com.sleepycat.je.rep.InsufficientReplicasException: (JE 7.3.7) Commit policy: SIMPLE_MAJORITY required 1 replica. But none were active with this master. ``` This issue occurs when the Leader FE node or Follower FE nodes use excessive memory resources, leading to Full GC. To solve this issue, you can either increase the JVM heap size or use the G1 GC algorithm. ##### 9. UnknownMasterException[​](#9-unknownmasterexception "Direct link to 9. UnknownMasterException") You can identify this issue based on the following error message: ```plain com.sleepycat.je.rep.UnknownMasterException: (JE 18.3.16) Could not determine master from helpers at:[/xxx.xxx.xxx.xxx:9010, /xxx.xxx.xxx.xxx:9010] at com.sleepycat.je.rep.elections.Learner.findMaster(Learner.java:443) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.util.ReplicationGroupAdmin.getMasterSocket(ReplicationGroupAdmin.java:186) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.util.ReplicationGroupAdmin.doMessageExchange(ReplicationGroupAdmin.java:607) ~[starrocks-bdb-je-18.3.16.jar:?] at com.sleepycat.je.rep.util.ReplicationGroupAdmin.getGroup(ReplicationGroupAdmin.java:406) ~[starrocks-bdb-je-18.3.16.jar:?] at com.starrocks.ha.BDBHA.getElectableNodes(BDBHA.java:178) ~[starrocks-fe.jar:?] at com.starrocks.common.proc.FrontendsProcNode.getFrontendsInfo(FrontendsProcNode.java:96) ~[starrocks-fe.jar:?] at com.starrocks.common.proc.FrontendsProcNode.fetchResult(FrontendsProcNode.java:80) ~[starrocks-fe.jar:?] at com.starrocks.sql.ast.ShowProcStmt.getMetaData(ShowProcStmt.java:74) ~[starrocks-fe.jar:?] at com.starrocks.qe.ShowExecutor.handleShowProc(ShowExecutor.java:872) ~[starrocks-fe.jar:?] at com.starrocks.qe.ShowExecutor.execute(ShowExecutor.java:286) ~[starrocks-fe.jar:?] at com.starrocks.qe.StmtExecutor.handleShow(StmtExecutor.java:1574) ~[starrocks-fe.jar:?] at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:688) ~[starrocks-fe.jar:?] at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:336) ~[starrocks-fe.jar:?] at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:530) ~[starrocks-fe.jar:?] at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:838) ~[starrocks-fe.jar:?] at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:69) ~[starrocks-fe.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:829) ~[?:?] ``` When executing `SHOW FRONTENDS`, if the Leader FE node cannot be found, there could be several reasons: * If it is observed that more than half of the FE nodes are undergoing Full GC, and the duration is notably long. * Or if the log contains the keyword `java.lang.OutOfMemoryError: Java heap space`. It is because there is insufficient memory. You need to increase the JVM memory allocation. ##### 10. Measure of the last resort[​](#10-measure-of-the-last-resort "Direct link to 10. Measure of the last resort") warning You can try the following solution as the last resort only when none of the preceding solutions works. This solution is designed only for extreme cases, such as: * The majority of Follower nodes cannot be restarted. * Follower nodes cannot perform Leader election due to bugs of BDBJE. * Exceptions other than those mentioned in preceding sections. Follow these steps to recover the metadata: 1. Stop all FE nodes. 2. Back up the metadata directories `meta_dir` of all FE nodes. 3. Run the following commands on **all the servers that host the FE nodes** to identify the node with the latest metadata. ```bash # You need to specify the exact .jar package the node uses in the command # as the package varies according to the StarRocks version. java -jar fe/lib/starrocks-bdb-je-18.3.16.jar DbPrintLog -h meta/bdb/ -vd ``` Example output: ```bash file 0x3b numRepRecords = 24479 firstVLSN = 1,434,126 lastVLSN = 1,458,604 file 0x3c numRepRecords = 22541 firstVLSN = 1,458,605 lastVLSN = 1,481,145 file 0x3d numRepRecords = 25176 firstVLSN = 1,481,146 lastVLSN = 1,506,321 ...... file 0x74 numRepRecords = 26903 firstVLSN = 2,927,458 lastVLSN = 2,954,360 file 0x75 numRepRecords = 26496 firstVLSN = 2,954,361 lastVLSN = 2,980,856 file 0x76 numRepRecords = 18727 firstVLSN = 2,980,857 lastVLSN = 2,999,583 ... 0 files at end First file: 0x3b Last file: 0x76 ``` The node with the largest `lastVLSN` value has the latest metadata. 4. Find the role (Follower or Observer) of the FE node that has the latest metadata from the **fe/meta/image/ROLE** file: ```bash cat fe/meta/image/ROLE #Fri Jan 19 20:03:14 CST 2024 role=FOLLOWER hostType=IP name=172.26.92.154_9312_1705568349984 ``` If multiple nodes have the latest metadata, it is recommended to proceed with a Follower node. If multiple Follower nodes have the latest metadata, you can proceed with any of them. 5. Perform the corresponding operations based on the role of the FE node you chose in the previous step. * Proceed with Follower Node * Proceed with Observer Node If a Follower node has the latest metadata, perform the following operations: 1. Add the following configurations to **fe.conf**: ```properties bdbje_reset_election_group = true ``` 2. Restart the node, and check whether your data and metadata are intact. 3. Check whether the current FE node is the Leader FE node. ```sql SHOW FRONTENDS; ``` * If the field `Alive` is `true`, this FE node is properly started and added to the cluster. * If the field `Role` is `LEADER`, this FE node is the Leader FE node. 4. If the data and metadata are intact, and the role of the node is Leader, you can remove the configuration you added earlier and restart the node. If an Observer node has the latest metadata, perform the following operations: 1. Change the role of the FE node from `OBSERVER` to `FOLLOWER` in the **fe/meta/image/ROLE** file. 2. Add the following configurations to **fe.conf**: ```properties bdbje_reset_election_group = true ``` 3. Restart the node, and check whether your data and metadata are intact. 4. Check whether the current FE node is the Leader FE node. ```sql SHOW FRONTENDS; ``` * If the field `Alive` is `true`, this FE node is properly started and added to the cluster. * If the field `Role` is `LEADER`, this FE node is the Leader FE node. 5. If the data and metadata are intact, and the role of the node is Leader, you can remove the configuration you added earlier. However, **do not restart the node**. 6. Add a new Follower node (on a new server) to the cluster. ```sql ALTER SYSTEM ADD FOLLOWER ":"; ``` 7. Start a new FE node on the new server using the temporary Leader FE node as the helper. ```bash # Replace with the IP address (priority_networks) # of the Leader FE node, and replace (Default: 9010) with # the Leader FE node's edit_log_port. ./fe/bin/start_fe.sh --helper : --daemon ``` 8. Once the new FE node is started successfully, check the status and roles of both FE nodes: ```sql SHOW FRONTENDS; ``` * If the field `Alive` is `true`, this FE node is properly started and added to the cluster. * If the field `Role` is `FOLLOWER`, this FE node is a Follower FE node. * If the field `Role` is `LEADER`, this FE node is the Leader FE node. 9. If the new Follower is successfully running in the cluster, you can then stop all nodes. 10. Add the following configurations to the **fe.conf of the new Follower only**: * For StarRocks v2.5, v3.0, v3.1.9 and earlier patch versions, and v3.2.4 and earlier patch versions: ```properties metadata_failure_recovery = true ``` * For StarRocks v3.1.10 and later patch versions, v3.2.5 and later patch versions, and v3.3 and later: ```properties bdbje_reset_election_group = true ``` 11. Restart the new Follower node, and check whether your data and metadata are intact. 12. Check whether the current FE node is the Leader FE node. ```sql SHOW FRONTENDS; ``` * If the field `Alive` is `true`, this FE node is properly started and added to the cluster. * If the field `Role` is `LEADER`, this FE node is the Leader FE node. 13. If the data and metadata are intact, and the role of the node is Leader, you can remove the configuration you added earlier and restart the node. 5) Clear the metadata directories `meta_dir` of the FE nodes that you want to add back to the cluster. 6) Start new Follower nodes using the new Leader FE node as the helper. ```bash # Replace with the IP address (priority_networks) # of the Leader FE node, and replace (Default: 9010) with # the Leader FE node's edit_log_port. ./fe/bin/start_fe.sh --helper : --daemon ``` 7) Add the Follower nodes back to the cluster. ```sql ALTER SYSTEM ADD FOLLOWER ":"; ``` After all nodes are added back to the cluster, the metadata is successfully recovered. #### Recover metadata on a new FE node with metadata backup[​](#recover-metadata-on-a-new-fe-node-with-metadata-backup "Direct link to Recover metadata on a new FE node with metadata backup") Follow these steps if you want to start a new FE node with the metadata backup: 1. Copy the backup metadata directory `meta_dir` to the new FE node. 2. In the configuration file of the FE node, set `bdbje_reset_election_group` to `true`. ```properties bdbje_reset_election_group = true ``` 3. Start the FE node. ```bash ./fe/bin/start_fe.sh ``` 4. Check whether the current FE node is the Leader FE node. ```sql SHOW FRONTENDS; ``` If the field `Role` is `LEADER`, this FE node is the Leader FE node. Make sure its IP address is the that of the current FE node. 5. If the data and metadata are intact, and the role of the node is Leader, you must remove the configuration `bdbje_reset_election_group` and restart the node. 6. Now you have successfully start a new Leader FE node with the metadata backup. You can add new Follower nodes using the new Leader FE node as the helper. ```bash # Replace with the IP address (priority_networks) # of the Leader FE node, and replace (Default: 9010) with # the Leader FE node's edit_log_port. ./fe/bin/start_fe.sh --helper : --daemon ``` #### Metadata recovery-related configurations[​](#metadata-recovery-related-configurations "Direct link to Metadata recovery-related configurations") tip You must remove the following configuration once the metadata recovery is completed. * [bdbje\_reset\_election\_group](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#bdbje_reset_election_group) * [metadata\_enable\_recovery\_mode](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#metadata_enable_recovery_mode) * [metadata\_ignore\_unknown\_operation\_type](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#metadata_ignore_unknown_operation_type) --- ### SQL Digest This topic introduces the SQL Digest feature of StarRocks. This feature is supported from v3.3.6 onwards. #### Overview[​](#overview "Direct link to Overview") SQL Digest is a fingerprint generated by historical SQL statements with parameters removed. It helps cluster SQL statements with the same structure but different parameters. Common use cases of SQL Digest include: * Finding other SQL statements with the same structure but different parameters in query history * Tracking execution frequency, cumulative time, and other statistics of SQL with the same structure * Identifying the most time-consuming SQL patterns in the system In StarRocks, SQL Digests are mainly recorded through audit logs **fe.audit.log**. For example, execute the following two SQL statements: ```sql SELECT count(*) FROM lineorder WHERE lo_orderdate > '19920101'; SELECT count(*) FROM lineorder WHERE lo_orderdate > '19920202'; ``` Two same Digest will be generated in **fe.audit.log**: ```sql Digest=f58bb71850d112014f773717830e7f77 Digest=f58bb71850d112014f773717830e7f77 ``` #### Usage[​](#usage "Direct link to Usage") ##### Prerequisites[​](#prerequisites "Direct link to Prerequisites") To enable this feature, you must set the FE configuration item `enable_sql_digest` to `true`. Execute the following statement to enable it dynamically: ```sql ADMIN SET FRONTEND CONFIG ('enable_sql_digest'='true'); ``` To enable it permanently, you must add `enable_sql_digest = true` to the FE configuration file `fe.conf` and restart FE. After enabling this feature, you can install the [AuditLoader](https://docs.starrocks.io/docs/administration/management/audit_loader.md) plugin to perform statistical analysis on SQL statements. ##### Find similar SQL[​](#find-similar-sql "Direct link to Find similar SQL") ```sql SELECT * FROM starrocks_audit_db__.starrocks_audit_tbl__ WHERE digest = '' LIMIT 1; ``` ##### Track daily execution count and time of similar SQL[​](#track-daily-execution-count-and-time-of-similar-sql "Direct link to Track daily execution count and time of similar SQL") ```sql SELECT date_trunc('day', `timestamp`) query_date, count(*), sum(queryTime), sum(scanRows), sum(cpuCostNs), sum(memCostBytes) FROM starrocks_audit_db__.starrocks_audit_tbl__ WHERE digest = '' GROUP BY query_date ORDER BY query_date DESC LIMIT 30; ``` ##### Calculate average execution time of similar SQL[​](#calculate-average-execution-time-of-similar-sql "Direct link to Calculate average execution time of similar SQL") ```sql SELECT avg(queryTime), min(queryTime), max(queryTime), stddev(queryTime) FROM starrocks_audit_db__.starrocks_audit_tbl__ WHERE digest = ''; ``` ##### Aggregate similar SQL to analyze the most time-consuming pattern[​](#aggregate-similar-sql-to-analyze-the-most-time-consuming-pattern "Direct link to Aggregate similar SQL to analyze the most time-consuming pattern") ```sql WITH top_sql AS ( SELECT digest, sum(queryTime) FROM starrocks_audit_db__.starrocks_audit_tbl__ GROUP BY digest ORDER BY sum(queryTime) DESC LIMIT 10 ) SELECT * FROM starrocks_audit_db__.starrocks_audit_tbl__ WHERE digest IN (SELECT digest FROM top_sql); ``` #### Parameter normalization rules[​](#parameter-normalization-rules "Direct link to Parameter normalization rules") * Constant values in SQL will be normalized. For example, similar SQL statements with `WHERE a = 1` and `WHERE a = 2` will have the same Digest. * IN predicates will be normalized. For example, similar SQL statements with `IN (1,2,3)` and `IN (1,2)` will have the same Digest. * `LIMIT N` clauses will be normalized. For example, similar SQL statements with `LIMIT 10` and `LIMIT 30` will have the same Digest. --- ### Deploy and Manage StarRocks with StarGo This topic describes how to deploy and manage StarRocks clusters with StarGo. StarGo is a command line tool for multiple StarRocks clusters management. You can easily deploy, check, upgrade, downgrade, start, and stop multiple clusters through StarGo. #### Install StarGo[​](#install-stargo "Direct link to Install StarGo") Download the following files to your central control node: * **sr-ctl**: The binary file of StarGo. You do not need to install it after downloading it. * **sr-c1.yaml**: The template for the deployment configuration file. * **repo.yaml**: The configuration file for the download path of StarRocks installer. > Note You can access `http://cdn-thirdparty.starrocks.com` to get the corresponding installation index files and installers. ```shell wget https://github.com/wangtianyi2004/starrocks-controller/raw/main/stargo-pkg.tar.gz wget https://github.com/wangtianyi2004/starrocks-controller/blob/main/sr-c1.yaml wget https://github.com/wangtianyi2004/starrocks-controller/blob/main/repo.yaml ``` Grant **sr-ctl** access. ```shell chmod 751 sr-ctl ``` #### Deploy StarRocks cluster[​](#deploy-starrocks-cluster "Direct link to Deploy StarRocks cluster") You can deploy a StarRocks cluster with StarGo. ##### Prerequisites[​](#prerequisites "Direct link to Prerequisites") * The cluster to be deployed must have at least one central control node and three deployment nodes. All nodes can be deployed on one machine. * You need to deploy StarGo on the central control node. * You need to build mutual SSH authentication between the central control node and three deployment nodes. The following example builds mutual authentication between central control node sr-dev@r0 and three deployment nodes starrocks@r1, starrocks@r2, and starrocks@r3. ```plain ## Build the mutual authentication between sr-dev@r0 and starrocks@r1, 2, 3. [sr-dev@r0 ~]$ ssh-keygen [sr-dev@r0 ~]$ ssh-copy-id starrocks@r1 [sr-dev@r0 ~]$ ssh-copy-id starrocks@r2 [sr-dev@r0 ~]$ ssh-copy-id starrocks@r3 ## Verify the mutual authentication between sr-dev@r0 and starrocks@r1, 2, 3. [sr-dev@r0 ~]$ ssh starrocks@r1 date [sr-dev@r0 ~]$ ssh starrocks@r2 date [sr-dev@r0 ~]$ ssh starrocks@r3 date ``` ##### Create configuration file[​](#create-configuration-file "Direct link to Create configuration file") Create the StarRocks deployment topology file based on the following YAML template. See [Configuration](https://docs.starrocks.io/docs/administration/management/FE_configuration.md) for detailed information. ```yaml global: user: "starrocks" ## The current OS user. ssh_port: 22 fe_servers: - host: 192.168.XX.XX ssh_port: 22 http_port: 8030 rpc_port: 9020 query_port: 9030 edit_log_port: 9010 deploy_dir: StarRocks/fe meta_dir: StarRocks/fe/meta log_dir: StarRocks/fe/log priority_networks: 192.168.XX.XX/24 # Specify the unique IP for current node when the machine has multiple IP addresses. config: sys_log_level: "INFO" - host: 192.168.XX.XX ssh_port: 22 http_port: 8030 rpc_port: 9020 query_port: 9030 edit_log_port: 9010 deploy_dir: StarRocks/fe meta_dir: StarRocks/fe/meta log_dir: StarRocks/fe/log priority_networks: 192.168.XX.XX/24 # Specify the unique IP for current node when the machine has multiple IP addresses. config: sys_log_level: "INFO" - host: 192.168.XX.XX ssh_port: 22 http_port: 8030 rpc_port: 9020 query_port: 9030 edit_log_port: 9010 deploy_dir: StarRocks/fe meta_dir: StarRocks/fe/meta log_dir: StarRocks/fe/log priority_networks: 192.168.XX.XX/24 # Specify the unique IP for current node when the machine has multiple IP addresses. config: sys_log_level: "INFO" be_servers: - host: 192.168.XX.XX ssh_port: 22 be_port: 9060 be_http_port: 8040 heartbeat_service_port: 9050 brpc_port: 8060 deploy_dir : StarRocks/be storage_dir: StarRocks/be/storage log_dir: StarRocks/be/log priority_networks: 192.168.XX.XX/24 # Specify the unique IP for current node when the machine has multiple IP addresses. config: create_tablet_worker_count: 3 - host: 192.168.XX.XX ssh_port: 22 be_port: 9060 be_http_port: 8040 heartbeat_service_port: 9050 brpc_port: 8060 deploy_dir : StarRocks/be storage_dir: StarRocks/be/storage log_dir: StarRocks/be/log priority_networks: 192.168.XX.XX/24 # Specify the unique IP for current node when the machine has multiple IP addresses. config: create_tablet_worker_count: 3 - host: 192.168.XX.XX ssh_port: 22 be_port: 9060 be_http_port: 8040 heartbeat_service_port: 9050 brpc_port: 8060 deploy_dir : StarRocks/be storage_dir: StarRocks/be/storage log_dir: StarRocks/be/log priority_networks: 192.168.XX.XX/24 # Specify the unique IP for current node when the machine has multiple IP addresses. config: create_tablet_worker_count: 3 ``` ##### Create deployment directory (Optional)[​](#create-deployment-directory-optional "Direct link to Create deployment directory (Optional)") If the paths under which StarRocks to be deployed does not exist, and you have the privilege to create such paths, you do not have create these paths, and StarGo will create them for you based on the configuration file. If the the paths already exist, make sure you have the write access to them. You can also create necessary deployment directories on each node by running the following commands. * Create **meta** directory on FE nodes. ```shell mkdir -p StarRocks/fe/meta ``` * Create **storage** directory on BE nodes. ```shell mkdir -p StarRocks/be/storage ``` > Caution Make sure the above paths are identical with the configuration items `meta_dir` and `storage_dir` in the configuration file. ##### Deploy StarRocks[​](#deploy-starrocks "Direct link to Deploy StarRocks") Deploy StarRocks cluster by running the following command. ```shell ./sr-ctl cluster deploy ``` | Parameter | Description | | -------------- | ------------------------------- | | cluster\_name | Name of the cluster to deploy. | | version | StarRocks version. | | topology\_file | Name of the configuration file. | If the deployment is successful, the cluster will be started automatically. When beStatus and feStatus are true, the cluster is started successfully. Example: ```plain [sr-dev@r0 ~]$ ./sr-ctl cluster deploy sr-c1 v2.0.1 sr-c1.yaml [20220301-234817 OUTPUT] Deploy cluster [clusterName = sr-c1, clusterVersion = v2.0.1, metaFile = sr-c1.yaml] [20220301-234836 OUTPUT] PRE CHECK DEPLOY ENV: PreCheck FE: IP ssh auth meta dir deploy dir http port rpc port query port edit log port -------------------- --------------- ------------------------- ------------------------- --------------- --------------- --------------- --------------- 192.168.xx.xx PASS PASS PASS PASS PASS PASS PASS 192.168.xx.xx PASS PASS PASS PASS PASS PASS PASS 192.168.xx.xx PASS PASS PASS PASS PASS PASS PASS PreCheck BE: IP ssh auth storage dir deploy dir webSer port heartbeat port brpc port be port -------------------- --------------- ------------------------- ------------------------- --------------- --------------- --------------- --------------- 192.168.xx.xx PASS PASS PASS PASS PASS PASS PASS 192.168.xx.xx PASS PASS PASS PASS PASS PASS PASS 192.168.xx.xx PASS PASS PASS PASS PASS PASS PASS [20220301-234836 OUTPUT] PreCheck successfully. RESPECT [20220301-234836 OUTPUT] Create the deploy folder ... [20220301-234838 OUTPUT] Download StarRocks package & jdk ... [20220302-000515 INFO] The file starrocks-2.0.1-quickstart.tar.gz [1227406189] download successfully [20220302-000515 OUTPUT] Download done. [20220302-000515 OUTPUT] Decompress StarRocks pakcage & jdk ... [20220302-000520 INFO] The tar file /home/sr-dev/.starrocks-controller/download/starrocks-2.0.1-quickstart.tar.gz has been decompressed under /home/sr-dev/.starrocks-controller/download [20220302-000547 INFO] The tar file /home/sr-dev/.starrocks-controller/download/StarRocks-2.0.1.tar.gz has been decompressed under /home/sr-dev/.starrocks-controller/download [20220302-000556 INFO] The tar file /home/sr-dev/.starrocks-controller/download/jdk-8u301-linux-x64.tar.gz has been decompressed under /home/sr-dev/.starrocks-controller/download [20220302-000556 OUTPUT] Distribute FE Dir ... [20220302-000603 INFO] Upload dir feSourceDir = [/home/sr-dev/.starrocks-controller/download/StarRocks-2.0.1/fe] to feTargetDir = [StarRocks/fe] on FeHost = [192.168.xx.xx] [20220302-000615 INFO] Upload dir JDKSourceDir = [/home/sr-dev/.starrocks-controller/download/jdk1.8.0_301] to JDKTargetDir = [StarRocks/fe/jdk] on FeHost = [192.168.xx.xx] [20220302-000615 INFO] Modify JAVA_HOME: host = [192.168.xx.xx], filePath = [StarRocks/fe/bin/start_fe.sh] [20220302-000622 INFO] Upload dir feSourceDir = [/home/sr-dev/.starrocks-controller/download/StarRocks-2.0.1/fe] to feTargetDir = [StarRocks/fe] on FeHost = [192.168.xx.xx] [20220302-000634 INFO] Upload dir JDKSourceDir = [/home/sr-dev/.starrocks-controller/download/jdk1.8.0_301] to JDKTargetDir = [StarRocks/fe/jdk] on FeHost = [192.168.xx.xx] [20220302-000634 INFO] Modify JAVA_HOME: host = [192.168.xx.xx], filePath = [StarRocks/fe/bin/start_fe.sh] [20220302-000640 INFO] Upload dir feSourceDir = [/home/sr-dev/.starrocks-controller/download/StarRocks-2.0.1/fe] to feTargetDir = [StarRocks/fe] on FeHost = [192.168.xx.xx] [20220302-000652 INFO] Upload dir JDKSourceDir = [/home/sr-dev/.starrocks-controller/download/jdk1.8.0_301] to JDKTargetDir = [StarRocks/fe/jdk] on FeHost = [192.168.xx.xx] [20220302-000652 INFO] Modify JAVA_HOME: host = [192.168.xx.xx], filePath = [StarRocks/fe/bin/start_fe.sh] [20220302-000652 OUTPUT] Distribute BE Dir ... [20220302-000728 INFO] Upload dir BeSourceDir = [/home/sr-dev/.starrocks-controller/download/StarRocks-2.0.1/be] to BeTargetDir = [StarRocks/be] on BeHost = [192.168.xx.xx] [20220302-000752 INFO] Upload dir BeSourceDir = [/home/sr-dev/.starrocks-controller/download/StarRocks-2.0.1/be] to BeTargetDir = [StarRocks/be] on BeHost = [192.168.xx.xx] [20220302-000815 INFO] Upload dir BeSourceDir = [/home/sr-dev/.starrocks-controller/download/StarRocks-2.0.1/be] to BeTargetDir = [StarRocks/be] on BeHost = [192.168.xx.xx] [20220302-000815 OUTPUT] Modify configuration for FE nodes & BE nodes ... ############################################# START FE CLUSTER ############################################# ############################################# START FE CLUSTER ############################################# [20220302-000816 INFO] Starting leader FE node [host = 192.168.xx.xx, editLogPort = 9010] [20220302-000836 INFO] The FE node start succefully [host = 192.168.xx.xx, queryPort = 9030] [20220302-000836 INFO] Starting follower FE node [host = 192.168.xx.xx, editLogPort = 9010] [20220302-000857 INFO] The FE node start succefully [host = 192.168.xx.xx, queryPort = 9030] [20220302-000857 INFO] Starting follower FE node [host = 192.168.xx.xx, editLogPort = 9010] [20220302-000918 INFO] The FE node start succefully [host = 192.168.xx.xx, queryPort = 9030] [20220302-000918 INFO] List all FE status: feHost = 192.168.xx.xx feQueryPort = 9030 feStatus = true feHost = 192.168.xx.xx feQueryPort = 9030 feStatus = true feHost = 192.168.xx.xx feQueryPort = 9030 feStatus = true ############################################# START BE CLUSTER ############################################# ############################################# START BE CLUSTER ############################################# [20220302-000918 INFO] Starting BE node [BeHost = 192.168.xx.xx HeartbeatServicePort = 9050] [20220302-000939 INFO] The BE node start succefully [host = 192.168.xx.xx, heartbeatServicePort = 9050] [20220302-000939 INFO] Starting BE node [BeHost = 192.168.xx.xx HeartbeatServicePort = 9050] [20220302-001000 INFO] The BE node start succefully [host = 192.168.xx.xx, heartbeatServicePort = 9050] [20220302-001000 INFO] Starting BE node [BeHost = 192.168.xx.xx HeartbeatServicePort = 9050] [20220302-001020 INFO] The BE node start succefully [host = 192.168.xx.xx, heartbeatServicePort = 9050] [20220302-001020 OUTPUT] List all BE status: beHost = 192.168.xx.xx beHeartbeatServicePort = 9050 beStatus = true beHost = 192.168.xx.xx beHeartbeatServicePort = 9050 beStatus = true beHost = 192.168.xx.xx beHeartbeatServicePort = 9050 beStatus = true ``` You can test the cluster by [viewing cluster information](#view-cluster-information). You can also test it by connecting the cluster with MySQL client. ```shell mysql -h 127.0.0.1 -P9030 -uroot ``` #### View cluster information[​](#view-cluster-information "Direct link to View cluster information") You can view the information of the cluster that StarGo manages. ##### View the information of all clusters[​](#view-the-information-of-all-clusters "Direct link to View the information of all clusters") View the information of all clusters by running the following command. ```shell ./sr-ctl cluster list ``` Example: ```shell [sr-dev@r0 ~]$ ./sr-ctl cluster list [20220302-001640 OUTPUT] List all clusters ClusterName User CreateDate MetaPath PrivateKey --------------- ---------- ------------------------- ------------------------------------------------------------ -------------------------------------------------- sr-c1 starrocks 2022-03-02 00:08:15 /home/sr-dev/.starrocks-controller/cluster/sr-c1 /home/sr-dev/.ssh/id_rsa ``` ##### View the information of a specific cluster[​](#view-the-information-of-a-specific-cluster "Direct link to View the information of a specific cluster") View the information of a specific cluster by running the following command. ```shell ./sr-ctl cluster display ``` Example: ```plain [sr-dev@r0 ~]$ ./sr-ctl cluster display sr-c1 [20220302-002310 OUTPUT] Display cluster [clusterName = sr-c1] clusterName = sr-c1 ID ROLE HOST PORT STAT DATADIR DEPLOYDIR -------------------------- ------ -------------------- --------------- ---------- -------------------------------------------------- -------------------------------------------------- 192.168.xx.xx:9010 FE 192.168.xx.xx 9010/9030 UP StarRocks/fe /dataStarRocks/fe/meta 192.168.xx.xx:9010 FE 192.168.xx.xx 9010/9030 UP StarRocks/fe /dataStarRocks/fe/meta 192.168.xx.xx:9010 FE 192.168.xx.xx 9010/9030 UP StarRocks/fe /dataStarRocks/fe/meta 192.168.xx.xx:9060 BE 192.168.xx.xx 9060/9050 UP StarRocks/be /dataStarRocks/be/storage 192.168.xx.xx:9060 BE 192.168.xx.xx 9060/9050 UP StarRocks/be /dataStarRocks/be/storage 192.168.xx.xx:9060 BE 192.168.xx.xx 9060/9050 UP StarRocks/be /dataStarRocks/be/storage ``` #### Start cluster[​](#start-cluster "Direct link to Start cluster") You can start StarRocks clusters via StarGo. ##### Start all nodes in a cluster[​](#start-all-nodes-in-a-cluster "Direct link to Start all nodes in a cluster") Start all nodes in a cluster by running the following command. ```shell ./sr-ctl cluster start ``` Example: ```plain [root@nd1 sr-controller]# ./sr-ctl cluster start sr-c1 [20220303-190404 OUTPUT] Start cluster [clusterName = sr-c1] [20220303-190404 INFO] Starting FE node [FeHost = 192.168.xx.xx, EditLogPort = 9010] [20220303-190435 INFO] Starting FE node [FeHost = 192.168.xx.xx, EditLogPort = 9010] [20220303-190446 INFO] Starting FE node [FeHost = 192.168.xx.xx, EditLogPort = 9010] [20220303-190457 INFO] Starting BE node [BeHost = 192.168.xx.xx, HeartbeatServicePort = 9050] [20220303-190458 INFO] Starting BE node [BeHost = 192.168.xx.xx, HeartbeatServicePort = 9050] [20220303-190458 INFO] Starting BE node [BeHost = 192.168.xx.xx, HeartbeatServicePort = 9050] ``` ##### Start nodes of a specific role[​](#start-nodes-of-a-specific-role "Direct link to Start nodes of a specific role") * Start all FE nodes in a cluster. ```shell ./sr-ctl cluster start --role FE ``` * Start all BE nodes in a cluster. ```shell ./sr-ctl cluster start --role BE ``` Example: ```plain [root@nd1 sr-controller]# ./sr-ctl cluster start sr-c1 --role FE [20220303-191529 OUTPUT] Start cluster [clusterName = sr-c1] [20220303-191529 INFO] Starting FE cluster .... [20220303-191529 INFO] Starting FE node [FeHost = 192.168.xx.xx, EditLogPort = 9010] [20220303-191600 INFO] Starting FE node [FeHost = 192.168.xx.xx, EditLogPort = 9010] [20220303-191610 INFO] Starting FE node [FeHost = 192.168.xx.xx, EditLogPort = 9010] [root@nd1 sr-controller]# ./sr-ctl cluster start sr-c1 --role BE [20220303-194215 OUTPUT] Start cluster [clusterName = sr-c1] [20220303-194215 INFO] Starting BE node [BeHost = 192.168.xx.xx, HeartbeatServicePort = 9050] [20220303-194216 INFO] Starting BE node [BeHost = 192.168.xx.xx, HeartbeatServicePort = 9050] [20220303-194217 INFO] Starting BE node [BeHost = 192.168.xx.xx, HeartbeatServicePort = 9050] [20220303-194217 INFO] Starting BE cluster ... ``` ##### Start a specific node[​](#start-a-specific-node "Direct link to Start a specific node") Start a specific node in the cluster. Currently, only BE nodes are supported. ```shell ./sr-ctl cluster start --node ``` You can check the ID of a specific node by [viewing the information of a specific cluster](#view-the-information-of-a-specific-cluster). Example: ```plain [root@nd1 sr-controller]# ./sr-ctl cluster start sr-c1 --node 192.168.xx.xx:9060 [20220303-194714 OUTPUT] Start cluster [clusterName = sr-c1] [20220303-194714 INFO] Start BE node. [BeHost = 192.168.xx.xx, HeartbeatServicePort = 9050] ``` #### Stop cluster[​](#stop-cluster "Direct link to Stop cluster") You can stop StarRocks clusters via StarGo. ##### Stop all nodes in a cluster[​](#stop-all-nodes-in-a-cluster "Direct link to Stop all nodes in a cluster") Stop all nodes in a cluster by running the following command. ```shell ./sr-ctl cluster stop ``` Example: ```plain [sr-dev@nd1 sr-controller]$ ./sr-ctl cluster stop sr-c1 [20220302-180140 OUTPUT] Stop cluster [clusterName = sr-c1] [20220302-180140 OUTPUT] Stop cluster sr-c1 [20220302-180140 INFO] Waiting for stoping FE node [FeHost = 192.168.xx.xx] [20220302-180143 OUTPUT] The FE node stop succefully [host = 192.168.xx.xx, queryPort = 9030] [20220302-180143 INFO] Waiting for stoping FE node [FeHost = 192.168.xx.xx] [20220302-180145 OUTPUT] The FE node stop succefully [host = 192.168.xx.xx, queryPort = 9030] [20220302-180145 INFO] Waiting for stoping FE node [FeHost = 192.168.xx.xx] [20220302-180148 OUTPUT] The FE node stop succefully [host = 192.168.xx.xx, queryPort = 9030] [20220302-180148 OUTPUT] Stop cluster sr-c1 [20220302-180148 INFO] Waiting for stoping BE node [BeHost = 192.168.xx.xx] [20220302-180148 INFO] The BE node stop succefully [BeHost = 192.168.xx.xx, HeartbeatServicePort = 9050] [20220302-180148 INFO] Waiting for stoping BE node [BeHost = 192.168.xx.xx] [20220302-180149 INFO] The BE node stop succefully [BeHost = 192.168.xx.xx, HeartbeatServicePort = 9050] [20220302-180149 INFO] Waiting for stoping BE node [BeHost = 192.168.xx.xx] [20220302-180149 INFO] The BE node stop succefully [BeHost = 192.168.xx.xx, HeartbeatServicePort = 9050] ``` ##### Stop nodes of a specific role[​](#stop-nodes-of-a-specific-role "Direct link to Stop nodes of a specific role") * Stop all FE nodes in a cluster. ```shell ./sr-ctl cluster stop --role FE ``` * Stop all BE nodes in a cluster. ```shell ./sr-ctl cluster stop --role BE ``` Example: ```plain [sr-dev@nd1 sr-controller]$ ./sr-ctl cluster stop sr-c1 --role BE [20220302-180624 OUTPUT] Stop cluster [clusterName = sr-c1] [20220302-180624 OUTPUT] Stop cluster sr-c1 [20220302-180624 INFO] Waiting for stoping BE node [BeHost = 192.168.xx.xx] [20220302-180624 INFO] The BE node stop succefully [BeHost = 192.168.xx.xx, HeartbeatServicePort = 9050] [20220302-180624 INFO] Waiting for stoping BE node [BeHost = 192.168.xx.xx] [20220302-180625 INFO] The BE node stop succefully [BeHost = 192.168.xx.xx, HeartbeatServicePort = 9050] [20220302-180625 INFO] Waiting for stoping BE node [BeHost = 192.168.xx.xx] [20220302-180625 INFO] The BE node stop succefully [BeHost = 192.168.xx.xx, HeartbeatServicePort = 9050] [20220302-180625 INFO] Stopping BE cluster ... ########################################################################### [sr-dev@nd1 sr-controller]$ ./sr-ctl cluster stop sr-c1 --role FE [20220302-180849 OUTPUT] Stop cluster [clusterName = sr-c1] [20220302-180849 INFO] Stopping FE cluster .... [20220302-180849 OUTPUT] Stop cluster sr-c1 [20220302-180849 INFO] Waiting for stoping FE node [FeHost = 192.168.xx.xx] [20220302-180851 OUTPUT] The FE node stop succefully [host = 192.168.xx.xx, queryPort = 9030] [20220302-180851 INFO] Waiting for stoping FE node [FeHost = 192.168.xx.xx] [20220302-180854 OUTPUT] The FE node stop succefully [host = 192.168.xx.xx, queryPort = 9030] [20220302-180854 INFO] Waiting for stoping FE node [FeHost = 192.168.xx.xx] [20220302-180856 OUTPUT] The FE node stop succefully [host = 192.168.xx.xx, queryPort = 9030] ``` ##### Stop a specific node[​](#stop-a-specific-node "Direct link to Stop a specific node") Stop a specific node in the cluster. ```shell ./sr-ctl cluster stop --node ``` You can check the ID of a specific node by [viewing the information of a specific cluster](#view-the-information-of-a-specific-cluster). Example: ```plain [root@nd1 sr-controller]# ./sr-ctl cluster display sr-c1 [20220303-185400 OUTPUT] Display cluster [clusterName = sr-c1] clusterName = sr-c1 [20220303-185400 WARN] All FE nodes are down, please start FE node and display the cluster status again. ID ROLE HOST PORT STAT DATADIR DEPLOYDIR -------------------------- ------ -------------------- --------------- ---------- -------------------------------------------------- -------------------------------------------------- 192.168.xx.xx:9010 FE 192.168.xx.xx 9010/9030 DOWN StarRocks/fe /dataStarRocks/fe/meta 192.168.xx.xx:9010 FE 192.168.xx.xx 9010/9030 DOWN StarRocks/fe /dataStarRocks/fe/meta 192.168.xx.xx:9010 FE 192.168.xx.xx 9010/9030 DOWN StarRocks/fe /dataStarRocks/fe/meta 192.168.xx.xx:9060 BE 192.168.xx.xx 9060/9050 DOWN StarRocks/be /dataStarRocks/be/storage 192.168.xx.xx:9060 BE 192.168.xx.xx 9060/9050 DOWN StarRocks/be /dataStarRocks/be/storage 192.168.xx.xx:9060 BE 192.168.xx.xx 9060/9050 DOWN StarRocks/be /dataStarRocks/be/storage [root@nd1 sr-controller]# ./sr-ctl cluster stop sr-c1 --node 192.168.xx.xx:9060 [20220303-185510 OUTPUT] Stop cluster [clusterName = sr-c1] [20220303-185510 INFO] Stopping BE node. [BeHost = 192.168.xx.xx] [20220303-185510 INFO] Waiting for stoping BE node [BeHost = 192.168.xx.xx] ``` #### Scale cluster out[​](#scale-cluster-out "Direct link to Scale cluster out") You can scale a cluster out via StarGo. ##### Create configuration file[​](#create-configuration-file-1 "Direct link to Create configuration file") Create the scale-out task topology file based on the following template. You can specify the file to add FE and/or BE nodes based on your demand. See [Configuration](https://docs.starrocks.io/docs/administration/management/FE_configuration.md) for detailed information. ```yaml # Add an FE node. fe_servers: - host: 192.168.xx.xx # The IP address of the new FE node. ssh_port: 22 http_port: 8030 rpc_port: 9020 query_port: 9030 edit_log_port: 9010 deploy_dir: StarRocks/fe meta_dir: StarRocks/fe/meta log_dir: StarRocks/fe/log priority_networks: 192.168.xx.xx/24 # Specify the unique IP for current node when the machine has multiple IP addresses. config: sys_log_level: "INFO" sys_log_delete_age: "1d" # Add a BE node. be_servers: - host: 192.168.xx.xx # The IP address of the new BE node. ssh_port: 22 be_port: 9060 be_http_port: 8040 heartbeat_service_port: 9050 brpc_port: 8060 deploy_dir : StarRocks/be storage_dir: StarRocks/be/storage log_dir: StarRocks/be/log config: create_tablet_worker_count: 3 ``` ##### Build SSH mutual authentication[​](#build-ssh-mutual-authentication "Direct link to Build SSH mutual authentication") If you are adding a new node to the cluster, you must build mutual authentication between the new node and the central control node. See [Prerequisites](#prerequisites) for detailed instruction. ##### Create deployment directory (Optional)[​](#create-deployment-directory-optional-1 "Direct link to Create deployment directory (Optional)") If the path under which the new node to be deployed does not exist, and you have the privilege to create such path, you do not have create these path, and StarGo will create them for you based on the configuration file. If the the paths already exist, make sure you have the write access to them. You can also create necessary deployment directories on each node by running the following commands. * Create **meta** directory on FE nodes. ```shell mkdir -p StarRocks/fe/meta ``` * Create **storage** directory on BE nodes. ```shell mkdir -p StarRocks/be/storage ``` > Caution Make sure the above paths are identical with the configuration items `meta_dir` and `storage_dir` in the configuration file. ##### Scale the cluster out[​](#scale-the-cluster-out "Direct link to Scale the cluster out") Scale the cluster out by running the following command. ```shell ./sr-ctl cluster scale-out ``` Example: ```plain # Status of the cluster before scale-out. [root@nd1 sr-controller]# ./sr-ctl cluster display sr-test [20220503-210047 OUTPUT] Display cluster [clusterName = sr-test] clusterName = sr-test clusterVerison = v2.0.1 ID ROLE HOST PORT STAT DATADIR DEPLOYDIR -------------------------- ------ -------------------- --------------- ---------- -------------------------------------------------- -------------------------------------------------- 192.168.xx.xx:9010 FE 192.168.xx.xx 9010/9030 UP /opt/starrocks-test/fe /opt/starrocks-test/fe/meta 192.168.xx.xx:9060 BE 192.168.xx.xx 9060/9050 UP /opt/starrocks-test/be /opt/starrocks-test/be/storage # Scale the cluster out. [sr-dev@nd1 sr-controller]$ ./sr-ctl cluster scale-out sr-test sr-out.yaml [20220503-213725 OUTPUT] Scale out cluster. [ClusterName = sr-test] [20220503-213731 OUTPUT] PRE CHECK DEPLOY ENV: PreCheck FE: IP ssh auth meta dir deploy dir http port rpc port query port edit log port -------------------- --------------- ------------------------------ ------------------------------ --------------- --------------- --------------- --------------- 192.168.xx.xx PASS PASS PASS PASS PASS PASS PASS PreCheck BE: IP ssh auth storage dir deploy dir webSer port heartbeat port brpc port be port -------------------- --------------- ------------------------------ ------------------------------ --------------- --------------- --------------- --------------- 192.168.xx.xx PASS PASS PASS PASS PASS PASS PASS [20220503-213731 OUTPUT] PreCheck successfully. RESPECT [20220503-213731 OUTPUT] Create the deploy folder ... [20220503-213732 OUTPUT] Download StarRocks package & jdk ... [20220503-213732 INFO] The package has already exist [fileName = starrocks-2.0.1-quickstart.tar.gz, fileSize = 1227406189, fileModTime = 2022-05-03 17:32:03.478661923 +0800 CST] [20220503-213732 OUTPUT] Download done. [20220503-213732 OUTPUT] Decompress StarRocks pakage & jdk ... [20220503-213741 INFO] The tar file /home/sr-dev/.starrocks-controller/download/starrocks-2.0.1-quickstart.tar.gz has been decompressed under /home/sr-dev/.starrocks-controller/download [20220503-213837 INFO] The tar file /home/sr-dev/.starrocks-controller/download/StarRocks-2.0.1.tar.gz has been decompressed under /home/sr-dev/.starrocks-controller/download [20220503-213837 INFO] The tar file /home/sr-dev/.starrocks-controller/download/jdk-8u301-linux-x64.tar.gz has been decompressed under /home/sr-dev/.starrocks-controller/download [20220503-213837 OUTPUT] Distribute FE Dir ... [20220503-213845 INFO] Upload dir feSourceDir = [/home/sr-dev/.starrocks-controller/download/StarRocks-2.0.1/fe] to feTargetDir = [StarRocks/fe] on FeHost = [192.168.xx.xx] [20220503-213857 INFO] Upload dir JDKSourceDir = [/home/sr-dev/.starrocks-controller/download/jdk1.8.0_301] to JDKTargetDir = [StarRocks/fe/jdk] on FeHost = [192.168.xx.xx] [20220503-213857 INFO] Modify JAVA_HOME: host = [192.168.xx.xx], filePath = [StarRocks/fe/bin/start_fe.sh] [20220503-213857 OUTPUT] Distribute BE Dir ... [20220503-213924 INFO] Upload dir BeSourceDir = [/home/sr-dev/.starrocks-controller/download/StarRocks-2.0.1/be] to BeTargetDir = [StarRocks/be] on BeHost = [192.168.xx.xx] [20220503-213924 OUTPUT] Modify configuration for FE nodes & BE nodes ... ############################################# SCALE OUT FE CLUSTER ############################################# ############################################# SCALE OUT FE CLUSTER ############################################# [20220503-213925 INFO] Starting follower FE node [host = 192.168.xx.xx, editLogPort = 9010] [20220503-213945 INFO] The FE node start succefully [host = 192.168.xx.xx, queryPort = 9030] [20220503-213945 INFO] List all FE status: feHost = 192.168.xx.xx feQueryPort = 9030 feStatus = true ############################################# START BE CLUSTER ############################################# ############################################# START BE CLUSTER ############################################# [20220503-213945 INFO] Starting BE node [BeHost = 192.168.xx.xx HeartbeatServicePort = 9050] [20220503-214016 INFO] The BE node start succefully [host = 192.168.xx.xx, heartbeatServicePort = 9050] [20220503-214016 OUTPUT] List all BE status: beHost = 192.168.xx.xx beHeartbeatServicePort = 9050 beStatus = true # Status of the cluster after scale-out. [sr-dev@nd1 sr-controller]$ ./sr-ctl cluster display sr-test [20220503-214302 OUTPUT] Display cluster [clusterName = sr-test] clusterName = sr-test clusterVerison = v2.0.1 ID ROLE HOST PORT STAT DATADIR DEPLOYDIR -------------------------- ------ -------------------- --------------- ---------- -------------------------------------------------- -------------------------------------------------- 192.168.xx.xx:9010 FE 192.168.xx.xx 9010/9030 UP /opt/starrocks-test/fe /opt/starrocks-test/fe/meta 192.168.xx.xx:9010 FE 192.168.xx.xx 9010/9030 UP StarRocks/fe StarRocks/fe/meta 192.168.xx.xx:9060 BE 192.168.xx.xx 9060/9050 UP /opt/starrocks-test/be /opt/starrocks-test/be/storage 192.168.xx.xx:9060 BE 192.168.xx.xx 9060/9050 UP StarRocks/be StarRocks/be/storage ``` #### Scale cluster in[​](#scale-cluster-in "Direct link to Scale cluster in") Remove a node in the cluster by running the following command. ```shell ./sr-ctl cluster scale-in --node ``` You can check the ID of a specific node by [viewing the information of a specific cluster](#view-the-information-of-a-specific-cluster). Example: ```plain [sr-dev@nd1 sr-controller]$ ./sr-ctl cluster display sr-c1 [20220505-145649 OUTPUT] Display cluster [clusterName = sr-c1] clusterName = sr-c1 clusterVerison = v2.0.1 ID ROLE HOST PORT STAT DATADIR DEPLOYDIR -------------------------- ------ -------------------- --------------- ---------- -------------------------------------------------- -------------------------------------------------- 192.168.xx.xx:9010 FE 192.168.xx.xx 9010/9030 UP StarRocks/fe /dataStarRocks/fe/meta 192.168.xx.xx:9010 FE 192.168.xx.xx 9010/9030 UP StarRocks/fe /dataStarRocks/fe/meta 192.168.xx.xx:9010 FE 192.168.xx.xx 9010/9030 UP StarRocks/fe /dataStarRocks/fe/meta 192.168.xx.xx:9060 BE 192.168.xx.xx 9060/9050 UP StarRocks/be /dataStarRocks/be/storage 192.168.xx.xx:9060 BE 192.168.xx.xx 9060/9050 UP StarRocks/be /dataStarRocks/be/storage 192.168.xx.xx:9060 BE 192.168.xx.xx 9060/9050 UP StarRocks/be /dataStarRocks/be/storage [sr-dev@nd1 sr-controller]$ ./sr-ctl cluster scale-in sr-c1 --node 192.168.88.83:9010 [20220621-010553 OUTPUT] Scale in cluster [clusterName = sr-c1, nodeId = 192.168.88.83:9010] [20220621-010553 INFO] Waiting for stoping FE node [FeHost = 192.168.88.83] [20220621-010606 OUTPUT] Scale in FE node successfully. [clusterName = sr-c1, nodeId = 192.168.88.83:9010] [sr-dev@nd1 sr-controller]$ ./sr-ctl cluster display sr-c1 [20220621-010623 OUTPUT] Display cluster [clusterName = sr-c1] clusterName = sr-c1 clusterVerison = ID ROLE HOST PORT STAT DATADIR DEPLOYDIR -------------------------- ------ -------------------- --------------- ---------- -------------------------------------------------- -------------------------------------------------- 192.168.88.84:9010 FE 192.168.xx.xx 9010/9030 UP StarRocks/fe /dataStarRocks/fe/meta 192.168.88.85:9010 FE 192.168.xx.xx 9010/9030 UP/L StarRocks/fe /dataStarRocks/fe/meta 192.168.88.83:9060 BE 192.168.xx.xx 9060/9050 UP StarRocks/be /dataStarRocks/be/storage 192.168.88.84:9060 BE 192.168.xx.xx 9060/9050 UP StarRocks/be /dataStarRocks/be/storage 192.168.88.85:9060 BE 192.168.xx.xx 9060/9050 UP StarRocks/be /dataStarRocks/be/storage ``` #### Upgrade or downgrade the cluster[​](#upgrade-or-downgrade-the-cluster "Direct link to Upgrade or downgrade the cluster") You can upgrade or downgrade a cluster via StarGo. * Upgrade a cluster. ```shell ./sr-ctl cluster upgrade ``` * Downgrade a cluster. ```shell ./sr-ctl cluster downgrade ``` Example: ```plain [sr-dev@nd1 sr-controller]$ ./sr-ctl cluster list [20220515-195827 OUTPUT] List all clusters ClusterName Version User CreateDate MetaPath PrivateKey --------------- ---------- ---------- ------------------------- ------------------------------------------------------------ -------------------------------------------------- sr-test2 v2.0.1 test222 2022-05-15 19:35:36 /home/sr-dev/.starrocks-controller/cluster/sr-test2 /home/sr-dev/.ssh/id_rsa [sr-dev@nd1 sr-controller]$ ./sr-ctl cluster upgrade sr-test2 v2.1.3 [20220515-200358 OUTPUT] List all clusters ClusterName Version User CreateDate MetaPath PrivateKey --------------- ---------- ---------- ------------------------- ------------------------------------------------------------ -------------------------------------------------- sr-test2 v2.1.3 test222 2022-05-15 20:03:01 /home/sr-dev/.starrocks-controller/cluster/sr-test2 /home/sr-dev/.ssh/id_rsa [sr-dev@nd1 sr-controller]$ ./sr-ctl cluster downgrade sr-test2 v2.0.1 [sr-dev@nd1 sr-controller]$ ./sr-ctl cluster list [20220515-200915 OUTPUT] List all clusters ClusterName Version User CreateDate MetaPath PrivateKey --------------- ---------- ---------- ------------------------- ------------------------------------------------------------ -------------------------------------------------- sr-test2 v2.0.1 test222 2022-05-15 20:08:40 /home/sr-dev/.starrocks-controller/cluster/sr-test2 /home/sr-dev/.ssh/id_rsa ``` #### Relevant commands[​](#relevant-commands "Direct link to Relevant commands") | Command | Description | | --------- | ------------------------------------------- | | deploy | Deploy a cluster. | | start | Start a cluster. | | stop | Stop a cluster. | | scale-in | Scale in a cluster. | | scale-out | Scale out a cluster. | | upgrade | Upgrade a cluster. | | downgrade | Downgrade a cluster | | display | View the information of a specific cluater. | | list | View all clusters. | --- ### JSON Web Token Authentication This topic describes how to enable JSON Web Token authentication in StarRocks. From v3.5.0 onwards, StarRocks supports authenticating client access using JSON Web Tokens. JSON Web Token (JWT) is an open standard (RFC 7519) that defines a compact and self-contained way for securely transmitting information between parties as a JSON object. This information can be verified and trusted because it is digitally signed. JWTs can be signed using a secret (with the HMAC algorithm) or a public/private key pair using RSA or ECDSA. This topic describes how to manually create and authenticate users using JWT in StarRocks. For instructions on how to integrate StarRocks with JWT authentication using security integration, see [Authenticate with Security Integration](https://docs.starrocks.io/docs/administration/user_privs/authentication/security_integration.md). For more information on how to authenticate user groups using JWT, see [Authenticate User Groups](https://docs.starrocks.io/docs/administration/user_privs/group_provider.md). #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") If you want to connect to StarRocks from a MySQL client, the MySQL client version must be 9.2 or later. #### Create a user with JWT[​](#create-a-user-with-jwt "Direct link to Create a user with JWT") When creating a user, specify the authentication method as JWT by `IDENTIFIED WITH authentication_jwt [AS '{xxx}']`. `{xxx}` is the JWT properties of the user. In addition to the following method, you can configure the default JWT properties in the FE configuration file. You need to manually modify all **fe.conf** files and restart all FEs for configuration to take effect. After the FE configurations are set, StarRocks will use the default properties specified in your configuration file and you can omit the `AS '{xxx}'` part. Syntax: ```sql CREATE USER IDENTIFIED WITH authentication_jwt [AS '{ "jwks_url": "", "principal_field": "", "required_issuer": "", "required_audience": "" }'] ``` | Property | Corresponding FE Configuration | Description | | ------------------- | ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `jwks_url` | `jwt_jwks_url` | The URL to the JSON Web Key Set (JWKS) service or the path to the public key local file under the `fe/conf` directory. | | `principal_field` | `jwt_principal_field` | The string used to identify the field that indicates the subject (`sub`) in the JWT. The default value is `sub`. The value of this field must be identical with the username for logging in to StarRocks. | | `required_issuer` | `jwt_required_issuer` | (Optional) The list of strings used to identify the issuers (`iss`) in the JWT. The JWT is considered valid only if one of the values in the list match the JWT issuer. | | `required_audience` | `jwt_required_audience` | (Optional) The list of strings used to identify the audience (`aud`) in the JWT. The JWT is considered valid only if one of the values in the list match the JWT audience. | Example: ```sql CREATE USER tom IDENTIFIED WITH authentication_jwt AS '{ "jwks_url": "http://localhost:38080/realms/master/protocol/jwt/certs", "principal_field": "preferred_username", "required_issuer": "http://localhost:38080/realms/master", "required_audience": "starrocks" }'; ``` If you have set the JWT properties in the FE configuration files, you can directly execute the following statement: ```sql CREATE USER tom IDENTIFIED WITH authentication_jwt; ``` #### Connect from MySQL client with JWT[​](#connect-from-mysql-client-with-jwt "Direct link to Connect from MySQL client with JWT") To connect from a MySQL client to StarRocks using JWT, you need to enable the `authentication_openid-connect_client` plugin, and pass the necessary token (using the path to the token file) to authenticate the mapped user. Syntax: ```bash mysql -h -P --authentication-openid-connect-client-id-token-file= -u ``` Example: ```bash mysql -h 127.0.0.1 -P 9030 --authentication-openid-connect-client-id-token-file=/path/to/token/file -u tom ``` --- ### LDAP Authentication In addition to native password-based authentication, StarRocks also supports the LDAP authentication. This topic describes how to manually create and authenticate users using LDAP in StarRocks. For instructions on how to integrate StarRocks with your LDAP service using security integration, see [Authenticate with Security Integration](https://docs.starrocks.io/docs/administration/user_privs/authentication/security_integration.md). For more information on how to authenticate user groups in your LDAP service, see [Authenticate User Groups](https://docs.starrocks.io/docs/administration/user_privs/group_provider.md). #### Enable LDAD authentication[​](#enable-ldad-authentication "Direct link to Enable LDAD authentication") To use LDAP authentication, you need to add the LDAP service into the FE node configuration first. ```properties # Add the LDAP service IP address. authentication_ldap_simple_server_host = # Add the LDAP service port, with a default value of 389. authentication_ldap_simple_server_port = # Whether to allow non-encrypted connections to the LDAP server. Default value: `true`. Setting this value to `false` indicates that SSL encryption is required to access LDAP. authentication_ldap_simple_ssl_conn_allow_insecure = # Local path to store the SSL CA certificate of the LDAP server. Supports pem and jks formats. You do not need to set this item if the certificate is issued by a trusted organization. authentication_ldap_simple_ssl_conn_trust_store_path = # The password used to access the locally stored SSL CA certificate of the LDAP server. pem-formatted certificates do not require a password. Only jsk-formatted certificates do. authentication_ldap_simple_ssl_conn_trust_store_pwd = ``` If you wish to authenticate users by means of StarRocks retrieving them directly in the LDAP system (search-and-bind mode), you will need to **add the following additional configuration items**. ```properties # Add the Base DN of the user, specifying the user's retrieval range. authentication_ldap_simple_bind_base_dn = # Add the name of the attribute that identifies the user in the LDAP object. Default: uid. authentication_ldap_simple_user_search_attr = # Add the admin DN for retrieving users. authentication_ldap_simple_bind_root_dn = # Add the admin password for retrieving users. authentication_ldap_simple_bind_root_pwd = ``` If you wish to use **direct bind mode** (skip the search step and bind directly with a constructed DN), you can configure a DN pattern instead. This is useful when the user DN structure is predictable. ```properties # The DN pattern for direct bind authentication. # Use ${USER} as a placeholder for the username. # Multiple patterns can be separated by semicolon ';'. authentication_ldap_simple_bind_dn_pattern = ``` For example: `uid=${USER},ou=People,dc=example,dc=com` If you have users across multiple OUs, you can specify multiple patterns separated by semicolons: `uid=${USER},ou=Engineering,dc=example,dc=com;uid=${USER},ou=Marketing,dc=example,dc=com` The system will try each pattern in order and return the first successful bind. note The pattern must produce a valid LDAP Distinguished Name (DN). UPN-style patterns like `${USER}@corp.example.com` are not supported, because the result is not a DN and would break downstream group lookups. If your DN contains `@` in an attribute value (e.g., `uid=${USER}@corp.example.com,ou=People,dc=example,dc=com`), that is valid. #### DN Matching Mechanism[​](#dn-matching-mechanism "Direct link to DN Matching Mechanism") Starting from v3.5.0, StarRocks supports recording and passing user Distinguished Name (DN) information during LDAP authentication to provide more accurate group resolution. ##### How it Works[​](#how-it-works "Direct link to How it Works") 1. **Authentication Phase**: LDAPAuthProvider records both pieces of information after successful user authentication: * Login username (for traditional group matching) * User's complete DN (for DN-based group matching) 2. **Group Resolution Phase**: LDAPGroupProvider determines the matching strategy based on the `ldap_user_search_attr` parameter configuration: * **When `ldap_user_search_attr` is configured**, it uses username as the key for group matching. * **When `ldap_user_search_attr` is not configured**, it uses DN as the key for group matching. ##### Use Cases[​](#use-cases "Direct link to Use Cases") * **Traditional LDAP Environment**: Group members use simple usernames (such as `cn` attribute). Administrators need to configure `ldap_user_search_attr`. * **Microsoft AD Environment**: Group members may lack username attributes. `ldap_user_search_attr` cannot be configured. The system will use DN directly for matching. * **Mixed Environment**: Flexible switching between both matching methods is supported. #### Authentication priority[​](#authentication-priority "Direct link to Authentication priority") When a user logs in with LDAP authentication, StarRocks determines the user's DN using the following priority: 1. **Per-user DN**: If the user was created with an explicit DN (`CREATE USER ... AS 'dn'`), that DN is used directly. 2. **Direct bind via DN pattern**: If `authentication_ldap_simple_bind_dn_pattern` is configured, the system constructs the DN from the pattern and attempts to bind directly. Multiple patterns are tried in order. 3. **Search-and-bind**: If neither of the above applies, the system uses the admin account to search for the user in LDAP, then binds with the found DN. #### Create a user with LDAP[​](#create-a-user-with-ldap "Direct link to Create a user with LDAP") When creating a user, specify the authentication method as LDAP authentication by `IDENTIFIED WITH authentication_ldap_simple AS 'xxx'`. xxx is the DN (Distinguished Name) of the user in LDAP. Example 1: Create a user with an explicit DN. ```sql CREATE USER tom IDENTIFIED WITH authentication_ldap_simple AS 'uid=tom,ou=company,dc=example,dc=com' ``` Example 2: Create a user without specifying the DN. The system will resolve the DN at login time using either the DN pattern (direct bind) or search-and-bind, depending on the configuration. ```sql CREATE USER tom IDENTIFIED WITH authentication_ldap_simple ``` If using **search-and-bind** mode, the following additional FE configuration is needed: * `authentication_ldap_simple_bind_base_dn`: The base DN of the user, specifying the retrieval range of the user. * `authentication_ldap_simple_user_search_attr`: The name of the attribute in the LDAP object that identifies the user, uid by default. * `authentication_ldap_simple_bind_root_dn`: The DN of the administrator account used to retrieve the user information. * `authentication_ldap_simple_bind_root_pwd`: The password of the administrator account used when retrieving the user information. If using **direct bind** mode, configure `authentication_ldap_simple_bind_dn_pattern` instead. This does not require an admin account. #### Authenticate users[​](#authenticate-users "Direct link to Authenticate users") LDAP authentication requires the client to pass on a clear-text password to StarRocks. There are three ways to pass on a clear-text password: ##### Connect from MySQL client with LDAP[​](#connect-from-mysql-client-with-ldap "Direct link to Connect from MySQL client with LDAP") Add `--default-auth mysql_clear_password --enable-cleartext-plugin` when executing: ```sql mysql -utom -P9030 -h127.0.0.1 -p --default-auth mysql_clear_password --enable-cleartext-plugin ``` ##### Connect from JDBC/ODBC client with LDAP[​](#connect-from-jdbcodbc-client-with-ldap "Direct link to Connect from JDBC/ODBC client with LDAP") * **JDBC** Note that when you use JDBC connections, you must enable SSL on the server side. For more information, see [SSL Authentication](https://docs.starrocks.io/docs/administration/user_privs/ssl_authentication.md). JDBC 5: ```java Properties properties = new Properties(); properties.put("authenticationPlugins", "com.mysql.jdbc.authentication.MysqlClearPasswordPlugin"); properties.put("defaultAuthenticationPlugin", "com.mysql.jdbc.authentication.MysqlClearPasswordPlugin"); properties.put("disabledAuthenticationPlugins", "com.mysql.jdbc.authentication.MysqlNativePasswordPlugin"); ``` JDBC 8: ```java Properties properties = new Properties(); properties.put("authenticationPlugins", "com.mysql.cj.protocol.a.authentication.MysqlClearPasswordPlugin"); properties.put("defaultAuthenticationPlugin", "com.mysql.cj.protocol.a.authentication.MysqlClearPasswordPlugin"); properties.put("disabledAuthenticationPlugins", "com.mysql.cj.protocol.a.authentication.MysqlNativePasswordPlugin"); ``` * **ODBC** Add `default\_auth=mysql_clear_password` and `ENABLE_CLEARTEXT\_PLUGIN=1` in the DSN of ODBC: , along with username and password. --- ### Native Authentication Create and manage users using the native authentication within StarRocks through SQL commands. StarRocks native authentication is a password-based authentication method. In addition to that, StarRocks also supports integrating with external authentication systems such as LDAP. For more instructions, see [Authenticate with Security Integration](https://docs.starrocks.io/docs/administration/user_privs/authentication/security_integration.md). note Users with the system-defined role `user_admin` can create users, alter users, and drop users in StarRocks. #### Create user[​](#create-user "Direct link to Create user") You can create a user by specifying the user identity, the authentication method, and optionally the default role. To enable the native authentication for the user, you need to explicitly specify the password in plaintext or ciphertext. The following example creates the user `jack`, allows it to connect only from the IP address `172.10.1.10`, enables the native authentication, sets the password to `12345` in plaintext, and assigns the role `example_role` to it as its default role: ```sql CREATE USER jack@'172.10.1.10' IDENTIFIED BY '12345' DEFAULT ROLE 'example_role'; ``` note * StarRocks encrypts users' passwords before storing them. You can get the encrypted password using the password() function. * A system-defined default role `PUBLIC` is assigned to a user if no default role is specified during user creation. The default role of a user is automatically activated when the user connects to StarRocks. For instructions on how to enable all (default and granted) roles for a user after connection, see [Enable all roles](https://docs.starrocks.io/docs/administration/user_privs/authorization/User_privilege.md#enable-all-roles). For more information and advanced instructions on creating a user, see [CREATE USER](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/CREATE_USER.md). #### Alter user[​](#alter-user "Direct link to Alter user") You can alter the password, default role, or property for a user. For instructions on how to alter the default role for a user, see [Alter default role](https://docs.starrocks.io/docs/administration/user_privs/authorization/User_privilege.md#alter-the-default-role-of-a-user). ##### Alter the property of a user[​](#alter-the-property-of-a-user "Direct link to Alter the property of a user") You can set the property of a user using [ALTER USER](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/ALTER_USER.md). The following example sets the maximum number of connections for user `jack` to `1000`. User identities that have the same user name share the same property. Therefore, you only need to set the property for `jack` and this setting takes effect for all the user identities with the user name `jack`. ```sql ALTER USER 'jack' SET PROPERTIES ("max_user_connections" = "1000"); ``` ##### Reset password for a user[​](#reset-password-for-a-user "Direct link to Reset password for a user") You can reset the password for a user using [SET PASSWORD](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SET_PASSWORD.md) or [ALTER USER](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/ALTER_USER.md). > **NOTE** > > * Any user can reset their own passwords without needing any privileges. > * Only the `root` user itself can set its password. If you have lost its password and cannot connect to StarRocks, see [Reset lost root password](#reset-lost-root-password) for more instructions. Both the following examples reset the password of `jack` to `54321`: * Reset the password using SET PASSWORD: ```sql SET PASSWORD FOR jack@'172.10.1.10' = PASSWORD('54321'); ``` * Reset the password using ALTER USER: ```sql ALTER USER jack@'172.10.1.10' IDENTIFIED BY '54321'; ``` ###### Reset lost root password[​](#reset-lost-root-password "Direct link to Reset lost root password") If you have lost the password of the `root` user and cannot connect to StarRocks, you can reset it by following these procedures: 1. Add the following configuration item to the configuration files **fe/conf/fe.conf** of **all FE nodes** to disable user authentication: ```yaml enable_auth_check = false ``` 2. Restart **all FE nodes** to allow the configuration to take effect. ```bash ./fe/bin/stop_fe.sh ./fe/bin/start_fe.sh ``` 3. Connect from a MySQL client to StarRocks via the `root` user. You do not need to specify the password when user authentication is disabled. ```bash mysql -h -P -uroot ``` 4. Reset the password for the `root` user. ```sql SET PASSWORD for root = PASSWORD('xxxxxx'); ``` 5. Re-enable user authentication by setting the configuration item `enable_auth_check` to `true` in the configuration files **fe/conf/fe.conf** of **all FE nodes**. ```yaml enable_auth_check = true ``` 6. Restart **all FE nodes** to allow the configuration to take effect. ```bash ./fe/bin/stop_fe.sh ./fe/bin/start_fe.sh ``` 7. Connect from a MySQL client to StarRocks using the `root` user and the new password to verify whether the password is reset successfully. ```bash mysql -h -P -uroot -p ``` #### Drop a user[​](#drop-a-user "Direct link to Drop a user") You can drop a user using [DROP USER](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/DROP_USER.md). The following example drops the user `jack`: ```sql DROP USER jack@'172.10.1.10'; ``` #### View users[​](#view-users "Direct link to View users") You can view all the users within the StarRocks cluster using SHOW USERS. ```sql SHOW USERS; ``` #### View user property[​](#view-user-property "Direct link to View user property") You can view the property of a user using [SHOW PROPERTY](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_PROPERTY.md). The following example shows the property of the user `jack`: ```sql SHOW PROPERTY FOR 'jack'; ``` Or to view a specific property: ```sql SHOW PROPERTY FOR 'jack' LIKE 'max_user_connections'; ``` --- ### OAuth 2.0 Authentication This topic describes how to enable OAuth 2.0 authentication in StarRocks. From v3.5.0 onwards, StarRocks supports authenticating client access using OAuth 2.0. You can enable OAuth 2.0 authentication over HTTP for the Web UI and the JDBC driver. StarRocks uses the [Authorization Code](https://tools.ietf.org/html/rfc6749#section-1.3.1) flow which exchanges an authorization code for a token. Generally, the flow includes the following steps: 1. The StarRocks coordinator redirects the user’s browser to the Authorization Server. 2. The user authenticates from the Authorization Server. 3. After the request is approved, the browser is redirected back to the StarRocks FE with an authorization code. 4. the StarRocks coordinator exchanges the authorization code for a token. This topic describes how to manually create and authenticate users using OAuth 2.0 in StarRocks. For instructions on how to integrate StarRocks with your OAuth 2.0 service using security integration, see [Authenticate with Security Integration](https://docs.starrocks.io/docs/administration/user_privs/authentication/security_integration.md). For more information on how to authenticate user groups in your OAuth 2.0 service, see [Authenticate User Groups](https://docs.starrocks.io/docs/administration/user_privs/group_provider.md). #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") If you want to connect to StarRocks from a MySQL client, the MySQL client version must be 9.2 or later. For more information, see [MySQL official document](https://dev.mysql.com/doc/refman/9.7/en/openid-pluggable-authentication.html). #### Create a user with OAuth 2.0[​](#create-a-user-with-oauth-20 "Direct link to Create a user with OAuth 2.0") When creating a user, specify the authentication method as OAuth 2.0 by `IDENTIFIED WITH authentication_oauth2 [AS '{xxx}']`. `{xxx}` is the OAuth 2.0 properties of the user. In addition to the following method, you can configure the default OAuth 2.0 properties in the FE configuration file. You need to manually modify all **fe.conf** files and restart all FEs for configuration to take effect. After the FE configurations are set, StarRocks will use the default properties specified in your configuration file and you can omit the `AS '{xxx}'` part. Syntax: ```sql CREATE USER IDENTIFIED WITH authentication_oauth2 [AS '{ "auth_server_url": "", "token_server_url": "", "client_id": "", "client_secret": "", "redirect_url": "", "jwks_url": "", "principal_field": "", "required_issuer": "", "required_audience": "" }'] ``` | Property | Corresponding FE Configuration | Description | | ------------------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `auth_server_url` | `oauth2_auth_server_url` | The authorization URL. The URL to which the users’ browser will be redirected in order to begin the OAuth 2.0 authorization process. | | `token_server_url` | `oauth2_token_server_url` | The URL of the endpoint on the authorization server from which StarRocks obtains the access token. | | `client_id` | `oauth2_client_id` | The public identifier of the StarRocks client. | | `client_secret` | `oauth2_client_secret` | The secret used to authorize StarRocks client with the authorization server. | | `redirect_url` | `oauth2_redirect_url` | The URL to which the users’ browser will be redirected after the OAuth 2.0 authentication succeeds. The authorization code will be sent to this URL. In most cases, it need to be configured as `http://:/api/oauth2`. | | `jwks_url` | `oauth2_jwks_url` | The URL to the JSON Web Key Set (JWKS) service or the path to the local file under the `conf` directory. | | `principal_field` | `oauth2_principal_field` | The string used to identify the field that indicates the subject (`sub`) in the JWT. The default value is `sub`. The value of this field must be identical with the username for logging in to StarRocks. | | `required_issuer` | `oauth2_required_issuer` | (Optional) The list of strings used to identify the issuers (`iss`) in the JWT. The JWT is considered valid only if one of the values in the list match the JWT issuer. | | `required_audience` | `oauth2_required_audience` | (Optional) The list of strings used to identify the audience (`aud`) in the JWT. The JWT is considered valid only if one of the values in the list match the JWT audience. | Example: ```sql CREATE USER tom IDENTIFIED WITH authentication_oauth2 AS '{ "auth_server_url": "http://localhost:38080/realms/master/protocol/openid-connect/auth", "token_server_url": "http://localhost:38080/realms/master/protocol/openid-connect/token", "client_id": "12345", "client_secret": "LsWyD9vPcM3LHxLZfzJsuoBwWQFBLcoR", "redirect_url": "http://localhost:8030/api/oauth2", "jwks_url": "http://localhost:38080/realms/master/protocol/openid-connect/certs", "principal_field": "preferred_username", "required_issuer": "http://localhost:38080/realms/master", "required_audience": "12345" }'; ``` If you have set the OAuth 2.0 properties in the FE configuration files, you can directly execute the following statement: ```sql CREATE USER tom IDENTIFIED WITH authentication_oauth2; ``` #### Connect from JDBC client with OAuth 2.0[​](#connect-from-jdbc-client-with-oauth-20 "Direct link to Connect from JDBC client with OAuth 2.0") StarRocks supports the MySQL protocol. You can customize a MySQL plugin to automatically launch the browser login method. For the example code of the JDBC OAuth2 plugin, see the official document for [starrocks-jdbc-oauth2-plugin](https://github.com/StarRocks/starrocks/tree/main/contrib/starrocks-jdbc-oauth2-plugin). #### Connect from MySQL client with OAuth 2.0[​](#connect-from-mysql-client-with-oauth-20 "Direct link to Connect from MySQL client with OAuth 2.0") If you cannot access a browser in your environment (such as using terminal or server), you can also access StarRocks via native MySQL client or JDBC driver: * When you first connect to StarRocks, a URL will be returned. * You need to access this URL on a browser and complete the authentication. * After the authentication, you can then interact with StarRocks. --- ### Authenticate with Security Integration Integrate StarRocks with external authentication systems using security integration. By creating a security integration within your StarRocks cluster, you can allow access of your external authentication service to StarRocks. With the security integration, you do not need to manually create users within StarRocks. When a user tries to log in using an external identity, StarRocks will use the corresponding security integration according to the configuration in `authentication_chain` to authenticate the user. After the authentication is successful and the user is allowed to log in, StarRocks creates a virtual user in the session for the user to perform subsequent operations. Please note that if you use the security integration to configure an external authentication method, you must also [integrate StarRocks with Apache Ranger](https://docs.starrocks.io/docs/administration/user_privs/authorization/ranger_plugin.md) to enable external authorization. Currently, integrating Security Integration with the StarRocks native authorization is not supported. You can also enable [Group Provider](https://docs.starrocks.io/docs/administration/user_privs/group_provider.md) for StarRocks to access the group information in you external authentication systems, thus allowing creating, authenticating, and authorizing user groups in StarRocks. Manually creating and managing users with external authentication services are also supported in case of specific corner cases. For more instructions, you can refer to [See also](#see-also). #### Create a security integration[​](#create-a-security-integration "Direct link to Create a security integration") Currently, StarRocks' security integration supports the following authentication systems: * LDAP * JSON Web Token (JWT) * OAuth 2.0 note StarRocks does not offer connectivity checks when you create a security integration. ##### Create a security integration with LDAP[​](#create-a-security-integration-with-ldap "Direct link to Create a security integration with LDAP") ###### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE SECURITY INTEGRATION PROPERTIES ( "type" = "authentication_ldap_simple", "authentication_ldap_simple_server_host" = "", "authentication_ldap_simple_server_port" = "", "authentication_ldap_simple_bind_base_dn" = "", "authentication_ldap_simple_user_search_attr" = "", "authentication_ldap_simple_bind_root_dn" = "", "authentication_ldap_simple_bind_root_pwd" = "", "authentication_ldap_simple_bind_dn_pattern" = "", "authentication_ldap_simple_ssl_conn_allow_insecure" = "{true | false}", "authentication_ldap_simple_ssl_conn_trust_store_path" = "", "authentication_ldap_simple_ssl_conn_trust_store_pwd" = "", "comment" = "" ) ``` ###### Parameters[​](#parameters "Direct link to Parameters") ###### security\_integration\_name[​](#security_integration_name "Direct link to security_integration_name") * Required: Yes * Description: The name of the security integration.
**NOTE**
The security integration name is globally unique. You cannot specify this parameter as `native`. ###### type[​](#type "Direct link to type") * Required: Yes * Description: The type of the security integration. Specify it as `authentication_ldap_simple`. ###### authentication\_ldap\_simple\_server\_host[​](#authentication_ldap_simple_server_host "Direct link to authentication_ldap_simple_server_host") * Required: No * Description: The IP address of your LDAP service. Default: `127.0.0.1`. ###### authentication\_ldap\_simple\_server\_port[​](#authentication_ldap_simple_server_port "Direct link to authentication_ldap_simple_server_port") * Required: No * Description: The port of your LDAP service. Default: `389`. ###### authentication\_ldap\_simple\_bind\_base\_dn[​](#authentication_ldap_simple_bind_base_dn "Direct link to authentication_ldap_simple_bind_base_dn") * Required: No * Description: The base Distinguished Name (DN) of the LDAP user for which the cluster searches. Required when using search-and-bind mode. Not needed when using direct bind mode with `authentication_ldap_simple_bind_dn_pattern`. ###### authentication\_ldap\_simple\_user\_search\_attr[​](#authentication_ldap_simple_user_search_attr "Direct link to authentication_ldap_simple_user_search_attr") * Required: No * Description: The user's attribute used to log in to the LDAP service, for example, `uid`. Required when using search-and-bind mode. note **DN Passing Mechanism**: LDAP security integration supports DN passing functionality. * After successful authentication, the system records both the user's login name and complete DN. * When combined with Group Provider, DN information is automatically passed to the Group Provider. * If `ldap_user_search_attr` is not configured for the Group Provider, DN will be used for group matching. * This mechanism is particularly suitable for complex LDAP environments like Microsoft AD. For more details, see the DN matching mechanism in [Authenticate User Groups](https://docs.starrocks.io/docs/administration/user_privs/group_provider.md). ###### authentication\_ldap\_simple\_bind\_root\_dn[​](#authentication_ldap_simple_bind_root_dn "Direct link to authentication_ldap_simple_bind_root_dn") * Required: No * Description: The admin DN of your LDAP service. Required when using search-and-bind mode. ###### authentication\_ldap\_simple\_bind\_root\_pwd[​](#authentication_ldap_simple_bind_root_pwd "Direct link to authentication_ldap_simple_bind_root_pwd") * Required: No * Description: The admin password of your LDAP service. Required when using search-and-bind mode. ###### authentication\_ldap\_simple\_bind\_dn\_pattern[​](#authentication_ldap_simple_bind_dn_pattern "Direct link to authentication_ldap_simple_bind_dn_pattern") * Required: No * Description: The DN pattern for direct bind authentication. Use `${USER}` as a placeholder for the username. The pattern must produce a valid LDAP Distinguished Name (DN); UPN-style patterns like `${USER}@domain` are not supported. For example, `uid=${USER},ou=People,dc=example,dc=com`. Multiple patterns can be separated by semicolons, and the system will try each pattern in order until one succeeds. When this parameter is set, the system skips the search step and directly binds with the constructed DN, so `authentication_ldap_simple_bind_base_dn`, `authentication_ldap_simple_user_search_attr`, `authentication_ldap_simple_bind_root_dn`, and `authentication_ldap_simple_bind_root_pwd` are not required. ###### authentication\_ldap\_simple\_ssl\_conn\_allow\_insecure[​](#authentication_ldap_simple_ssl_conn_allow_insecure "Direct link to authentication_ldap_simple_ssl_conn_allow_insecure") * Required: No * Description: Whether to allow non-encrypted connections to the LDAP server. Default value: `true`. Setting this value to `false` indicates that SSL encryption is required to access LDAP. ###### authentication\_ldap\_simple\_ssl\_conn\_trust\_store\_path[​](#authentication_ldap_simple_ssl_conn_trust_store_path "Direct link to authentication_ldap_simple_ssl_conn_trust_store_path") * Required: No * Description: Local path to store the SSL CA certificate of the LDAP server. Supports pem and jks formats. You do not need to set this item if the certificate is issued by a trusted organization. ###### ldap\_ssl\_conn\_trust\_store\_pwd[​](#ldap_ssl_conn_trust_store_pwd "Direct link to ldap_ssl_conn_trust_store_pwd") * Required: No * Description: The password used to access the locally stored SSL CA certificate of the LDAP server. pem-formatted certificates do not require a password. Only jsk-formatted certificates do. ###### group\_provider[​](#group_provider "Direct link to group_provider") * Required: No * Description: The name of the group provider(s) to be combined with the security integration. Multiple group providers are separated by commas. Once set, StarRocks will record the user's group information under each specified provider upon login. Supported from v3.5 onwards. For detailed instructions on enabling Group Provider, see [Authenticate User Groups](https://docs.starrocks.io/docs/administration/user_privs/group_provider.md). ###### permitted\_groups[​](#permitted_groups "Direct link to permitted_groups") * Required: No * Description: The name of group(s) whose members are allowed to log in to StarRocks. Multiple groups are separated by commas. Make sure that the specified groups can be retrieved by the combined group provider(s). Supported from v3.5 onwards. ###### comment[​](#comment "Direct link to comment") * Required: No * Description: The description of the security integration. ##### Create a security integration with JWT[​](#create-a-security-integration-with-jwt "Direct link to Create a security integration with JWT") ###### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE SECURITY INTEGRATION PROPERTIES ( "type" = "authentication_jwt", "jwks_url" = "", "principal_field" = "", "required_issuer" = "", "required_audience" = "" "comment" = "" ); ``` ###### Parameters[​](#parameters "Direct link to Parameters") ###### security\_integration\_name[​](#security_integration_name "Direct link to security_integration_name") * Required: Yes * Description: The name of the security integration.
**NOTE**
The security integration name is globally unique. You cannot specify this parameter as `native`. ###### type[​](#type "Direct link to type") * Required: Yes * Description: The type of the security integration. Specify it as `jwt`. ###### jwks\_url[​](#jwks_url "Direct link to jwks_url") * Required: Yes * Description: The URL to the JSON Web Key Set (JWKS) service or the path to the local file under the `fe/conf` directory. ###### principal\_field[​](#principal_field "Direct link to principal_field") * Required: Yes * Description: The string used to identify the field that indicates the subject (`sub`) in the JWT. The default value is `sub`. The value of this field must be identical with the username for logging in to StarRocks. ###### required\_issuer[​](#required_issuer "Direct link to required_issuer") * Required: No * Description: The list of strings used to identify the issuers (`iss`) in the JWT. The JWT is considered valid only if one of the values in the list match the JWT issuer. ###### required\_audience[​](#required_audience "Direct link to required_audience") * Required: No * Description: The list of strings used to identify the audience (`aud`) in the JWT. The JWT is considered valid only if one of the values in the list match the JWT audience. ###### comment[​](#comment "Direct link to comment") * Required: No * Description: The description of the security integration. ##### Create a security integration with OAuth 2.0[​](#create-a-security-integration-with-oauth-20 "Direct link to Create a security integration with OAuth 2.0") ###### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE SECURITY INTEGRATION PROPERTIES ( "type" = "authentication_oauth2", "auth_server_url" = "", "token_server_url" = "", "client_id" = "", "client_secret" = "", "redirect_url" = "", "jwks_url" = "", "principal_field" = "", "required_issuer" = "", "required_audience" = "" "comment" = "" ) ``` ###### Parameters[​](#parameters "Direct link to Parameters") ###### security\_integration\_name[​](#security_integration_name "Direct link to security_integration_name") * Required: Yes * Description: The name of the security integration.
**NOTE**
The security integration name is globally unique. You cannot specify this parameter as `native`. ###### auth\_server\_url[​](#auth_server_url "Direct link to auth_server_url") * Required: Yes * Description: The authorization URL. The URL to which the users’ browser will be redirected in order to begin the OAuth 2.0 authorization process. ###### token\_server\_url[​](#token_server_url "Direct link to token_server_url") * Required: Yes * Description: The URL of the endpoint on the authorization server from which StarRocks obtains the access token. ###### client\_id[​](#client_id "Direct link to client_id") * Required: Yes * Description: The public identifier of the StarRocks client. ###### client\_secret[​](#client_secret "Direct link to client_secret") * Required: Yes * Description: The secret used to authorize StarRocks client with the authorization server. ###### redirect\_url[​](#redirect_url "Direct link to redirect_url") * Required: Yes * Description: The URL to which the users’ browser will be redirected after the OAuth 2.0 authentication succeeds. The authorization code will be sent to this URL. In most cases, it need to be configured as `http://:/api/oauth2`. ###### type[​](#type "Direct link to type") * Required: Yes * Description: The type of the security integration. Specify it as `authentication_oauth2`. ###### jwks\_url[​](#jwks_url "Direct link to jwks_url") * Required: Yes * Description: The URL to the JSON Web Key Set (JWKS) service or the path to the local file under the `fe/conf` directory. ###### principal\_field[​](#principal_field "Direct link to principal_field") * Required: Yes * Description: The string used to identify the field that indicates the subject (`sub`) in the JWT. The default value is `sub`. The value of this field must be identical with the username for logging in to StarRocks. ###### required\_issuer[​](#required_issuer "Direct link to required_issuer") * Required: No * Description: The list of strings used to identify the issuers (`iss`) in the JWT. The JWT is considered valid only if one of the values in the list match the JWT issuer. ###### required\_audience[​](#required_audience "Direct link to required_audience") * Required: No * Description: The list of strings used to identify the audience (`aud`) in the JWT. The JWT is considered valid only if one of the values in the list match the JWT audience. ###### comment[​](#comment "Direct link to comment") * Required: No * Description: The description of the security integration. #### Configure authentication chain[​](#configure-authentication-chain "Direct link to Configure authentication chain") After the security integration is created, it is added to your StarRocks cluster as a new authentication method. You must enable the security integration by setting the order of the authentication methods via the FE dynamic configuration item `authentication_chain`. ```sql ADMIN SET FRONTEND CONFIG ( "authentication_chain" = "[... ,]" ); ``` note * StarRocks prioritizes native authentication for local users. If a local user with the same username does not exist, authentication is performed in the order configured by `authentication_chain`. If login fails using the native authentication method, the cluster will try the next authentication method in the specified order. * You can specify multiple security integrations in `authentication_chain` except for OAuth 2.0 security integration. You cannot specify multiple OAuth 2.0 security integrations or one with other security integrations. You can check the value of `authentication_chain` using the following statement: ```sql ADMIN SHOW FRONTEND CONFIG LIKE 'authentication_chain'; ``` #### Manage security integrations[​](#manage-security-integrations "Direct link to Manage security integrations") ##### Alter security integration[​](#alter-security-integration "Direct link to Alter security integration") You can alter the configuration of an existing security integration using the following statement: ```sql ALTER SECURITY INTEGRATION SET ( "key"="value"[, ...] ) ``` note You cannot alter the `type` of a security integration. ##### Drop security integration[​](#drop-security-integration "Direct link to Drop security integration") You can drop an existing security integration using the following statement: ```sql DROP SECURITY INTEGRATION ``` ##### View security integration[​](#view-security-integration "Direct link to View security integration") You can view all security integrations in your cluster using the following statement: ```sql SHOW SECURITY INTEGRATIONS; ``` Example: ```plain SHOW SECURITY INTEGRATIONS; +--------+--------+---------+ | Name | Type | Comment | +--------+--------+---------+ | LDAP1 | LDAP | NULL | +--------+--------+---------+ ``` | **Parameter** | **Description** | | ------------- | ------------------------------------------------------------------------------------------------------------------------------ | | Name | The name of the security integration. | | Type | The type of the security integration. | | Comment | The description of the security integration. `NULL` is returned when no description is specified for the security integration. | You can check the details of a security integration using the following statement: ```sql SHOW CREATE SECURITY INTEGRATION ``` Example: ```plain SHOW CREATE SECURITY INTEGRATION LDAP1； +----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Security Integration | Create Security Integration | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | LDAP1 | CREATE SECURITY INTEGRATION LDAP1 PROPERTIES ( "type" = "authentication_ldap_simple", "authentication_ldap_simple_server_host" = "", "authentication_ldap_simple_server_port" = "", "authentication_ldap_simple_bind_base_dn" = "", "authentication_ldap_simple_user_search_attr" = "" "authentication_ldap_simple_bind_root_dn" = "", "authentication_ldap_simple_bind_root_pwd" = "", "authentication_ldap_simple_ssl_conn_allow_insecure" = "{true | false}", "authentication_ldap_simple_ssl_conn_trust_store_path" = "", "authentication_ldap_simple_ssl_conn_trust_store_pwd" = "", "comment" = "" )| +----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ``` note `ldap_bind_root_pwd` is masked when SHOW CREATE SECURITY INTEGRATION is executed. #### Connect to StarRocks via a security integration[​](#connect-to-starrocks-via-a-security-integration "Direct link to Connect to StarRocks via a security integration") * For instructions on how to connect to StarRocks via LDAP, see [LDAP Authentication - Connect to StarRocks](https://docs.starrocks.io/docs/administration/user_privs/authentication/ldap_authentication.md#connect-from-mysql-client-with-ldap). * For instructions on how to connect to StarRocks via JWT, see [JSON Web Token Authentication - Connect to StarRocks](https://docs.starrocks.io/docs/administration/user_privs/authentication/jwt_authentication.md#connect-from-mysql-client-with-jwt). * For instructions on how to connect to StarRocks via OAuth 2.0, see [OAuth 2.0 Authentication - Connect to StarRocks](https://docs.starrocks.io/docs/administration/user_privs/authentication/oauth2_authentication.md#connect-from-jdbc-client-with-oauth-20). #### See also[​](#see-also "Direct link to See also") * For instructions on how to manually authenticate users via LDAP in StarRocks, see [LDAP Authentication](https://docs.starrocks.io/docs/administration/user_privs/authentication/ldap_authentication.md). * For instructions on how to manually authenticate users via JSON Web Token in StarRocks, see [JSON Web Token Authentication](https://docs.starrocks.io/docs/administration/user_privs/authentication/jwt_authentication.md). * For instructions on how to manually authenticate users via OAuth 2.0 in StarRocks, see [OAuth 2.0 Authentication](https://docs.starrocks.io/docs/administration/user_privs/authentication/oauth2_authentication.md). * For instructions on how to authenticate user groups, see [Authenticate User Groups](https://docs.starrocks.io/docs/administration/user_privs/group_provider.md). --- ### Built-in Roles supported by StarRocks In a StarRocks cluster, there are FIVE built-in roles: * `db_admin` * `cluster_admin` * `user_admin` * `security_admin` * `public` Each of the `admin` roles is granted with different privileges to allow them to perform administrative operations on their specific domain. By default, the `public` role has no privileges and is granted to every user that can access the cluster. For details of the privileges described below, see [Privilege Item](https://docs.starrocks.io/docs/administration/user_privs/authorization/privilege_item.md). #### `db_admin`[​](#db_admin "Direct link to db_admin") `db_admin` is the built-in database administrator. It has all data-related privileges and some basic privileges on operations and maintenance. * Focused on management of databases and data * Unavailable for user or cluster management * Immutable role Privilege scope: | Privilege Level | Privilege Item | | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | SYSTEM | - CREATE RESOURCE
- PLUGIN
- FILE
- BLACKLIST
- OPERATE
- CREATE EXTERNAL CATALOG
- REPOSITORY
- CREATE RESOURCE GROUP
- CREATE GLOBAL FUNCTION
- CREATE STORAGE VOLUME
- SECURITY | | CATALOG | - USAGE
- DROP
- ALTER
- CREATE DATABASE | | DATABASE | - DROP
- ALTER
- CREATE TABLE
- CREATE VIEW
- CREATE MATERIALIZED VIEW
- CREATE FUNCTION
- CREATE PIPE
- CREATE MASKING POLICY
- CREATE ROW ACCESS POLICY | | TABLE | - DROP
- ALTER
- INSERT
- UPDATE
- DELETE
- SELECT
- EXPORT | | VIEW | - DROP
- ALTER
- SELECT | | MATERIALIZED VIEW | - DROP
- ALTER
- SELECT
- REFRESH | | RESOURCE | - USAGE
- DROP
- ALTER | | RESOURCE GROUP | - DROP
- ALTER | | FUNCTION | - USAGE
- DROP | | GLOBAL FUNCTION | - USAGE
- DROP | | STORAGE VOLUME | - USAGE
- DROP
- ALTER | | PIPE | - USAGE
- DROP
- ALTER | #### `cluster_admin`[​](#cluster_admin "Direct link to cluster_admin") `cluster_admin` is the built-in cluster administrator. * Focused on management of cluster infrastructure * Granted with privileges on node management * Immutable role Privilege scope: | Privilege Level | Privilege Item | | --------------- | -------------- | | SYSTEM | NODE | #### `user_admin`[​](#user_admin "Direct link to user_admin") `user_admin` is the built-in user administrator. It can be used to manage users, roles, and authorization. * Focused on management of users and privileges * Able to create, alter, and drop users * Able to grant or revoke privileges or roles * Immutable role Privilege scope: | Privilege Level | Privilege Item | | --------------- | -------------- | | SYSTEM | GRANT | #### `security_admin`[​](#security_admin "Direct link to security_admin") `security_admin` is the built-in security administrator. It can be used to manage security integrations and group providers. * Focused on management of system security * Able to manage security-related configurations and strategies * Immutable role Privilege scope: | Privilege Level | Privilege Item | | --------------- | ------------------------- | | SYSTEM | - SECURITY
- OPERATE | #### `public`[​](#public "Direct link to public") `public` is the built-in role that is granted to every user that can access the cluster. By default, it has no privilege. * Automatically granted and activated to all cluster users * Mutable role. You can grant privileges or roles to this role if you want to grant them to all cluster users. --- ### Privilege FAQ #### Why is the error message "no permission" still reported even after the required role has been assigned to a user?[​](#why-is-the-error-message-no-permission-still-reported-even-after-the-required-role-has-been-assigned-to-a-user "Direct link to Why is the error message \"no permission\" still reported even after the required role has been assigned to a user?") This error may happen if the role is not activated. You can run `select current_role();` to query the roles that have been activated for the user in the current session. If the required role is not activated, run [SET ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SET_ROLE.md) to activate this role and perform operations using this role. If you want roles to be automatically activated upon login, the `user_admin` role can run [SET DEFAULT ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SET_DEFAULT_ROLE.md) or [ALTER USER DEFAULT ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/ALTER_USER.md) to set a default role for each user. After the default role is set, it will be automatically activated when the user logs in. If you want all the assigned roles of all users to be automatically activated upon login, you can run the following command. This operation requires the OPERATE permission at the System level. ```sql SET GLOBAL activate_all_roles_on_login = TRUE; ``` However, we recommend that you follow the principle of "least privilege" by setting default roles with limited privileges to prevent potential risks. For example: * Common users can set the `read_only` role that has only the SELECT privilege as the default role, while avoiding setting roles with privileges like ALTER, DROP, and INSERT as default roles. * Administrators can set the `db_admin` role as the default role, while avoiding setting the `node_admin` role, which has the privilege to add and drop nodes, as the default role. This approach helps ensure that users are assigned roles with appropriate permissions, reducing the risk of unintended operations. You can run [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to assign the required privileges or roles to users. #### I have granted a user the privilege on all tables in a database (`GRANT ALL ON ALL TABLES IN DATABASE TO USER ;`), but the user still cannot create tables in the database. Why?[​](#i-have-granted-a-user-the-privilege-on-all-tables-in-a-database-grant-all-on-all-tables-in-database-db_name-to-user-user_identity-but-the-user-still-cannot-create-tables-in-the-database-why "Direct link to i-have-granted-a-user-the-privilege-on-all-tables-in-a-database-grant-all-on-all-tables-in-database-db_name-to-user-user_identity-but-the-user-still-cannot-create-tables-in-the-database-why") Creating tables within a database requires the database-level CREATE TABLE privilege. You need to grant the privilege to the user. ```sql GRANT CREATE TABLE ON DATABASE TO USER ;; ``` #### I have granted a user all the privileges on a database using `GRANT ALL ON DATABASE TO USER ;`, but nothing is returned when the user runs `SHOW TABLES;` in this database. Why?[​](#i-have-granted-a-user-all-the-privileges-on-a-database-using-grant-all-on-database-db_name-to-user-user_identity-but-nothing-is-returned-when-the-user-runs-show-tables-in-this-database-why "Direct link to i-have-granted-a-user-all-the-privileges-on-a-database-using-grant-all-on-database-db_name-to-user-user_identity-but-nothing-is-returned-when-the-user-runs-show-tables-in-this-database-why") `SHOW TABLES;` returns only tables on which the user has any privilege. If the user has no privilege on a table, this table will not be returned. You can grant any privilege on all tables in this database (using SELECT for example) to the user: ```sql GRANT SELECT ON ALL TABLES IN DATABASE TO USER ; ``` The statement above is equivalent to `GRANT select_priv ON db.* TO ;` used in versions earlier than v3.0. #### What privileges are required to access the StarRocks Web Console `http://:`?[​](#what-privileges-are-required-to-access-the-starrocks-web-console-httpfe_ipfe_http_port "Direct link to what-privileges-are-required-to-access-the-starrocks-web-console-httpfe_ipfe_http_port") The user must have the `db_admin` and `user_admin` roles. #### How did the privilege retention mechanism change before and after StarRocks v3.0?[​](#how-did-the-privilege-retention-mechanism-change-before-and-after-starrocks-v30 "Direct link to How did the privilege retention mechanism change before and after StarRocks v3.0?") Before v3.0, after a user is granted privileges on a table, the privileges would still remain even if the table was dropped and recreated. Starting from v3.0, privileges will no longer be retained after a table is dropped and recreated. #### How to query users and granted privileges in StarRocks?[​](#how-to-query-users-and-granted-privileges-in-starrocks "Direct link to How to query users and granted privileges in StarRocks?") You can obtain the full user list via querying the system view `sys.grants_to_users` or executing SHOW USERS, and then query each user individually using `SHOW GRANTS FOR `. #### What is the impact on FE resources when querying system views on privilege metadata of large numbers of users and tables?[​](#what-is-the-impact-on-fe-resources-when-querying-system-views-on-privilege-metadata-of-large-numbers-of-users-and-tables "Direct link to What is the impact on FE resources when querying system views on privilege metadata of large numbers of users and tables?") When the number of users or tables is very large, queries on system views `sys.grants_to_users`, `sys.grants_to_roles`, and `sys.role_edges` may take a long time. These views are computed in real time, consuming a proportion of FE resources. Therefore, it is not recommended to run such operations frequently at large scale. #### Will recreating a catalog cause permission loss? How should privileges be backed up and restored?[​](#will-recreating-a-catalog-cause-permission-loss-how-should-privileges-be-backed-up-and-restored "Direct link to Will recreating a catalog cause permission loss? How should privileges be backed up and restored?") Yes. Recreating a catalog will cause its relative privileges to be lost. You should back up all user privileges first and restore them after the catalog is recreated. #### Is there a tool that supports automatic permission migration?[​](#is-there-a-tool-that-supports-automatic-permission-migration "Direct link to Is there a tool that supports automatic permission migration?") Not at the moment. Users must manually back up and restore permissions using SHOW GRANTS for each user. #### Are there restrictions on using the KILL command? Can it be restricted to only killing a user’s own queries?[​](#are-there-restrictions-on-using-the-kill-command-can-it-be-restricted-to-only-killing-a-users-own-queries "Direct link to Are there restrictions on using the KILL command? Can it be restricted to only killing a user’s own queries?") Yes. The KILL command now requires the OPERATE privilege, and a user can only kill queries that were initiated by themselves. #### Why do the granted privileges change after renaming or deleting a table? Can the system retain old permissions while adding permissions for the renamed table?[​](#why-do-the-granted-privileges-change-after-renaming-or-deleting-a-table-can-the-system-retain-old-permissions-while-adding-permissions-for-the-renamed-table "Direct link to Why do the granted privileges change after renaming or deleting a table? Can the system retain old permissions while adding permissions for the renamed table?") For native tables, privileges are tied to the table ID, not the table name. This ensures data security, as table names can change arbitrarily. If privileges were to follow table names, it could cause data leakage. Similarly, when a table is dropped, its permissions are removed because the object no longer exists. For external tables, historical versions behaved the same as internal tables. However, because external table metadata is not managed by StarRocks, delays or privilege loss can occur. To address this, future versions will use table-name–based permission management for external tables, which aligns with the expected behavior. #### How to back up user privileges?[​](#how-to-back-up-user-privileges "Direct link to How to back up user privileges?") Below is a sample script for backing up the user privilege information in the cluster. ```bash #!/bin/bash # MySQL connection info HOST="" PORT="9030" USER="root" PASSWORD="" OUTPUT_FILE="user_privileges.txt" # Clear output file > $OUTPUT_FILE # Get user list users=$(mysql -h$HOST -P$PORT -u$USER -p$PASSWORD -e "SHOW USERS;" | sed -e '1d' -e '/^+/d') # Loop through users and get privileges for user in $users; do echo "Privileges for $user:" >> $OUTPUT_FILE mysql -h$HOST -P$PORT -u$USER -p$PASSWORD -e "SHOW GRANTS FOR $user;" >> $OUTPUT_FILE echo "" >> $OUTPUT_FILE done echo "All user privileges have been written to $OUTPUT_FILE" ``` #### Why does granting USAGE on a normal function produce the error “Unexpected input 'IN', the most similar input is TO.”? What is the correct way to grant permissions on functions?[​](#why-does-granting-usage-on-a-normal-function-produce-the-error-unexpected-input-in-the-most-similar-input-is-to-what-is-the-correct-way-to-grant-permissions-on-functions "Direct link to why-does-granting-usage-on-a-normal-function-produce-the-error-unexpected-input-in-the-most-similar-input-is-to-what-is-the-correct-way-to-grant-permissions-on-functions") Normal functions cannot be granted using IN ALL DATABASES; they can only be granted within the current database. While global functions are granted on the ALL DATABASES scale. --- ### Privileges supported by StarRocks Privileges granted to a user or role determine which operations the user or role can perform on certain objects. Privileges can be used to implement fine-grained access control to safeguard data security. This topic describes privileges provided by StarRocks on different objects and their meanings. Privileges are granted and revoked by using [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) and [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE.md). The privileges that can be granted on an object are specific to the object type. For example, table privileges are different from database privileges. important The privileges described in this topic are available only from v3.0. The privilege framework and syntax in v3.0 are not backward compatible with those in earlier versions. After an upgrade to v3.0, most of your original privileges are still retained except those for specific operations. For the detailed differences, see [Upgrade notes](#upgrade-notes) at the end of this topic. #### Privilege list[​](#privilege-list "Direct link to Privilege list") This section describes privileges that are available on different objects. ##### SYSTEM[​](#system "Direct link to SYSTEM") | Privilege | Description | | ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | NODE | Operates nodes, such as adding, deleting, or decommissioning nodes. To ensure cluster security, this privilege cannot be directly granted to users or roles. The `cluster_admin` role has this privilege. | | GRANT | Creates a user or role, alters a user or role, or grants privileges to a user or role. This privilege cannot be directly granted to users or roles. The `user_admin` role has this privilege. | | CREATE RESOURCE GROUP | Creates a resource group. | | CREATE RESOURCE | Creates resources for Spark Load jobs or external tables. | | CREATE EXTERNAL CATALOG | Creates an external catalog. | | PLUGIN | Installs or uninstalls a plugin. | | REPOSITORY | Creates, deletes, or views repositories. | | BLACKLIST | Creates, deletes, or displays SQL blacklists and BE Blacklist. | | FILE | Creates, deletes, or views files. | | OPERATE | Manages replicas, configuration items, variables, and transactions. | | CREATE GLOBAL FUNCTION | Creates a global UDF. | | CREATE STORAGE VOLUME | Creates a storage volume for a remote storage system. | | SECURITY | Creates or manages security integrations and group providers. | ##### RESOURCE GROUP[​](#resource-group "Direct link to RESOURCE GROUP") | Privilege | Description | | --------- | ------------------------------------------------- | | ALTER | Adds or deletes classifiers for a resource group. | | DROP | Deletes a resource group. | | ALL | Has all the above privileges on a resource group. | ##### RESOURCE[​](#resource "Direct link to RESOURCE") | Privilege | Description | | --------- | ------------------------------------------- | | USAGE | Uses a resource. | | ALTER | Alters a resource. | | DROP | Deletes a resource. | | ALL | Has all the above privileges on a resource. | ##### USER[​](#user "Direct link to USER") | Privilege | Description | | ----------- | ---------------------------------------------- | | IMPERSONATE | Allows user A to perform operations as user B. | ##### GLOBAL FUNCTION (Global UDFs)[​](#global-function-global-udfs "Direct link to GLOBAL FUNCTION (Global UDFs)") | Privilege | Description | | --------- | ------------------------------------------- | | USAGE | Uses a function in a query. | | DROP | Deletes a function. | | ALL | Has all the above privileges on a function. | ##### CATALOG[​](#catalog "Direct link to CATALOG") | Object | Privilege | Description | | -------------------------- | --------------- | ----------------------------------------------------- | | CATALOG (internal catalog) | USAGE | Uses the internal catalog (default\_catalog). | | CATALOG (internal catalog) | CREATE DATABASE | Creates databases in the internal catalog. | | CATALOG (internal catalog) | ALL | Has all the above privileges on the internal catalog. | | CATALOG (external catalog) | USAGE | Uses an external catalog. | | CATALOG (external catalog) | DROP | Deletes an external catalog. | | CATALOG (external catalog) | ALL | Has all the above privileges on the external catalog. | > Notes: StarRocks internal catalog cannot be deleted. ##### DATABASE[​](#database "Direct link to DATABASE") | Privilege | Description | | ------------------------ | --------------------------------------------------------------------------------- | | ALTER | Sets properties for a database, rename a database, or sets quotas for a database. | | DROP | Deletes a database. | | CREATE TABLE | Creates tables in a database. | | CREATE VIEW | Creates a view. | | CREATE FUNCTION | Creates a function. | | CREATE MATERIALIZED VIEW | Creates a materialized view. | | ALL | Has all the above privileges on a database. | ##### TABLE[​](#table "Direct link to TABLE") | Privilege | Description | | --------- | ------------------------------------------------------------------------------------------------ | | ALTER | Modifies a table or refreshes metadata in an external table. | | DROP | Drops a table. | | SELECT | Queries data in a table. | | INSERT | Inserts data into a table. | | UPDATE | Updates data in a table. | | EXPORT | Exports data from a StarRocks table. | | DELETE | Deletes data from a table based on the specified condition or deletes all the data from a table. | | ALL | Has all the above privileges on a table. | ##### VIEW[​](#view "Direct link to VIEW") | Privilege | Description | | --------- | --------------------------------------- | | SELECT | Queries data in a view. | | ALTER | Modifies the definition of a view. | | DROP | Deletes a logical view. | | ALL | Has all the above privileges on a view. | ##### MATERIALIZED VIEW[​](#materialized-view "Direct link to MATERIALIZED VIEW") | Privilege | Description | | --------- | ---------------------------------------------------- | | SELECT | Queries a materialized view to accelerate queries. | | ALTER | Changes a materialized view. | | REFRESH | Refreshes a materialized view. | | DROP | Deletes a materialized view. | | ALL | Has all the above privileges on a materialized view. | ##### FUNCTION (Database-level UDFs)[​](#function-database-level-udfs "Direct link to FUNCTION (Database-level UDFs)") | Privilege | Description | | --------- | ------------------------------------------- | | USAGE | Uses a function. | | DROP | Deletes a function. | | ALL | Has all the above privileges on a function. | ##### STORAGE VOLUME[​](#storage-volume "Direct link to STORAGE VOLUME") | Privilege | Description | | --------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | ALTER | Alters the credential properties, comment, or status (enabled) of a storage volume, and sets a storage volume as the default storage volume. | | DROP | Drops a storage volume. | | USAGE | Describes a storage volume. | | ALL | Has all the above privileges on a storage volume. | #### Upgrade notes[​](#upgrade-notes "Direct link to Upgrade notes") During an upgrade from v2.x to v3.0, some of your operations may be unable to perform due to the introduction of the new privilege system. The following table describes the changes before and after the upgrade. | **Operation** | **Commands involved** | **Before** | **After** | | --------------------------- | ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Change table | ALTER TABLE, CANCEL ALTER TABLE | Users who have the `LOAD_PRIV` privilege on a table or the database to which the table belongs can perform the `ALTER TABLE` and `CANCEL ALTER TABLE` operations. | You must have the ALTER privilege on the table to perform these two operations. | | Refresh external table | REFRESH EXTERNAL TABLE | Users who have the `LOAD_PRIV` privilege on an external table can refresh the external table. | You must have the ALTER privilege on the external table to perform this operation. | | Backup and restore | BACKUP, RESTORE | Users who have the `LOAD_PRIV` privilege on a database can back up and restore the database or any table in the database. | The administrator must grant backup and restore privileges to users again after the upgrade. | | Recover after deletion | RECOVER | Users who have the `ALTER_PRIV`, `CREATE_PRIV`, and `DROP_PRIV` privileges on the database and table can recover the database and table. | You must have the CREATE DATABASE privilege on the default\_catalog to recover the database. You must have the CREATE TABLE privilege on the database and the DROP privilege on the table. | | Create and change users | CREATE USER, ALTER USER | Users who have the `GRANT_PRIV` privilege on the database can create and change users. | You must have the `user_admin` role to create and change users. | | Grant and revoke privileges | GRANT, REVOKE | Users who have the `GRANT_PRIV` privilege on an object can grant privileges on the object to other users or roles. | After the upgrade, you can still grant the privileges you already have on that object to other users or roles after the upgrade.
In the new privilege system: - You must have the `user_admin` role to grant privileges to other users or roles.
- If your GRANT statement includes `WITH GRANT OPTION`, you can grant the privileges involved in the statement to other users or roles. | In v2.x, StarRocks does not fully implement role-based access control (RBAC). When you assign a role to a user, StarRocks directly grants all the privileges of the role to the user, instead of the role itself. Therefore, the user does not actually own the role. In v3.0, StarRocks renovates its privilege system. After an upgrade to v3.0, your original roles are retained but there is still no ownership between users and roles. If you want to use the new RBAC system, perform the GRANT operation to assign roles and privileges. --- ### Manage permissions with Apache Ranger [Apache Ranger](https://ranger.apache.org/) provides a centralized security management framework that allows users to customize access policies through a visual web page. This helps determine which roles can access which data and exercise fine-grained data access control for various components and services in the Hadoop ecosystem. Apache Ranger provides the following core modules: * **Ranger Admin**: the core module of Ranger with a built-in web page. Users can create and update security policies on this page or through a REST interface. Plugins of various components of the Hadoop ecosystem poll and pull these policies at a regular basis. * **Agent Plugin**: plugins of components embedded in the Hadoop ecosystem. These plugins pull security policies from Ranger Admin on a regular basis and store the policies in local files. When users access a component, the corresponding plugin assesses the request based on the configured security policy and sends the authentication results to the corresponding component. * **User Sync**: used to pull user and user group information, and synchronize the permission data of users and user groups to Ranger's database. In addition to the native RBAC privilege system, StarRocks v3.1.9 also supports access control through Apache Ranger. Currently, StarRocks supports: * Creates access policies, masking policies, and row-level filter policies through Apache Ranger. * Ranger audit logs. * **Ranger Servers that use Kerberos for authentication are not supported.** * You can register multiple StarRocks Services in the same Apache Ranger service to manage privileges in different StarRocks clusters. This topic describes the permission control methods and integration process of StarRocks and Apache Ranger. For information on how to create security policies on Ranger to manage data security, see the [Apache Ranger official website](https://ranger.apache.org/). From v3.5.0 onwards, StarRocks supports Group Provider to collect group information from external authentication systems for user group management. For more information, see [Authenticate User Groups](https://docs.starrocks.io/docs/administration/user_privs/group_provider.md). #### Permission control method[​](#permission-control-method "Direct link to Permission control method") StarRocks integrated with Apache Ranger provides the following permission control methods: * Create StarRocks Service in Ranger to implement permission control. When users access StarRocks internal tables, external tables, or other objects, access control is performed according to the access policies configured in StarRocks Service. * When users access an external data source, the external service (such as the Hive Service) on Apache Ranger can be reused for access control. StarRocks can match Ranger services with different External Catalogs and implements access control based on the Ranger service corresponding to the data source. After StarRocks is integrating with Apache Ranger, you can achieve the following access control patterns: * Use Apache Ranger to uniformly manage access to StarRocks internal tables, external tables, and all objects. * Use Apache Ranger to manage access to StarRocks internal tables and objects. For External Catalogs, reuse the policy of the corresponding external service on Ranger for access control. * Use Apache Ranger to manage access to External Catalogs by reusing the Service corresponding to the external data source. Use StarRocks native RBAC privilege system to manage access to StarRocks internal tables and objects. **Authentication process** 1. You can also use LDAP for user authentication, then use Ranger to synchronize LDAP users and configure access rules for them. StarRocks can also complete user login authentication through LDAP. 2. When users initiate a query, StarRocks parses the query statement, passes user information and required privileges to Apache Ranger. Ranger determines whether the user has the required privilege based on the access policy configured in the corresponding Service, and returns the authentication result to StarRocks. If the user has access, StarRocks returns the query data; if not, StarRocks returns an error. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Apache Ranger 2.1.0 or later has been installed. For the instructions on how to install Apache Ranger, see [Ranger quick start](https://ranger.apache.org/quick_start_guide.html). * All StarRocks FE machines have access to Apache Ranger. You can check this by running the following command on each FE machine: ```sql telnet ``` If `Connected to ` is displayed, the connection is successful. #### Integrate StarRocks Service with Ranger[​](#integrate-starrocks-service-with-ranger "Direct link to Integrate StarRocks Service with Ranger") ##### (Optional) Install ranger-starrocks-plugin[​](#optional-install-ranger-starrocks-plugin "Direct link to (Optional) Install ranger-starrocks-plugin") note The main purpose of this step is to use Ranger's resource name autocomplete feature. When authoring policies in Ranger Admin, users need to enter the name of the resources whose access need to be protected. To make it easier for users to enter the resource names, Ranger Admin provides the autocomplete feature, which looks up the available resources in the service that match the input entered so far and automatically completes the resource name. If you do not have the permissions to operate the Ranger cluster or do not need this feature, you can skip this step. Also, please notice that if you didn't install the ranger-starrocks-plugin, then you cannot use `test connection` when creating StarRocks service. However, that doesn't mean that you can not create the service successfully. 1. Create the `starrocks` folder in the Ranger Admin directory `ews/webapp/WEB-INF/classes/ranger-plugins`. ```sql mkdir {path-to-ranger}/ews/webapp/WEB-INF/classes/ranger-plugins/starrocks ``` 2. Download [plugin-starrocks/target/ranger-starrocks-plugin-3.0.0-SNAPSHOT.jar](https://www.starrocks.io/download/community) and [mysql-connector-j.jar](https://dev.mysql.com/downloads/connector/j/), and place them in the `starrocks` folder. 3. Restart Ranger Admin. * Ranger 0.5.x: ```sql ranger-admin restart ``` * Ranger 2.x and above: ```sql ./ews/ranger-admin-services.sh restart ``` ##### Configure StarRocks Service on Ranger Admin[​](#configure-starrocks-service-on-ranger-admin "Direct link to Configure StarRocks Service on Ranger Admin") note This step configures the StarRocks Service on Ranger so that users can perform access control on StarRocks objects through Ranger. 1. Copy [ranger-servicedef-starrocks.json](https://github.com/StarRocks/starrocks/blob/main/conf/ranger/ranger-servicedef-starrocks.json) to any directory of the StarRocks FE machine or Ranger machine. ```sql wget https://raw.githubusercontent.com/StarRocks/starrocks/main/conf/ranger/ranger-servicedef-starrocks.json ``` note If you do not need Ranger's autocomplete feature (which means you did not install the ranger-starrocks-plugin), you must set `implClass` in the .json file to empty: ```json "implClass": "", ``` If you need Ranger's autocomplete feature (which means you have installed the ranger-starrocks-plugin), you must set `implClass` in the .json file to `org.apache.ranger.services.starrocks.RangerServiceStarRocks`: ```json "implClass": "org.apache.ranger.services.starrocks.RangerServiceStarRocks", ``` 2. Add StarRocks Service by running the following command as a Ranger administrator. ```bash curl -u : \ -X POST -H "Accept: application/json" \ -H "Content-Type: application/json" http://:/service/plugins/definitions -d@ranger-servicedef-starrocks.json ``` 3. Access `http://:/login.jsp` to log in to the Apache Ranger page. The STARROCKS service appears on the page. ![home](/assets/images/ranger_home-c446eb652890149c921b41f81e301926.png) 4. Click the plus sign (`+`) after **STARROCKS** to configure StarRocks Service. ![service detail](/assets/images/ranger_service_details-df17ab63545b9b49e05f6d3c56477186.png) ![property](/assets/images/ranger_properties-d836dd82cdc123eafdaf4406cd1105fc.png) * `Service Name`: You must enter a service name. * `Display Name`: The name you want to display for the service under STARROCKS. If it is not specified, `Service Name` will be displayed. * `Username` and `Password`: FE username and password, used to auto-complete object names when creating policies. The two parameters do not affect the connectivity between StarRocks and Ranger. If you want to use auto-completion, configure at least one user with the `db_admin` role activated. * `jdbc.url`: Enter the StarRocks FE IP address and port. The following figure shows a configuration example. ![example](/assets/images/ranger_show_config-1112dfa7140bc9972ed4628a26a8ec09.png) The following figure shows the added service. ![added service](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAc8AAABxCAIAAAA8rb/NAAAeNUlEQVR4nOzdCXxTZb438OdkT5qtSZOugZaW7itQKAUKLes4oMg24yiXUWHuzMdRX2eucx2v+qqvjuOIMzLX915H9M4rgisqssjSUlm0QAuFtpS2dEv3Nc2+Nct5P+1TDjFt06QkRcr/++HD5/TkyXmenLS/PHnOc85hkCSJAAAABBJJkrTb3QYAAJjmcKcW0hYAAAKIGj+AtAUAgEBxHaqFtAUAgIBwOyoGaQsAAFMB0hYAAPxv9HQvSFsAAPCzMWfWQtoCAEDAwXxbAADws/FOGYO0BQCAwIKzGwAAwM88XAsB0hYAAAIIziUDAAA/89CxhaNkAAAQQHDmLgAATB04SgYAAH4z4bXCIW0BAGAqMEavam7ruh0tAQCAO5hr3zZGEU6todaPkbYIoUGTbqpaCAAA08KNsGXyBGM+DiMJAABwy7y4vyOkLQAATAVIWwAACBRq0BbObgAAgCkCaQsAAFMB0hYAAKYCpC0AAAQcQRCQtgAAMBUgbQEAYCqMfS6ZBw31DWUXy7RaretKkUg0b9682bNn+7VtAIA7ld1mN5lMBEEIhGOfWOXTplQDKtugbbwCDAZDKpUyWUyfNmsymew2+3iPMllMLpfrY0sn4Fvatra2Fp0sQggJBAKCIEiSxP9rtdqTJ09yOGyFYoZfmkU6SY1GYzAYWGyWUCDk8vz8sgOFRKoBldlsDpGGcLicCUuaTCaxSCwQCBAx+TrtNrtWqzUajUFBQUKh0NffOQD8zmF37Pton9lsRgilpaUtWrRo0pvq6e45dvwY3pQHLBZr5cqVCoXCm22q+lWFhYUarcZzMalUunLFSnGw2Jf2usMJiZd9S9uzZ88ihB74+QMisch1vU6r++jjj86cOfvggw/eSsuwq1VXL5ResNlufpTJ5fIli5fI5DKD3rB3314Pz01ISMjPz0cIlXxfUllViT/3Hn3kUYJ2M8+KTxZfr78++rl8Pj81NTUzIxNnn5fFMIPeUFRU1N3TTa1hMpnZ87LTM9Ldnj5myYU5C5OTk/EGzSbzB3s+wC3fvn07LuOwOw4eOtjT04Pfv02bNkmlUrvNfu78uerqatftx8XGLVq06I75fALT0eDgIJWParV68hsiUWFh4YRRi2ssKira+tBWBnOCTHM6nMeOH9Pr9RNuU6VSnSg8sWXLFl9a/AOuUetz2ur1eg6H4xa1CCGhSMhgMPR6fYuyxe0h3P/lC/hSqdSbKi6WXbx46aLbyt7e3i+/+nLzps0sFsvz050O53Ct6FrNNbzGbrcrlcqYWTE3mzTOKc0Gg+H8+fMajWbZsmXeFxvq8re0Hj121O3qljabreRcibJFec9P7qF+A8Yreebsme6e7oKCgqGX4HTi9Q6Hg3pRhw4doqJ2w4YNQzuTRIcPH3ZNbayhsaGnt2fL5i3QyQV3Op1OZzAaPBSIjo7u7e01mUwIIavVqtaoZTKZ522qNerRUSsWi9PS0giXDtT1+uvd3d0DAwMGvYEv4E/cVmLiSyX4PG5LEGN/6Q0NDe3o6Dh67Oh4T0xPT8/NzfW8cbvNjqOWIIjVq1bLQ+Vmk/lC6YXW1laSJMsuli0vWL5gwQJcWD2gxn1PFouVlZWFV4ZIQxBCSqXSbr85InO1+qpr2lLiZ8fPmjVr6H0atLYoW5qamxBCtbW12fOyg/hBXhYzGoxUgIaFhSUlJXE53N6+3kuXLpEk2dnZef78+cVLFiOERpfk8XgdHR1XrlwZenevX1coFKPHvp0O5+EjI6nKYDDuv/9+/LnV2dmJV/L5/NWrVvP5fLVaXfxtscFg0Ov19Q31Q51lAO5kVIdjTHPmzJk/f75Oqzvw9QEcuA67p/LYmOO/PC4vJTnF9dsqnUHv7u7GnaFJtn4U39LWrWPsavny5Y0NjTabjUb7wTwHEpGkk2xsaqysrJTJZJ6PpKk1I186OBzOjBkzaHQaj8fLX5Z/+PBhvOsZTAYVrO3t7ThtBXwBtRK7Wn0VL7BYrMHBwY6ODrPZPHrMWy6XR8dE4+WE+IT33n8PZ7RWp3VNW8/FTp85jfdJTEzM6lWr8Rs2Y+aM2NjYTz/9FDcmOTlZIpWMWVKhUEiCJcXfFiOEqqur3fYP6SSPfHOks7MTDzhs3LCRGkXq7evFC+Fh4TL50Oc5l8fNy8u7cP4CHo7wsJ8BuNPhqB36Yi0USiQSnLaT1tnVWVxcPPTl8haOoIwJd0/xH77Pfdvx8Hi8tPS08R5NS0t7/3/er6io8Jy2weJgvGA2m/d8uCc5KXnmzJmhoaGbt2z2viUWs6W9vX1oa8HBSYlJJedKEEJ1dXWZmZkenmUfhpdlIeN+GRldrKtr5OLr8+bNc32rgoODo6KicEu6e7olUsl4JePi4oKCgkhE0ml0t+qOfHOko6MDR+3mTZuFIiH1EDUyU99QPzAwkJiYqFAoZgzzckcB4C8Ws+Xz/Z8bjcbRD7W3t7/zzjvUj0lJSUuXLr3F6qioRST67vvv8F/ZLWKxJxilvHV+S1vP8BjihH1yBpORnp5eWVk59P5ZLOWXy8svlxMEoVAosrOzJxyRwerq6vBCclLyrFmzcNpevXp1dNq2trXio2dmk7mhsQGvDAkJcRvx9FDMNmijXpRU4j4wHR4ejn8PBgYGPJSk0WmRUZGjXwhJktSvEZ1O5/F4ro9GRkRKJVLVgAohpBpQfV/yPQ7l+Pj4uXPm8oJ43uwrAPxCo9GMGbWjtba2TmL7ISEhQbygltaW0VF79erVSWzQTWpq6uJFi0f6QCSaZA93nKFbakhgitKWdJI4CyYsmbswVygQnr9wnupCkiTZOmz58uXeTOmtulqFF+Li4rg8bnBwsFqtNhgMqn6VNOQHMYc367pGLBbfd+99bhv0UMxiseA1DAZj9DvE5YyMXRiNRuq46pglJ2SxWM6cOVOwvIBaQ6PT7rvvvgulF1znJNhsturq6rq6uo0bNgZLgn2uBoAAG+/Aj2epqamJCYnFxcV8AT+gUVtaWmrQGwoKCia8paNPcODeatraBm1VVVWNTY1Go5EYL0WIkWGL/v7+Dz74YCgpaLTo6OjsedljTEolUGpaampqand3t7JF2djYSB1APHny5KyYWXSG+9dtV319fQaDAY/89qv6kWooGfEclOpr1Xl5eW67AH8ADA4O4jUajebatWsZmRleFgsKGhnetdvtpJN0nWeGENIbRlouEon4fL6Hkp0dnXgXuXVyCYKIj4/HvfXr9dfj4uJmzLw5UMBis5YsWbIwZ2Fbe1trS2tjUyNuod1uP1l8ctOmTR52FAB+JJVKIyMjqe6tw+Gg/mzpdLpAcPMEh/jZ8ZPY/lBGE+jmuCqJvvvuO+rwDGUSEekWteXl5Xh6g9HkVVfdS35IW51W99VXX5ktZhyg4xUjSZJGo4lEIqfTaTAYcMXV1dUqlWr9+vWuJRsbGru6u/DgTlh4WFh4WE5OTk93z4GvD+D92NvbGx4R7qFJ16pHJn5ZLJYjR464PlRTU7N40WIa/WY7F+UuSk1LxZ8Z5ZfLL1++jBAqu1jmlrYeitHoNC6Xi/utyhZlTMwPZj40NIwMO0ilUg8luzq7Dh46iI/pPfLII65buHfdveHh4eoBNT4mdqLwxEMPPoQ/oi5dvIT3fM6CnJhhS5ctra6uxnOi+/v7HXaH508mAPyFyWKuW7eO+pGaM47H09auXeufajxG7SSMGbUIITxkMckWjhP4BEFMPm2dDueXX31psVhyF+YmJib6Osa8Z88ePMHClUqlwl8NCIKgzj8JDQsNlYfi2U52x7hn2uEm1dbVjvcoSZJKpXJW7KzRDzFZzHlz5125coUkSbvdbjaZxzw7YMxisbGxuM0lJSUR4RFsDhsXrq2ppT7eIyOHeqzjlbx85TJemPHDM/EIgsAfLStXrtz30T7caS06WYR/d5uVzf39/fhZVIc3JTkFp+3kPucBuCNQ3zJvRVRU1JhRGyCTSVtq7v3BQwctFsviRYtxv88nTY1NJpMJZ5CroXAZTp6qqioGgxEXF0cjaE3NTdQcfs8HypqamnDEhIaGrlm9hlpfU1NTWlaKJ2ONmbZ4ep1EIlGpVLhfPN65WKOLzc+eX11dTZKkXq//5NNP8Czarq6uxsZG/JQ5c+bgA1yjS/L5/BZlCzUonJo69p4UCAW5C3Px4b729vba2trExMSIiAicticKT+Tm5oaHh9tt9isVV0aeIhBMeFINAHek4SEFg8HQ2dV5K5sZGBjQ6XRCkdDPUTv+uLRvf5AkSeIRzLKysu7u7oT4hElEbV9f34nCE3Q6fcWKFW4PKRSKuNg4fNz/8jDXR7PnZXM4ni4+QA2ZpySnuMZlUlISTtvOzk4PZwEKBAIcoyaTycMhJrdiLDZr9arVx08cJ0nSbDa7vW1hoWFz58zFy55Lpqenh4WHjVdpenp6TW0NHoA+ffq0IkqRPS+7qanJYDDY7fYzZ864lS/ILxh/PwFwZ6COq9fU1LieuUAicswJtt58w6aG10wm04GvD8jlcqVS6ak83T/DcT7fl4wgCD6f39zUfOnSJalEml+Q72uVtkHbgQMHEELr1q0b8xI7BQUFOTk5brMXWCxWXl7e3Llzx2kWwlN0qS6w2/gpl8cNCQnBy/X19eO1TS6T44W2tjYPL2F0seiY6Ad/8WBYWJhbm3Nzc9evX+86eDpmSSaTuXz58glOtCPQ6lWr8SJJksdPHGeymBs3bhx9wphUIr133b2eR7cBCCgud+QvjiCI0V9hvccX8CUSCUKou7v7zNkz1L+zZ8+OvqaMQCCgJux7ECwOZrNHBvFMJpPnqOUH8YVCoa/NHj3vAn/nHuPcsOa2rkGTbsyt7P1wr8VqsdvtBEH8ctsvqZFH732x/4u+/r68vLwJzislkdFo1Oq0dDpdKBT6/dJngUIitUZtMVskEskEO+dGSZFY5DaR1lcOu0Ov1xuNRg6XIxTANcDAtKJRawqLCvG3SQ/EIvHKlSvdpniOp62trbCwcMLBXy6Xu2b1mtCwUF/aO4TFE0ZH3exRURnrY9ru3YvnV224f6M81KtzDVydPn26pqYmJSVlyZIlvj4XAHDXMpvNnq5vy2T42mWxDdoGBgY8nG/FZDGlEunkDn6weMIYRTiO1slfA4zNZhsMhsyMTF+j1ulwXrx0saamRhYig6gFAPiEO8yPG2SymJPotPqEIAhqTgHmW9pmZWYVnSyqqKzAs2K9mWOEZ9f29vbiSf5uE2wBAGBaGh2PvqVt3Oy4wcHB8svl+Fqr3uNyuEnJSdnzst3OoQIAgOln5LDYDy+a6Nu4LQAAAM/GO0oG99wFAAB/cuvCUpe4hbQFAICpAGkLAAD+M/5VaSBtAQAggGDcFgAA/G38ObEwbgsAAAHkesQM0hYAAPzBY8cW0hYAAALFbSrY2OeSsXg+X2QMAADuWh4uY0A9NHbaRoZ6deEyAAAAbmnb0TP29SFhJAEAAPzP7YqLvt1z11lcRJZeQPV15MBAwFp4ByAkEhSfQFuQSyzz+dYVAIDpx8s7rnqVtqRK5dz1JllVecutmg6GPmzOn3OcP0ecLqY98TsieOKbcwAAgFcjCc6dr0PUjkZeuex8a+ftbgUA4EfKrc87cdo6C4+TtdcC2aQ7GFlZ4Swuut2tAAD8uLjlLP7Ri5GE8yXUIrFwEf0XW1FUVEAaeKdob3d89CF57nv8E3muBBW436odAHCX8HLQ1qszd521NdQy/YEH7/aoRQhFRQ3tB0pTw+1sDADgDuFF39ZkurmsmBHQ1twxXPbDXT5DA0wPVqtVq9WazWYve2oTIgiCy+WKRCI22/1W/1NZ12BLm/7DLyxVdcjh8EtdiE7npCUItm5kzVT49DzfZoABAKYlq9Xa3d0tk8nCw8PpdLpftulwOPR6fXd3d1hYmGsITmVdgy1tfX/4k99y9kZllivXLFV1sr886zlwR3+WwNkNANzttFqtTCYTi8X+ir/hLiBdLBbLZDKtVnu76tJ/+IWfo5bicAxt3EeQtgDc7cxms0AgCMSWBQKB2Wy+XXVZquoCUdGkN37HpO3FNlVlp/p2twKAaYgkST/2NF3R6fTRc6GmrK5AdWwnu3E/pG15+8D6D7679e14tr+irbi+x9dnmQYdOW8XtWtMPj3rnxcaXz8JU4wBAP7kh7S12Bw1/QZ/NMb/SJKs01msdt8+hfRWe5/RGrBGAQDuOgRB+DYnYdDheKO4Zk9le6/Vvi5a+uo96Vc7NS8V1/Ra7WveO/3XtRkRIt7/OVF9uKGn12q/Z6b0jbWZESLuc99ULImRnWnuPd+u/s97s/7ybe2vcmJf+7bmNzmxs2XC14uvfdHYlyHhbc2csW1+LING2B3Od0rq91a01eksm+Jkz69ImSHhU20wWGz//k2FmMN4eU3GFxWtf/2+vk5nWSAXPFeQtHiW3LW1LQOGR/aXIYQe3V/2eE7cxswZ39Z3//Xs9e+6dVtmy59bmaIQB/XoLC8cr/ysvlfOZmxKCnt+Zdrfz9a9f7m112q/2qs7tn2p//Y2AOAuQhCEV1cTH8/Ra517Ktu/+EUOj0n/08lrfyq69vKatO3zop8urn1tTZpCHPR68bUrPZrPH1jAZTIe/7p836XmpwuSm9Wmz2sq18aFPp+fZLY5v27ur1UZHl8YGyvlr93z/YaEsNIdS1sGDI8drhhKxpy4d0rq3y9v+fvazGAea+ep2l99eYlKPa3Ztu2zCyEc5rNrMzs0xn89WrVvfVZ6RPD+itZ1n5T2/fs9DPrN3rqMz31pRcq6T0p/vzh+rkJ6uX1gw+cXP75/zlsywQdlzfd+8H3pb5f/7UyN1mov/9eleqt92+dlSfLWzRmKHr21z2h5Ki/BT7sdAOCt6upqpVLpuobJZGZnZ7PZ7NbW1sTExEBUyp2bxoqOQghZrzdbqmpvfYNuUUsQhM8jCWwGvddqr+/TCznMN+7NeiovQcbnJIeK5GxGVpREwGH+NCniH/fPSwkTc5l0MYepMo18H88KFb55X9ay2WH4x/+8N+vBuTGVnVoeg/bST9JmywQrEsKfXTL77QtNQ4+WNv/v/KQlsfLUcPGff5rx6NwYu8OJEGrXmn+2rySIQf/vTfOCWAw2Y+ijorpba3M4fp07++zDi0lEtgwYLrcP4H8kSWZFShBCyWGiCBF3X3nLjrTIzMjgIBZjx8JYpXHwvLKfy2QoNaZ2jXmWlH9o2+Jls0NjpIKZwTw5n5MVJbn1nQ4A8N5777134sQJt5VWq9XpdO7fv/+xxx775ptv/F4pUxEueeQBvBzy24cZUrHfq8B869uuiA97Iz9xV0n9I4crMiS8Py5NjJf/4J46PBbjlaLqc52aGBFXY7FFi7l4fX6MzLVYZuTQ6+nUm9JkAtaNA5SJcpHSONhvtPZa7bNCRoYOwoTczVkjJ269f7VjTkjQxW6txmyT8emhQs6nG+a+e6HpT+eb5GzGEzmzkkJFey8p91/rxOU/+8XCCCGXqrRBbTzdodld1UGt6dCan8hLIBHafqAcD328sCIZiXzaJQDcLfR6/alTp5qbm41GY1hY2Pz581NSUvxbxblz59577z3cE6SYTKZXXnklOzt76dKlR48eveeee/xVneTRn7OiFfQQCY3F4mamDnVC2azQl/7NodZaaus1+w54s5HRIwbj8S1tu/Tm1Unh2xfG9egs715o+PlX5cromzFqd5JbPrnwWHbMu1uyWXT6a0XVWssgfkjA+UFFzOGEDQliXx8wOkmSNrxzO3VmOZsh5rIQQr16Cy7ZrjF9XK58cunQl/pHUyNf+2n6un9+99zRync2zddYBhVi3v5ti/QW24naru3fVKaFif5jZep/rEylKjJa7dSyQsh5MSbuybyRbyKVHWqFJKhLa3oyL+GFVakNffoXTlz9tyMVhx7J82mfAHA3KCoqevvtt+12+/r16zUazd69ez/55JOUlJSXXnrJj/NnSZIcHbW7du3icDhvv/32H/7wh/nz5/urLoSQ7uvjEW+9TBcOtZ8hu3F7MEmwjcnU/+1dP1aEQ9m3kYRj17p++Vlpt84sE7AzI4b6pzQC0Qmi12pv15jMVnuv1R4l4tEIWmWH+v9VtJnsTg+hvzQutE5nebekwWJzXO/VvfxtzfasGQwa8Whq5M6z19u1pl695S+nanoMVtz/jRJxmXT639Zmflbfe7i6Xaky5P7P2SvtA3w2c44ieLgxhFsVNNrQmla10TToWJsY8V9lysvtAxab41B1++ZPLtAR8efimlcLq02D9hhpUJJMQB8e9qXTiS69pc9gmeR+BWB6OXbs2MGDB/Pz8zdu3KhUKk+fPo3XL1269JlnnnE7g8uPcNTabLbTp09HRERkZWUF+/Xi/fZ+NXI4Bju6et/6h+6bIsOZcz0v/9Wh1ZF2h2PgVl8UMcx1jW992wfmzCxtVyW9fRIhJGcz3v1JuoDDjJMLMiS8tP8qPvkvuS8sinv0SAU6UjEnJOhXc2e+UtK4Prl7zBwcTk/eoZ/Pf/zQlT+erkMIPZapeGzxUB/22RXJfzxSkfZ/ixFCW5PDn8yLv9H6of+TwkQvL5n9LwevXH9ixXO5sfl7Ri4I+cScGTkzQ9yq4DLpm+JkGz6/+OelCTty454aMBYMl18gF/z3ukwhl/ncypTffHVJ8eZxhFCCkPOP++cihLKjpH8/3xT/9yL1s2t928EATDuFhYWnTp16/fXXd+7caTKZzp07h9fb7faMjAy5XP7ss8++8cYbPB7Pv/WaTKbnn39erVa3tLRERES8+eabcrncv1WMcJLISZJO59A/hxM5nQGpxde05XOYu7cs+JvVZrE7xBwWngAg5bFP/boAF5gTJflVTpzdSYq4TITQ9gVxQg6zID6M2sJchcQ1whbPkl9+cpXaPChgMajpBBIe+x+b579lc5Ak4rFGRnX/vDaTetbjSxIeXzKUy79flvTbxfF6i43PZnKYY5+gsnvLgt03ln+9aPaOhXF6qw2PVyCE4mXCoh35WsugkySDuWwc6PNnSut+v8anPQPAdNXY2JiVlcXlcjMyMnbt2kWtDwkJiYyMFAgENpvNaDT6PW1379791FNPffzxxzab7bXXXnME9MSwW+Pl0O1kzm7gs5khQRzXuVaugtgMHLUIIRGXOVan1l0wlzV6a1wmnYpaD9gMegifM17UjkanEVTUYgSBxFyWhMf2pqkA3G127NjR0NDw6aefrl27ds2akV6IVCp94YUXNBrNc8899+STT8pkMr/Xq1Kp3nnnHZPJ9Oabb1ZVVZWXl/u9ikBzG0yAKy4CADyh0+nPPPPMzp07d+3a9bOf/Wzr1q1arTY8PLysrGzXrl2/+93v/DgHVigUajQasViMENq2bZtOp4uOjhaLxbW1tatWrfJXLZ4EsssFaQsAmACdTn/66adLS0t3797d2dlJkiSLxVqwYMGrr77q317tww8//OKLL+p0OteVBEEsXLjQ77PNMOegjRUZFrLjIcRgIBqNNyedLuDbVX67ABbu23p7XzIiOJhU36i7rRVu34DwfriBCIaTIMD0R6PRcoYFtJaMjIy33noroFW40R08Jly3GtFu9mmdRpP2kPsZFhPyMHRLPeRF3zZ6FlJfwouOj/fRH9qGIiJ8bcq00t7u+HjfzR9jY29nYwAAt0B36KTu0MlA14IDd+K0pa1a47g8krbkue/tN+41CzDinnW3uwkAgNtvwu7txHMSiJyFRHpGANo2HRDzF9Cy5tzuVgAAfhSI8Sc20Wg0r2aA0f7X74mMTL+2ajogMrPov3n8drcCAHBn8GpOAhEsob/4CnnqW+f5ElRfd5ff05uQSlF8Am3+QmJZ/u1uCwDgx8VD99aHGWDEsnw65AsA0w5BEA6HIxC3C3M4HG7pM5V1ITo9gLcm8/0l3DF3gQQABAiXy9Xr9YHYsl6v53K5t6suTloAbwjgYePjdW8hbQG424lEor6+Po1G48drETgcDo1G09fXJxKJblddgq0bJ9ED9QqdPrTx8Y0ZuGNMWWhu64oMlfq7dQCAHy+r1arVas1ms5cXxp4QQRBcLlckErHZ7NtY12BLm/7DLyxVdX4bUqDTOWkJgq0bWTMV4xXp6FHFKMJHvzpIWwAA8CecttQJuxQYSQAAgIBwG0+AtAUAgEBxDVxIWwAACCAqcCFtAQAgsHDgQtoCAEDA+XzPXQAAAJMDaQsAAFNh7OskdPSoprwlAAAwnXl1Y14AAAC3CEYSAABgKkDaAgDAVIC0BQCAqQBpCwAAUwHSFgAApgKkLQAATAVIWwAAmAqQtgAAMBX+fwAAAP//mYeA7H9mpYUAAAAASUVORK5CYII=) 5. Click **Test connection** to test the connectivity, and save it after the connection is successful. If you didn't install ranger-starrocks-plugin, then you can skip test connection and create directly. 6. On each FE machine of the StarRocks cluster, create [ranger-starrocks-security.xml](https://github.com/StarRocks/ranger/blob/master/plugin-starrocks/conf/ranger-starrocks-security.xml) in the `fe/conf` folder and copy the content. You must modify the following two parameters and save the modifications: * `ranger.plugin.starrocks.service.name`: Change to the name of the StarRocks Service you created in Step 4. * `ranger.plugin.starrocks.policy.rest the url`: Change to the address of the Ranger Admin. If you need to modify other configurations, refer to official documentation of Apache Ranger. For example, you can modify `ranger.plugin.starrocks.policy.pollIntervalMs` to change the interval for pulling policy changes. ```sql vim ranger-starrocks-security.xml ... ranger.plugin.starrocks.service.name starrocks -- Change it to the StarRocks Service name. Name of the Ranger service containing policies for this StarRocks instance ... ... ranger.plugin.starrocks.policy.rest.url http://localhost:6080 -- Change it to Ranger Admin address. URL to Ranger Admin ... ``` 7. (Optional) If you want to use the Audit Log service of Ranger, you need to create the [ranger-starrocks-audit.xml](https://github.com/StarRocks/starrocks/blob/main/conf/ranger/ranger-starrocks-audit.xml) file in the `fe/conf` folder of each FE machine. Copy the content, **replace `solr_url` in `xasecure.audit.solr.solr_url` with your own `solr_url`**, and save the file. 8. Add the configuration `access_control = ranger` to all FE configuration files. ```sql vim fe.conf access_control=ranger ``` 9. Restart all FE machines. ```sql -- Switch to the FE folder. cd.. bin/stop_fe.sh bin/start_fe.sh ``` #### Reuse other services to control access to external tables[​](#reuse-other-services-to-control-access-to-external-tables "Direct link to Reuse other services to control access to external tables") For External Catalog, you can reuse external services (such as Hive Service) for access control. StarRocks supports matching different Ranger external services for different Catalogs. When users access an external table, the system implements access control based on the access policy of the Ranger Service corresponding to the external table. The user permissions are consistent with the Ranger user with the same name. 1. Copy Hive's Ranger configuration files [ranger-hive-security.xml](https://github.com/StarRocks/ranger/blob/master/hive-agent/conf/ranger-hive-security.xml) and [ranger-hive-audit.xml](https://github.com/StarRocks/ranger/blob/master/hive-agent/conf/ranger-hive-audit.xml) to the `fe/conf` file of all FE machines. Make sure Ranger's IP and port are correct. 2. Restart all FE machines. 3. Configure External Catalog. * When you create an External Catalog, add the property `"ranger.plugin.hive.service.name"`. ```sql CREATE EXTERNAL CATALOG hive_catalog_1 PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "ranger.plugin.hive.service.name" = "" ) ``` * You can also add this property to an existing External Catalog. ```sql ALTER CATALOG hive_catalog_1 SET ("ranger.plugin.hive.service.name" = ""); ``` ​ This operation changes the authentication method of an existing Catalog to Ranger-based authentication. #### What to do next[​](#what-to-do-next "Direct link to What to do next") After adding a StarRocks Service, you can click the service to create access control policies for the service and assign different permissions to different users or user groups. When users access StarRocks data, access control will be implemented based on these policies. --- ### Manage user privileges This topic describes how to manage users, roles, and privileges in StarRocks. StarRocks employs both role-based access control (RBAC) and identity-based access control (IBAC) to manage privileges within a StarRocks cluster, allowing cluster administrators to easily restrict privileges within the cluster on different granular levels. Within a StarRocks cluster, privileges can be granted to users or roles. A role is a collection of privileges that can be assigned to users or other roles in the cluster as needed. A user can be granted one or more roles, which determine their permissions on different objects. #### View privilege and role information[​](#view-privilege-and-role-information "Direct link to View privilege and role information") Users with the system-defined role `user_admin` can view all the user and role information within the StarRocks cluster. ##### View privilege information[​](#view-privilege-information "Direct link to View privilege information") You can view the privileges granted to a user or a role using [SHOW GRANTS](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_GRANTS.md). * View the privileges of the current user. ```sql SHOW GRANTS; ``` > **NOTE** > > Any user can view their own privileges without needing any privileges. * View the privileges of a specific user. The following example shows the privileges of the user `jack`: ```sql SHOW GRANTS FOR jack@'172.10.1.10'; ``` * View the privileges of a specific role. The following example shows the privileges of the role `example_role`: ```sql SHOW GRANTS FOR ROLE example_role; ``` ##### View roles[​](#view-roles "Direct link to View roles") You can view all the roles within the StarRocks cluster using [SHOW ROLES](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_ROLES.md). ```sql SHOW ROLES; ``` #### Manage roles[​](#manage-roles "Direct link to Manage roles") Users with the system-defined role `user_admin` can create, grant, revoke, or drop roles in StarRocks. ##### Create a role[​](#create-a-role "Direct link to Create a role") You can create a role using [CREATE ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/CREATE_ROLE.md). By default, a user can have a maximum of 64 roles. You can adjust this setting by using the FE dynamic parameter `privilege_max_total_roles_per_user`. A role can have a maximum of 16 inheritance levels. You can adjust this setting by using the FE dynamic parameter `privilege_max_role_depth`. The following example creates the role `example_role`: ```sql CREATE ROLE example_role; ``` ##### Grant a role[​](#grant-a-role "Direct link to Grant a role") You can grant roles to a user or another role using [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md). * Grant a role to a user. The following example grants the role `example_role` to the user `jack`: ```sql GRANT example_role TO USER jack@'172.10.1.10'; ``` * Grant a role to another role. The following example grants the role `example_role` to the role `test_role`: ```sql GRANT example_role TO ROLE test_role; ``` * Grant a role to a user group. You can identify user groups from external authentication systems via [Group Provider](https://docs.starrocks.io/docs/administration/user_privs/group_provider.md). The following example grants the role `example_role` to the user group `analysts`: ```sql GRANT example_role TO EXTERNAL GROUP analysts; ``` ##### Alter the default role of a user[​](#alter-the-default-role-of-a-user "Direct link to Alter the default role of a user") The default role of a user is automatically activated when the user connects to StarRocks. For instructions on how to enable all (default and granted) roles for a user after connection, see [Enable all roles](#enable-all-roles). You can set the default role of a user using [SET DEFAULT ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SET_DEFAULT_ROLE.md) or [ALTER USER](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/ALTER_USER.md). Both of the following examples set the default role of `jack` to `db1_admin`. Note that `db1_admin` must have been assigned to `jack`. * Set the default role using SET DEFAULT ROLE: ```sql SET DEFAULT ROLE 'db1_admin' TO jack@'172.10.1.10'; ``` * Set the default role using ALTER USER: ```sql ALTER USER jack@'172.10.1.10' DEFAULT ROLE 'db1_admin'; ``` ##### Revoke a role[​](#revoke-a-role "Direct link to Revoke a role") You can revoke roles from a user or another role using [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE.md). > **NOTE** > > You cannot revoke the system-defined default role `PUBLIC` from a user. * Revoke a role from a user. The following example revokes the role `example_role` from the user `jack`: ```sql REVOKE example_role FROM USER jack@'172.10.1.10'; ``` * Revoke a role from another role. The following example revokes the role `example_role` from the role `test_role`: ```sql REVOKE example_role FROM ROLE test_role; ``` ##### Drop a role[​](#drop-a-role "Direct link to Drop a role") You can drop a role using [DROP ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/DROP_ROLE.md). The following example drops the role `example_role`: ```sql DROP ROLE example_role; ``` > **CAUTION** > > System-defined roles cannot be dropped. ##### Enable all roles[​](#enable-all-roles "Direct link to Enable all roles") The default roles of a user are roles that are automatically activated each time the user connects to the StarRocks cluster. If you want to enable all the roles (default and granted roles) for all StarRocks users when they connect to the StarRocks cluster, you can perform the following operation. This operation requires the system privilege OPERATE. ```sql SET GLOBAL activate_all_roles_on_login = TRUE; ``` You can also use SET ROLE to activate the roles assigned to you. For example, user `jack@'172.10.1.10'` has roles `db_admin` and `user_admin` but they are not default roles of the user and are not automatically activated when the user connects to StarRocks. If `jack@'172.10.1.10'` needs to activate `db_admin` and `user_admin`, he can run `SET ROLE db_admin, user_admin;`. Note that SET ROLE overwrites original roles. If you want to enable all your roles, run SET ROLE ALL. #### Manage privileges[​](#manage-privileges "Direct link to Manage privileges") Users with the system-defined role `user_admin` can grant or revoke privileges in StarRocks. ##### Grant privileges[​](#grant-privileges "Direct link to Grant privileges") You can grant privileges to a user or a role using [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md). * Grant a privilege to a user. The following example grants the SELECT privilege on the table `sr_member` to the user `jack`, and allows `jack` to grant this privilege to other users or roles (by specifying WITH GRANT OPTION in the SQL): ```sql GRANT SELECT ON TABLE sr_member TO USER jack@'172.10.1.10' WITH GRANT OPTION; ``` * Grant a privilege to a role. The following example grants the SELECT privilege on the table `sr_member` to the role `example_role`: ```sql GRANT SELECT ON TABLE sr_member TO ROLE example_role; ``` ##### Revoke privileges[​](#revoke-privileges "Direct link to Revoke privileges") You can revoke privileges from a user or a role using [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE.md). * Revoke a privilege from a user. The following example revokes the SELECT privilege on the table `sr_member` from the user `jack`, and disallows `jack` to grant this privilege to other users or roles: ```sql REVOKE SELECT ON TABLE sr_member FROM USER jack@'172.10.1.10'; ``` * Revoke a privilege from a role. The following example revokes the SELECT privilege on the table `sr_member` from the role `example_role`: ```sql REVOKE SELECT ON TABLE sr_member FROM ROLE example_role; ``` #### Best practices[​](#best-practices "Direct link to Best practices") ##### Multi-service access control[​](#multi-service-access-control "Direct link to Multi-service access control") Usually, a company-owned StarRocks cluster is managed by a sole service provider and maintains multiple lines of business (LOBs), each of which uses one or more databases. As shown below, a StarRocks cluster's users include members from the service provider and two LOBs (A and B). Each LOB is operated by two roles - analysts and executives. Analysts generate and analyze business statements, and executives query the statements. ![User Privileges](/assets/images/user_privilege_1-e144a82f732892eea1d207996c34a990.png) LOB A independently manages the database `DB_A`, and LOB B the database `DB_B`. LOB A and LOB B use different tables in `DB_C`. `DB_PUBLIC` can be accessed by all members of both LOBs. ![User Privileges](/assets/images/user_privilege_2-053e5c148bb7de7aa89561c05ac3222b.png) Because different members perform different operations on different databases and tables, we recommend you create roles in accordance with their services and positions, apply only the necessary privileges to each role, and assign these roles to corresponding members. As shown below: ![User Privileges](/assets/images/user_privilege_3-abd736675549c8da604cd515ea0c080f.png) 1. Assign the system-defined roles `db_admin`, `user_admin`, and `cluster_admin` to cluster maintainers, set `db_admin` and `user_admin` as their default roles for daily maintenance, and manually activate the role `cluster_admin` when they need to operate the nodes of the cluster. Example: ```sql GRANT db_admin, user_admin, cluster_admin TO USER user_platform; ALTER USER user_platform DEFAULT ROLE db_admin, user_admin; ``` 2. Create users for each member within the LOBs, and set complex passwords for each user. 3. Create roles for each position within the LOBs, and apply the corresponding privileges to each role. For the director of each LOB, grant their role the maximum collection of the privileges their LOBs need, and the corresponding GRANT privileges (by specifying WITH GRANT OPTION in the statement). Therefore, they can assign these privileges to the members of their LOB. Set the role as their default role if their daily work requires it. Example: ```sql GRANT SELECT, ALTER, INSERT, UPDATE, DELETE ON ALL TABLES IN DATABASE DB_A TO ROLE linea_admin WITH GRANT OPTION; GRANT SELECT, ALTER, INSERT, UPDATE, DELETE ON TABLE TABLE_C1, TABLE_C2, TABLE_C3 TO ROLE linea_admin WITH GRANT OPTION; GRANT linea_admin TO USER user_linea_admin; ALTER USER user_linea_admin DEFAULT ROLE linea_admin; ``` For analysts and executives, assign them the role with the corresponding privileges. Example: ```sql GRANT SELECT ON ALL TABLES IN DATABASE DB_A TO ROLE linea_query; GRANT SELECT ON TABLE TABLE_C1, TABLE_C2, TABLE_C3 TO ROLE linea_query; GRANT linea_query TO USER user_linea_salesa; GRANT linea_query TO USER user_linea_salesb; ALTER USER user_linea_salesa DEFAULT ROLE linea_query; ALTER USER user_linea_salesb DEFAULT ROLE linea_query; ``` 4. For the database `DB_PUBLIC`, which can be accessed by all cluster users, grant the SELECT privilege on `DB_PUBLIC` to the system-defined role `public`. Example: ```sql GRANT SELECT ON ALL TABLES IN DATABASE DB_PUBLIC TO ROLE public; ``` You can assign roles to others to achieve role inheritance in complicated scenarios. For example, if analysts require privileges to write into and query tables in `DB_PUBLIC`, and executives can only query these tables, you can create roles `public_analysis` and `public_sales`, apply relevant privileges to the roles, and assign them to the original roles of analysts and executives respectively. Example: ```sql CREATE ROLE public_analysis; CREATE ROLE public_sales; GRANT SELECT, ALTER, INSERT, UPDATE, DELETE ON ALL TABLES IN DATABASE DB_PUBLIC TO ROLE public_analysis; GRANT SELECT ON ALL TABLES IN DATABASE DB_PUBLIC TO ROLE public_sales; GRANT public_analysis TO ROLE linea_analysis; GRANT public_analysis TO ROLE lineb_analysis; GRANT public_sales TO ROLE linea_query; GRANT public_sales TO ROLE lineb_query; ``` ##### Customize roles based on scenarios[​](#customize-roles-based-on-scenarios "Direct link to Customize roles based on scenarios") We recommend you customize roles to manage privileges and users. The following examples classify a few combinations of privileges for some common scenarios. ###### Grant global read-only privileges on StarRocks tables[​](#grant-global-read-only-privileges-on-starrocks-tables "Direct link to Grant global read-only privileges on StarRocks tables") ```sql -- Create a role. CREATE ROLE read_only; -- Grant the USAGE privilege on all catalogs to the role. GRANT USAGE ON ALL CATALOGS TO ROLE read_only; -- Grant the privilege to query all tables to the role. GRANT SELECT ON ALL TABLES IN ALL DATABASES TO ROLE read_only; -- Grant the privilege to query all views to the role. GRANT SELECT ON ALL VIEWS IN ALL DATABASES TO ROLE read_only; -- Grant the privilege to query all materialized views and the privilege to accelerate queries with them to the role. GRANT SELECT ON ALL MATERIALIZED VIEWS IN ALL DATABASES TO ROLE read_only; ``` And you can further grant the privilege to use UDFs in queries: ```sql -- Grant the USAGE privilege on all database-level UDF to the role. GRANT USAGE ON ALL FUNCTIONS IN ALL DATABASES TO ROLE read_only; -- Grant the USAGE privilege on global UDF to the role. GRANT USAGE ON ALL GLOBAL FUNCTIONS TO ROLE read_only; ``` ###### Grant global write privileges on StarRocks tables[​](#grant-global-write-privileges-on-starrocks-tables "Direct link to Grant global write privileges on StarRocks tables") ```sql -- Create a role. CREATE ROLE write_only; -- Grant the USAGE privilege on all catalogs to the role. GRANT USAGE ON ALL CATALOGS TO ROLE write_only; -- Grant the INSERT and UPDATE privileges on all tables to the role. GRANT INSERT, UPDATE ON ALL TABLES IN ALL DATABASES TO ROLE write_only; -- Grant the REFRESH privilege on all materialized views to the role. GRANT REFRESH ON ALL MATERIALIZED VIEWS IN ALL DATABASES TO ROLE write_only; ``` ###### Grant read-only privileges on a specific external catalog[​](#grant-read-only-privileges-on-a-specific-external-catalog "Direct link to Grant read-only privileges on a specific external catalog") ```sql -- Create a role. CREATE ROLE read_catalog_only; -- Grant the USAGE privilege on the destination catalog to the role. GRANT USAGE ON CATALOG hive_catalog TO ROLE read_catalog_only; -- Switch to the corresponding catalog. SET CATALOG hive_catalog; -- Grant the privileges to query all tables and all views in the external catalog. GRANT SELECT ON ALL TABLES IN ALL DATABASES TO ROLE read_catalog_only; ``` tip For views in external catalogs, you can query only Hive table views (since v3.1). ###### Grant write-only privileges on a specific external catalog[​](#grant-write-only-privileges-on-a-specific-external-catalog "Direct link to Grant write-only privileges on a specific external catalog") You can only write data into Iceberg tables (since v3.1) and Hive tables (since v3.2). ```sql -- Create a role. CREATE ROLE write_catalog_only; -- Grant the USAGE privilege on the destination catalog to the role. GRANT USAGE ON CATALOG iceberg_catalog TO ROLE read_catalog_only; -- Switch to the corresponding catalog. SET CATALOG iceberg_catalog; -- Grant the privilege to write data into Iceberg tables. GRANT INSERT ON ALL TABLES IN ALL DATABASES TO ROLE write_catalog_only; ``` ###### Grant admin privileges on a specific database[​](#grant-admin-privileges-on-a-specific-database "Direct link to Grant admin privileges on a specific database") ```sql -- Create a role. CREATE ROLE db1_admin; -- Grant ALL privileges on the destination database to the role. This role can create tables, views, materialized views, and UDFs in this database. And it also can drop or modify this database. GRANT ALL ON DATABASE db1 TO ROLE db1_admin; -- Switch to the corresponding catalog. SET CATALOG iceberg_catalog; -- Grant all privileges on tables, views, materialized views, and UDFs in this database to the role. GRANT ALL ON ALL TABLES IN DATABASE db1 TO ROLE db1_admin; GRANT ALL ON ALL VIEWS IN DATABASE db1 TO ROLE db1_admin; GRANT ALL ON ALL MATERIALIZED VIEWS IN DATABASE db1 TO ROLE db1_admin; GRANT ALL ON ALL FUNCTIONS IN DATABASE db1 TO ROLE db1_admin; ``` ###### Grant privileges to perform backup and restore operations on global, database, table, and partition levels[​](#grant-privileges-to-perform-backup-and-restore-operations-on-global-database-table-and-partition-levels "Direct link to Grant privileges to perform backup and restore operations on global, database, table, and partition levels") * Grant privileges to perform global backup and restore operations: The privileges to perform global backup and restore operations allow the role to back up and restore any database, table, or partition. It requires the REPOSITORY privilege on the SYSTEM level, the privileges to create databases in the default catalog, to create tables in any database, and to load and export data on any table. ```sql -- Create a role. CREATE ROLE recover; -- Grant the REPOSITORY privilege on the SYSTEM level. GRANT REPOSITORY ON SYSTEM TO ROLE recover; -- Grant the privilege to create databases in the default catalog. GRANT CREATE DATABASE ON CATALOG default_catalog TO ROLE recover; -- Grant the privilege to create tables in any database. GRANT CREATE TABLE ON ALL DATABASES TO ROLE recover; -- Grant the privilege to load and export data on any table. GRANT INSERT, EXPORT ON ALL TABLES IN ALL DATABASES TO ROLE recover; ``` * Grant the privileges to perform database-level backup and restore operations: The privileges to perform database-level backup and restore operations require the REPOSITORY privilege on the SYSTEM level, the privilege to create databases in the default catalog, the privilege to create tables in any database, the privilege to load data into any table, and the privilege export data from any table in the database to be backed up. ```sql -- Create a role. CREATE ROLE recover_db; -- Grant the REPOSITORY privilege on the SYSTEM level. GRANT REPOSITORY ON SYSTEM TO ROLE recover_db; -- Grant the privilege to create databases. GRANT CREATE DATABASE ON CATALOG default_catalog TO ROLE recover_db; -- Grant the privilege to create tables. GRANT CREATE TABLE ON ALL DATABASES TO ROLE recover_db; -- Grant the privilege to load data into any table. GRANT INSERT ON ALL TABLES IN ALL DATABASES TO ROLE recover_db; -- Grant the privilege to export data from any table in the database to be backed up. GRANT EXPORT ON ALL TABLES IN DATABASE TO ROLE recover_db; ``` * Grant the privileges to perform table-level backup and restore operations: The privileges to perform table-level backup and restore operations require the REPOSITORY privilege on the SYSTEM level, the privilege to create tables in corresponding databases, the privilege to load data into any table in the database, and the privilege to export data from the table to be backed up. ```sql -- Create a role. CREATE ROLE recover_tbl; -- Grant the REPOSITORY privilege on the SYSTEM level. GRANT REPOSITORY ON SYSTEM TO ROLE recover_tbl; -- Grant the privilege to create tables in corresponding databases. GRANT CREATE TABLE ON DATABASE TO ROLE recover_tbl; -- Grant the privilege to load data into any table in a database. GRANT INSERT ON ALL TABLES IN DATABASE TO ROLE recover_db; -- Grant the privilege to export data from the table you want to back up. GRANT EXPORT ON TABLE TO ROLE recover_tbl; ``` * Grant the privileges to perform partition-level backup and restore operations: The privileges to perform partition-level backup and restore operations require the REPOSITORY privilege on the SYSTEM level, and the privilege to load and export data on the corresponding table. ```sql -- Create a role. CREATE ROLE recover_par; -- Grant the REPOSITORY privilege on the SYSTEM level. GRANT REPOSITORY ON SYSTEM TO ROLE recover_par; -- Grant the privilege to load and export data on the corresponding table. GRANT INSERT, EXPORT ON TABLE TO ROLE recover_par; ``` --- ### Overview of privileges This topic describes the basic concepts of StarRocks' privilege system. Privileges determine which users can perform which operations on which objects, so that you can more securely manage data and resources in a fine-grained manner. > NOTE: The privileges described in this topic are available only from v3.0. The privilege framework and syntax in v3.0 are not backward compatible with those in earlier versions. After an upgrade to v3.0, most of your original privileges are still retained except those for specific operations. For the detailed differences, see \[Upgrade notes] in [Privileges supported in StarRocks](https://docs.starrocks.io/docs/administration/user_privs/authorization/privilege_item.md). StarRocks adopts two privilege models: * Role-based access control (RBAC): Privileges are assigned to roles, which are then assigned to users. In this case, privileges are passed to users through roles. * Identity-based access control (IBAC): Privileges are directly assigned to user identities. Therefore, the maximum privilege scope of each user identity is the union of its own privileges and the privileges of the roles assigned to this user identity. **Basic concepts** for understanding StarRocks' privilege system: * **Object**: An entity to which access can be granted. Unless allowed by a grant, access is denied. Examples of objects include CATALOG, DATABASE, TABLE, and VIEW. For more information, see [Privileges supported in StarRocks](https://docs.starrocks.io/docs/administration/user_privs/authorization/privilege_item.md). * **Privilege**: A defined level of access to an object. Multiple privileges can be used to control the granularity of access granted on an object. Privileges are object-specific. Different objects may have different privileges. Examples of privileges include SELECT, ALTER, and DROP. * **User identity**: the unique identity of a user and also the entity to which privileges can be granted. User identity is represented as `username@'userhost'`, consisting of username and the IP from which the user logs in. Use identity simplifies attribute configuration. User identities that share the same user name share the same attribute. If you configure an attribute for a username, this attribute takes effect on all user identities that share this username. * **Role**: An entity to which privileges can be granted. Roles are an abstract collection of privileges. Roles can in turn assigned to users. Roles can also be assigned to other roles, creating a role hierarchy. To facilitate data management, StarRocks provides system-defined roles. To allow for more flexibility, you can also create custom roles according to business requirements. The following figure shows an example of privilege management under the RBAC and IBAC privilege models. In the models, access to objects is allowed through privileges assigned to roles and users. Roles are in turn assigned to other roles or users. ![privilege management](/assets/images/privilege-manage-1b5789075b4868ca896d1acc014df68b.png) #### Objects and privileges[​](#objects-and-privileges "Direct link to Objects and privileges") Objects have a logical hierarchy, which is related to the concept they represent. For example, Database is contained in Catalog, and Table, View, Materialized View, and Function are contained in Database. The following figure shows the object hierarchy in the StarRocks system. ![privilege objects](/assets/images/privilege-object-c7b962dd11136a98e64759615847e4c9.png) Each object has a set of privilege items that can be granted. These privileges define which operations can be performed on these objects. You can grant and revoke privileges from roles or users through the [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) and [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE.md) commands. #### Users[​](#users "Direct link to Users") ##### User identity[​](#user-identity "Direct link to User identity") In StarRocks, each user is identified by a unique user ID. It consists of the IP address (user host) and username, in the format of `username @'userhost'`. StarRocks identifies users with the same username but from different IP addresses as different user identities. For example, `user1@'172.65.xx.1'` and `172.65.xx.2'` are two user identities. Another representation of user identity is `username @['domain']`, where `domain` is a domain name that can be resolved by DNS as a set of IP addresses. `username @['domain']` is finally represented as a set of `username@'userhost'`. You can use `%` for the `userhost` part for fuzzy match. If `userhost` is not specified, it defaults to `'%'`, which means users of the same name logged in from any host. ##### Grant privileges to users[​](#grant-privileges-to-users "Direct link to Grant privileges to users") Users are entities to which privileges can be granted. Both privileges and roles can be assigned to users. The maximum privilege scope of each user identity is the union of its own privileges and the privileges of the roles assigned to this user identity. StarRocks ensures that each user can only perform authorized operations. We recommend that you **use roles to pass privileges** in most cases. For example, after you create a role, you can grant privileges to the role and then assign the role to users. If you want to grant temporary or special privileges, you can directly grant them to users. This simplifies privilege management and offers flexibility. #### Roles[​](#roles "Direct link to Roles") Roles are the entities to which privileges can be granted and revoked. Roles can be seen as a collection of privileges that can be assigned to users, to allow them to perform required actions. A user can be assigned multiple roles so they can perform different actions using separate sets of privileges. To simplify management, StarRocks recommends that you **manage privileges through roles**. Special and temporary privileges can be directly granted to users. To facilitate management, StarRocks provides several **system-defined roles** with specific privileges, which helps you meet daily management and maintenance requirements. You can also flexibly **customize roles** to meet specific business needs and security needs. Note that the privilege scope of system-defined roles cannot be modified. After a role is activated, users can perform operations that are authorized by the role. You can set **default roles** that are automatically activated when the user logs in. Users can also manually activate a role owned by this used in the current session. ##### System-defined roles[​](#system-defined-roles "Direct link to System-defined roles") StarRocks provides several types of system-defined roles. ![roles](/assets/images/privilege-role-bd6646cc406da9c1e890bb5f108a3764.png) * `root`: has global privileges. By default, the `root` user has the `root` role. After a StarRocks cluster is created, the system automatically generates a root user with root privileges. Because the root user and role have all privileges of the system, we recommend that you create new users and roles for subsequent operations to prevent any risky operations. Keep the password of the root user properly. * `cluster_admin`: has cluster management privileges to perform node-related operations, such as adding or dropping nodes. `cluster_admin` has the privileges to add, drop, and decommission cluster nodes. We recommend that you do not assign `cluster_admin` or any custom roles that contain this role as a default role to any user, to prevent unexpected node changes. * `db_admin`: has database management privileges, including the privileges to perform all operations on catalog, database, table, view, materialized view, function, global function, resource group, and plug-ins. * `user_admin`: has administrative privileges on users and roles, including privileges to create users, roles, and privileges. The above system-defined roles are designed to aggregate complex database privileges to facilitate your daily management. **The privilege scope of the above roles cannot be modified.** In addition, if you need to grant specific privileges to all users, StarRocks also provides a system-defined role `public`. * `public`: This role is owned by any user and activated by default in any session, including adding new users. The `public` role does not have any privileges by default. You can modify the privileges scope of this role. ##### Custom roles[​](#custom-roles "Direct link to Custom roles") You can create custom roles to meet specific business requirements and modify their privilege scope. At the same time, for the convenience of management, you can assign roles to other roles to create privilege hierarchy and inheritance. Then, the privileges associated with a role are inherited by another role. ###### Role hierarchy and privilege inheritance[​](#role-hierarchy-and-privilege-inheritance "Direct link to Role hierarchy and privilege inheritance") The following figure shows an example of privilege inheritance. > Note: The maximum number of inheritance levels for a role is 16 by default. The inheritance relationship cannot be bidirectional. ![role inheritance](/assets/images/privilege-role_inheri-5cf6cbc58151a23afa650d0057df6e54.png) As shown in the figure: * `role_s` is assigned to `role_p`. `role_p` implicitly inherits `priv_1` of `role_s`. * `role_p` is assigned to `role_g`, `role_g` implicitly inherits `priv_2` of `role_p` and `priv_1` of `role_s`. * After a role is assigned to a user, the user also has the privileges of this role. ##### Active roles[​](#active-roles "Direct link to Active roles") Active roles allow users to apply the privileges of the role under the current session. You can use `SELECT CURRENT_ROLE();` to view active roles in the current session. For more information, see [current\_role](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/current_role.md). ###### Default roles[​](#default-roles "Direct link to Default roles") Default roles are automatically activated when the user logs in to the cluster. It can be a role owned by one or more users. The administrator can set default roles using the `DEFAULT ROLE` keyword in [CREATE USER](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/CREATE_USER.md) and can change default roles using [ALTER USER](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/ALTER_USER.md). Users can also change their default roles using [SET DEFAULT ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SET_DEFAULT_ROLE.md). Default roles provide basic privilege protection for users. For example, User A has `role_query` and `role_delete`, which has query and delete privilege respectively. We recommend that you only use `role_query` as the default role to prevent data loss caused by high-risk operations such as `DELETE` or `TRUNCATE`. If you need to perform these operations, you can do it after manually setting active roles. A user who does not have a default role still has the `public` role, which is automatically activated after the user logs in to the cluster. ###### Manually activate roles[​](#manually-activate-roles "Direct link to Manually activate roles") In addition to default roles, users can also manually activate one or more existing roles within a session. You can use [SHOW GRANTS](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_GRANTS.md) to view the privileges and roles that can be activated, and use [SET ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SET_ROLE.md) to configure active roles that are effective in the current session. Note that the SET ROLE command overwrites each other. For example, after a user logs in, the `default_role` is activated by default. Then the user runs `SET ROLE role_s`. At this time, the user has only the privileges of `role_s` and their own privileges. `default_role` is overwritten. #### References[​](#references "Direct link to References") * [Privileges supported by StarRocks](https://docs.starrocks.io/docs/administration/user_privs/authorization/privilege_item.md) * [Manage user privileges](https://docs.starrocks.io/docs/administration/user_privs/authorization/User_privilege.md) --- ### Authenticate User Groups Enable Group Provider in StarRocks to authenticate, and authorize user groups from external authentication systems. From v3.5.0 onwards, StarRocks supports Group Provider to collect group information from external authentication systems for user group management. #### Overview[​](#overview "Direct link to Overview") To deepen its integration with external user authentication and authorization systems, such as LDAP and Apache Ranger, StarRocks supports collecting user group information for a better experience on the collective user management. With a group provider, you can fetch the group information from external user systems for different purposes. Group information is independent and can be integrated flexibly into authentication, authorization, or other processes without being tightly coupled to any specific workflow. A Group Provider is essentially a mapping between users and groups. Any process that requires group information can query this mapping as needed. ##### Workflow[​](#workflow "Direct link to Workflow") The following flow chart uses LDAP and Apache Ranger as an example to explain the workflow of Group Provider. ![Group Provider](/assets/images/group_provider-cfcd33985a599de8f9403e8fa8357a19.png) #### Create a group provider[​](#create-a-group-provider "Direct link to Create a group provider") StarRocks supports three types of group providers: * **LDAP group provider**: Search and match users with groups in your LDAP service * **Unix group provider**: Search and match users with groups in your operating system * **File group provider**: Search and match users with groups defined by a file ##### Syntax[​](#syntax "Direct link to Syntax") * LDAP group provider: ```sql CREATE GROUP PROVIDER PROPERTIES ( "type" = "ldap", ldap_info, ldap_search_group_arg, ldap_search_attr, [ldap_cache_attr] ) ldap_info ::= "ldap_conn_url" = "", "ldap_bind_root_dn" = "", "ldap_bind_root_pwd" = "", "ldap_bind_base_dn" = "", ["ldap_conn_timeout" = "",] ["ldap_conn_read_timeout" = ""] ["ldap_ssl_conn_allow_insecure" = ""] ["ldap_ssl_conn_trust_store_path" = ""] ["ldap_ssl_conn_trust_store_pwd" = ""] ldap_search_group_arg ::= { "ldap_group_dn" = "" | "ldap_group_filter" = "" }, "ldap_group_identifier_attr" = "" ldap_search_user_arg ::= "ldap_group_member_attr" = "", "ldap_user_search_attr" = "" ldap_cache_arg ::= "ldap_cache_refresh_interval" = "" ``` * Unix group provider: ```sql CREATE GROUP PROVIDER PROPERTIES ( "type" = "unix" ) ``` * File group provider: ```sql CREATE GROUP PROVIDER PROPERTIES ( "type" = "file", "group_file_url" = "" ) ``` ##### Parameters[​](#parameters "Direct link to Parameters") ###### `type`[​](#type "Direct link to type") The type of the group provider to create. Valid values: * `ldap`: Creates an LDAP group provider. When this value is set, you need to specify `ldap_info`, `ldap_search_group_arg`, `ldap_search_user_arg`, and optionally `ldap_cache_arg`. * `unix`: Creates a Unix group provider. * `file`: Creates a File group provider. When this value is set, you need to specify `group_file_url`. ###### `group_file_url`[​](#group_file_url "Direct link to group_file_url") The URL or relative path (under `fe/conf`) to the file that defines the user groups. note A group file contains a list of groups and their members. You can define a group in each line where the group name and members are separated by a colon. Multiple users are separated by commas. Example: `group_name:user_1,user_2,user_3`. ###### `ldap_info` parameter group[​](#ldap_info-parameter-group "Direct link to ldap_info-parameter-group") The information used to connect to your LDAP service. ###### `ldap_conn_url`[​](#ldap_conn_url "Direct link to ldap_conn_url") The URL of your LDAP server. Format: `ldap://:)`. ###### `ldap_bind_root_dn`[​](#ldap_bind_root_dn "Direct link to ldap_bind_root_dn") The admin Distinguished Name (DN) of your LDAP service. ###### `ldap_bind_root_pwd`[​](#ldap_bind_root_pwd "Direct link to ldap_bind_root_pwd") The admin password of your LDAP service. ###### `ldap_bind_base_dn`[​](#ldap_bind_base_dn "Direct link to ldap_bind_base_dn") The base DN of the LDAP user for which the cluster searches. ###### `ldap_conn_timeout`[​](#ldap_conn_timeout "Direct link to ldap_conn_timeout") Optional. The timeout duration for the connection to your LDAP service. ###### `ldap_conn_read_timeout`[​](#ldap_conn_read_timeout "Direct link to ldap_conn_read_timeout") Optional. The timeout duration for the read operations in the connection to your LDAP service. ###### `ldap_ssl_conn_allow_insecure`[​](#ldap_ssl_conn_allow_insecure "Direct link to ldap_ssl_conn_allow_insecure") Optional. Whether to allow non-encrypted connections to the LDAP server. Default value: `true`. Setting this value to `false` indicates that SSL encryption is required to access LDAP. ###### `ldap_ssl_conn_trust_store_path`[​](#ldap_ssl_conn_trust_store_path "Direct link to ldap_ssl_conn_trust_store_path") Optional. Local path to store the SSL CA certificate of the LDAP server. Supports pem and jks formats. You do not need to set this item if the certificate is issued by a trusted organization. ###### `ldap_ssl_conn_trust_store_pwd`[​](#ldap_ssl_conn_trust_store_pwd "Direct link to ldap_ssl_conn_trust_store_pwd") Optional. The password used to access the locally stored SSL CA certificate of the LDAP server. pem-formatted certificates do not require a password. Only jsk-formatted certificates do. ###### `ldap_search_group_arg` parameter group[​](#ldap_search_group_arg-parameter-group "Direct link to ldap_search_group_arg-parameter-group") The arguments used to control how StarRocks searches for a group. note You can only specify either `ldap_group_dn` or `ldap_group_filter`. Specifying both is not supported. ###### `ldap_group_dn`[​](#ldap_group_dn "Direct link to ldap_group_dn") The DN of the group to be searched for. The group will be queried directly using this DN. Example: `"cn=ldapgroup1,ou=Group,dc=starrocks,dc=com;cn=ldapgroup2,ou=Group,dc=starrocks,dc=com"`. ###### `ldap_group_filter`[​](#ldap_group_filter "Direct link to ldap_group_filter") A customized group filter that can be recognized by the LDAP server. It will be sent directly to your LDAP server for searching for the group. Example: `(&(objectClass=groupOfNames)(cn=testgroup))`. ###### `ldap_group_identifier_attr`[​](#ldap_group_identifier_attr "Direct link to ldap_group_identifier_attr") The attribute used as the identifier for the group name. ###### `ldap_search_user_arg` parameter group[​](#ldap_search_user_arg-parameter-group "Direct link to ldap_search_user_arg-parameter-group") The arguments used to control how StarRocks identifies for a user in a group. ###### `ldap_group_member_attr`[​](#ldap_group_member_attr "Direct link to ldap_group_member_attr") The attribute that represents group members. Valid values: `member` and `memberUid`. ###### `ldap_user_search_attr`[​](#ldap_user_search_attr "Direct link to ldap_user_search_attr") Specifies how to extract the user identifier from the member attribute value. You can explicitly define an attribute (for example, `cn` or `uid`) or use a regular expression. note **DN Matching Mechanism** * **When `ldap_user_search_attr` is configured**, the system extracts the specified value from group member DNs and uses it as usernames, and uses login username as key during group search. * **When `ldap_user_search_attr` is not configured**, the system uses the complete DN directly as user identifier, and uses the DN recorded during authentication as key during group search. This design enables LDAP Group Provider to adapt to different LDAP environments, especially complex environments like Microsoft AD. tip **Interaction with `authentication_ldap_simple_bind_dn_pattern`** When using DN pattern authentication (e.g., `uid=${USER}@abc.com,ou=People,dc=example,dc=com`) where the `${USER}` substitution is embedded within a larger attribute value: * **Recommended**: Do not configure `ldap_user_search_attr`. The system will use the complete DN for group matching, which avoids extraction errors. * **If you must configure it**: Use a regex that extracts only the username portion. For example, if the DN pattern is `uid=${USER}@abc.com,ou=People,dc=example,dc=com`, set `ldap_user_search_attr` to `uid=([^,@]+)@abc.com` to extract only the `${USER}` part. A simple `uid` or `uid=([^,]+)` would incorrectly extract `alice@abc.com` instead of `alice`. ###### `ldap_cache_arg` parameter group[​](#ldap_cache_arg-parameter-group "Direct link to ldap_cache_arg-parameter-group") The argument used to define the cache behavior for the LDAP group information. ###### `ldap_cache_refresh_interval`[​](#ldap_cache_refresh_interval "Direct link to ldap_cache_refresh_interval") Optional. The interval at which StarRocks automatically refreshes the cached LDAP group information. Unit: Seconds. Default: `900`. ##### Example[​](#example "Direct link to Example") Suppose an LDAP server contains the following group and member information. ```plain -- Group information # testgroup, Group, starrocks.com dn: cn=testgroup,ou=Group,dc=starrocks,dc=com objectClass: groupOfNames cn: testgroup member: uid=test,ou=people,dc=starrocks,dc=com member: uid=tom,ou=people,dc=starrocks,dc=com -- User information # test, People, starrocks.com dn: cn=test,ou=People,dc=starrocks,dc=com objectClass: inetOrgPerson cn: test uid: test sn: FTE userPassword:: ``` Create a group provider `ldap_group_provider` for members in `testgroup`: ```sql CREATE GROUP PROVIDER ldap_group_provider PROPERTIES( "type"="ldap", "ldap_conn_url"="ldap://xxxx:xxx", "ldap_bind_root_dn"="cn=admin,dc=starrocks,dc=com", "ldap_bind_root_pwd"="123456", "ldap_bind_base_dn"="dc=starrocks,dc=com", "ldap_group_filter"="(&(objectClass=groupOfNames)(cn=testgroup))", "ldap_group_identifier_attr"="cn", "ldap_group_member_attr"="member", "ldap_user_search_attr"="uid=([^,]+)" ) ``` The above example uses `ldap_group_filter` to search for a group with the `groupOfNames` objectClass and a `cn` of `testgroup`. Therefore, `cn` is specified in `ldap_group_identifier_attr` to identify the group. `ldap_group_member_attr` is set to `member` so that the `member` attribute is used in the `groupOfNames` objectClass to identify members. `ldap_user_search_attr` is set to an expression `uid=([^,]+)`, which is used to identify users in the `member` attribute. ##### Microsoft AD Environment Example[​](#microsoft-ad-environment-example "Direct link to Microsoft AD Environment Example") Suppose a Microsoft AD server contains the following group and member information: ```plain -- Group information # ADGroup, Groups, company.com dn: CN=ADGroup,OU=Groups,DC=company,DC=com objectClass: group cn: ADGroup member: CN=John Doe,OU=Users,DC=company,DC=com member: CN=Jane Smith,OU=Users,DC=company,DC=com -- User information # John Doe, Users, company.com dn: CN=John Doe,OU=Users,DC=company,DC=com objectClass: user cn: John Doe sAMAccountName: johndoe ``` Create a Group Provider for Microsoft AD environment: ```sql CREATE GROUP PROVIDER ad_group_provider PROPERTIES( "type"="ldap", "ldap_conn_url"="ldap://ad.company.com:389", "ldap_bind_root_dn"="CN=admin,OU=Users,DC=company,DC=com", "ldap_bind_root_pwd"="password", "ldap_bind_base_dn"="DC=company,DC=com", "ldap_group_filter"="(&(objectClass=group)(cn=ADGroup))", "ldap_group_identifier_attr"="cn", "ldap_group_member_attr"="member" -- Note: Do not configure ldap_user_search_attr, system will use complete DN for matching ) ``` In this example, since `ldap_user_search_attr` is not configured, the system will: 1. During group cache construction, directly use the complete DN (for example, `CN=John Doe,OU=Users,DC=company,DC=com`) as user identifier. 2. During group search, use the DN recorded during authentication as key to search user's groups. This approach is particularly suitable for Microsoft AD environments, as group members in AD may lack simple username attributes. #### Combine group provider with a security integration[​](#combine-group-provider-with-a-security-integration "Direct link to Combine group provider with a security integration") After creating the group provider, you can combine it with a security integration to allow users specified by the group provider to log in to StarRocks. For more information on creating a security integration, see [Authenticate with Security Integration](https://docs.starrocks.io/docs/administration/user_privs/authentication/security_integration.md). ##### Syntax[​](#syntax-1 "Direct link to Syntax") ```sql ALTER SECURITY INTEGRATION SET ( "group_provider" = "", "permitted_groups" = "" ) ``` ##### Parameters[​](#parameters-1 "Direct link to Parameters") ###### `group_provider`[​](#group_provider "Direct link to group_provider") The name of the group provider(s) to be combined with the security integration. Multiple group providers are separated by commas. Once set, StarRocks will record the user's group information under each specified provider upon login. ###### `permitted_groups`[​](#permitted_groups "Direct link to permitted_groups") Optional. The name of group(s) whose members are allowed to log in to StarRocks. Multiple groups are separated by commas. Make sure that the specified groups can be retrieved by the combined group provider(s). ##### Example[​](#example-1 "Direct link to Example") ```sql ALTER SECURITY INTEGRATION LDAP SET ( "group_provider"="ldap_group_provider", "permitted_groups"="testgroup" ); ``` #### Grant role to a user group[​](#grant-role-to-a-user-group "Direct link to Grant role to a user group") You can grant roles to a user group via [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md). The following example grants the role `example_role` to the user group `analysts`: ```sql GRANT example_role TO EXTERNAL GROUP analysts; ``` #### Combine group provider with external authorization system (Apache Ranger)[​](#combine-group-provider-with-external-authorization-system-apache-ranger "Direct link to Combine group provider with external authorization system (Apache Ranger)") Once you configure the associated group provider in the security integration, StarRocks will record the user's group information upon login. This group information will then be automatically included in the authorization process with Ranger, eliminating the need for additional configuration. For more instructions on integrating StarRocks with Ranger, see [Manage permissions with Apache Ranger](https://docs.starrocks.io/docs/administration/user_privs/authorization/ranger_plugin.md). --- ### SSL Authentication From v3.4.1 onwards, StarRocks supports secure connections encrypted by SSL. Unlike the traditional cleartext connections to DBMS, SSL connections provide endpoint verification and data encryption to ensure that the data transmitted between clients and StarRocks cannot be read by unauthorized users. #### Enable SSL authentication[​](#enable-ssl-authentication "Direct link to Enable SSL authentication") ##### Configure FE nodes[​](#configure-fe-nodes "Direct link to Configure FE nodes") To enable SSL authentication in StarRocks, configure the following parameters in the FE configuration file **fe.conf**: * `ssl_keystore_location`: Specifies the path to the keystore file that stores the SSL certificate and key. * `ssl_keystore_password`: The password for accessing the keystore file. StarRocks requires this password to read the keystore file. * `ssl_key_password`: The password for accessing the key. StarRocks requires this password to retrieve the key from the keystore. * `ssl_force_secure_transport`: Whether to force SSL authentication. Default value: `FALSE`. If this item is set to `TRUE`, the system will reject connections that are not encrypted with SSL. Example: ```properties ssl_keystore_location = // Path to the keystore file ssl_keystore_password = // Password for the keystore file ssl_key_password = // Password for accessing the key ``` ##### Generate SSL certificate[​](#generate-ssl-certificate "Direct link to Generate SSL certificate") In a production environment, it is recommended to use certificates provided by certificate authorities. In a development environment, you can generate a custom SSL certificate. Use the following command to generate an SSL certificate: ```bash keytool -genkeypair -alias starrocks \ -keypass \ -keyalg RSA -keysize 1024 -validity 365 \ -keystore \ -storepass ``` Parameters: * `-keypass`: The key password, corresponding to `ssl_key_password` in **fe.conf**. * `-storepass`: The keystore file password, corresponding to `ssl_keystore_password` in **fe.conf**. * `-keystore`: The storage path of the keystore file, corresponding to `ssl_keystore_location` in **fe.conf**. ##### Enable SSL on client[​](#enable-ssl-on-client "Direct link to Enable SSL on client") StarRocks is compatible with the MySQL protocol. For MySQL clients, SSL authentication is enabled by default. For JDBC connections, add the following options: ```properties useSSL=true verifyServerCertificate=false ``` #### Disable SSL authentication[​](#disable-ssl-authentication "Direct link to Disable SSL authentication") To disable SSL authentication, follow these steps: * **MySQL client**: Add the option `--ssl-mode=DISABLED`. * **JDBC**: Remove `useSSL=true` and `verifyServerCertificate=false`. #### LDAP authentication[​](#ldap-authentication "Direct link to LDAP authentication") See [Authentication methods](https://docs.starrocks.io/docs/administration/user_privs/authentication/ldap_authentication.md) for detailed instructions on enabling LDAP authentication. For JDBC connections, since StarRocks supports SSL authentication, you do not need to customize `AuthPlugin`. You can use the built-in `MysqlClearPasswordPlugin`. * When using JDBC 5 with LDAP authentication, configure the following settings: ```properties authenticationPlugins: com.mysql.jdbc.authentication.MysqlClearPasswordPlugin defaultAuthenticationPlugin: com.mysql.jdbc.authentication.MysqlClearPasswordPlugin disabledAuthenticationPlugins: com.mysql.jdbc.authentication.MysqlNativePasswordPlugin ``` * When using JDBC 8 with LDAP authentication, configure the following settings: ```properties authenticationPlugins: com.mysql.cj.protocol.a.authentication.MysqlClearPasswordPlugin defaultAuthenticationPlugin: com.mysql.cj.protocol.a.authentication.MysqlClearPasswordPlugin disabledAuthenticationPlugins: com.mysql.cj.protocol.a.authentication.MysqlNativePasswordPlugin ``` #### FAQ[​](#faq "Direct link to FAQ") ###### Q1: When I connect to StarRocks using DBeaver, an error is returned "Unable to load authentication plugin 'mysql\_native\_password'"[​](#q1-when-i-connect-to-starrocks-using-dbeaver-an-error-is-returned-unable-to-load-authentication-plugin-mysql_native_password "Direct link to Q1: When I connect to StarRocks using DBeaver, an error is returned \"Unable to load authentication plugin 'mysql_native_password'\"") A: You need to upgrade JDBC 5 to version 5.1.46 or later. --- ## Benchmarking ### SSB Flat-table Benchmarking Star schema benchmark (SSB) is designed to test basic performance metrics of OLAP database products. SSB uses a star schema test set that is widely applied in academia and industry. For more information, see [Star Schema Benchmark](https://www.cs.umb.edu/~poneil/StarSchemaB.PDF). ClickHouse flattens the star schema into a wide flat table and rewrites the SSB into a single-table benchmark. For more information, see [Star schema benchmark of ClickHouse](https://clickhouse.com/docs/getting-started/example-datasets/star-schema) This test compares the performance of StarRocks, Apache Druid, and ClickHouse against SSB single-table datasets. #### Test conclusions[​](#test-conclusions "Direct link to Test conclusions") The test is performed on an OLAP table in a shared-nothing StarRocks cluster, together with ClickHouse and Apache Druid, against the same dataset. Based on the results from 13 queries performed on the 100 GB SSB-Flat dataset, StarRocks has an overall query performance **1.87x that of ClickHouse and 4.75x that of Apache Druid**. The unit of the results are milliseconds. ![SSB-SR](/assets/images/SSB-SR-a6741eb2b92ecf5dedf472be518c29dd.png) #### Test preparation[​](#test-preparation "Direct link to Test preparation") ##### Hardware[​](#hardware "Direct link to Hardware") StarRocks, Apache Druid, and ClickHouse are deployed on hosts of the same configurations - [AWS m7i.4xlarge](https://aws.amazon.com/ec2/instance-types/m7i/?nc1=h_ls). | | **Spec** | | ------------------------ | ---------- | | Instance Number | 5 | | vCPU | 16 | | Memory (GiB) | 64 | | Network Bandwidth (Gbps) | Up to 12.5 | | EBS Bandwidth (Gbps) | Up to 10 | ##### Software[​](#software "Direct link to Software") | | **StarRocks** | **ClickHouse** | **Apache Druid** | | ----------------- | ----------------- | -------------- | ------------------------------------------------------------ | | **Cluster Size** | One FE, Three BEs | Three nodes | One Master Server, one Query Servers, and three Data Servers | | **Version** | 3.5.0 | 25.3.3.42 | 33.0.0 | | **Release Date** | 2025.6.13 | 2025.4.22 | 2025.4.29 | | **Configuration** | Default | Default | Default | #### Test results[​](#test-results "Direct link to Test results") The following table shows the performance test results on 13 queries. The unit of query latency is milliseconds. All queries are warmed up 1 time and then executed 3 times to take the average value as the result. `ClickHouse vs StarRocks` and `Druid vs StarRocks` in the table header means using the query response time of ClickHouse/Druid to divide the query response time of StarRocks. A larger value indicates better performance of StarRocks. | Query | StarRocks | ClickHouse | Druid | ClickHouse vs StarRocks | Druid vs StarRocks | | ----- | --------- | ---------- | ----- | ----------------------- | ------------------ | | SUM | 992 | 1858 | 4710 | 1.87 | 4.75 | | Q01 | 30 | 49 | 330 | 1.63 | 11.00 | | Q02 | 16 | 31 | 260 | 1.94 | 16.25 | | Q03 | 26 | 29 | 250 | 1.12 | 9.62 | | Q04 | 143 | 197 | 420 | 1.38 | 2.94 | | Q05 | 120 | 179 | 440 | 1.49 | 3.67 | | Q06 | 63 | 158 | 320 | 2.51 | 5.08 | | Q07 | 133 | 249 | 510 | 1.87 | 3.83 | | Q08 | 90 | 197 | 380 | 2.19 | 4.22 | | Q09 | 86 | 150 | 350 | 1.74 | 4.07 | | Q10 | 20 | 33 | 250 | 1.65 | 12.50 | | Q11 | 156 | 340 | 550 | 2.18 | 3.53 | | Q12 | 66 | 133 | 330 | 2.02 | 5.00 | | Q13 | 43 | 113 | 320 | 2.63 | 7.44 | --- ### TPC-DS test SQL ```sql -- query 1 with customer_total_return as (select sr_customer_sk as ctr_customer_sk ,sr_store_sk as ctr_store_sk ,sum(SR_RETURN_AMT) as ctr_total_return from store_returns ,date_dim where sr_returned_date_sk = d_date_sk and d_year =2000 group by sr_customer_sk ,sr_store_sk) select c_customer_id from customer_total_return ctr1 ,store ,customer where ctr1.ctr_total_return > (select avg(ctr_total_return)*1.2 from customer_total_return ctr2 where ctr1.ctr_store_sk = ctr2.ctr_store_sk) and s_store_sk = ctr1.ctr_store_sk and s_state = 'TN' and ctr1.ctr_customer_sk = c_customer_sk order by c_customer_id limit 100; -- query 2 with wscs as (select sold_date_sk ,sales_price from (select ws_sold_date_sk sold_date_sk ,ws_ext_sales_price sales_price from web_sales union all select cs_sold_date_sk sold_date_sk ,cs_ext_sales_price sales_price from catalog_sales) t), wswscs as (select d_week_seq, sum(case when (d_day_name='Sunday') then sales_price else null end) sun_sales, sum(case when (d_day_name='Monday') then sales_price else null end) mon_sales, sum(case when (d_day_name='Tuesday') then sales_price else null end) tue_sales, sum(case when (d_day_name='Wednesday') then sales_price else null end) wed_sales, sum(case when (d_day_name='Thursday') then sales_price else null end) thu_sales, sum(case when (d_day_name='Friday') then sales_price else null end) fri_sales, sum(case when (d_day_name='Saturday') then sales_price else null end) sat_sales from wscs ,date_dim where d_date_sk = sold_date_sk group by d_week_seq) select d_week_seq1 ,round(sun_sales1/sun_sales2,2) ,round(mon_sales1/mon_sales2,2) ,round(tue_sales1/tue_sales2,2) ,round(wed_sales1/wed_sales2,2) ,round(thu_sales1/thu_sales2,2) ,round(fri_sales1/fri_sales2,2) ,round(sat_sales1/sat_sales2,2) from (select wswscs.d_week_seq d_week_seq1 ,sun_sales sun_sales1 ,mon_sales mon_sales1 ,tue_sales tue_sales1 ,wed_sales wed_sales1 ,thu_sales thu_sales1 ,fri_sales fri_sales1 ,sat_sales sat_sales1 from wswscs,date_dim where date_dim.d_week_seq = wswscs.d_week_seq and d_year = 2001) y, (select wswscs.d_week_seq d_week_seq2 ,sun_sales sun_sales2 ,mon_sales mon_sales2 ,tue_sales tue_sales2 ,wed_sales wed_sales2 ,thu_sales thu_sales2 ,fri_sales fri_sales2 ,sat_sales sat_sales2 from wswscs ,date_dim where date_dim.d_week_seq = wswscs.d_week_seq and d_year = 2001+1) z where d_week_seq1=d_week_seq2-53 order by d_week_seq1; -- query 3 select dt.d_year ,item.i_brand_id brand_id ,item.i_brand brand ,sum(ss_ext_sales_price) sum_agg from date_dim dt ,store_sales ,item where dt.d_date_sk = store_sales.ss_sold_date_sk and store_sales.ss_item_sk = item.i_item_sk and item.i_manufact_id = 128 and dt.d_moy=11 group by dt.d_year ,item.i_brand ,item.i_brand_id order by dt.d_year ,sum_agg desc ,brand_id limit 100; -- query 4 with year_total as ( select c_customer_id customer_id ,c_first_name customer_first_name ,c_last_name customer_last_name ,c_preferred_cust_flag customer_preferred_cust_flag ,c_birth_country customer_birth_country ,c_login customer_login ,c_email_address customer_email_address ,d_year dyear ,sum(((ss_ext_list_price-ss_ext_wholesale_cost-ss_ext_discount_amt)+ss_ext_sales_price)/2) year_total ,'s' sale_type from customer ,store_sales ,date_dim where c_customer_sk = ss_customer_sk and ss_sold_date_sk = d_date_sk group by c_customer_id ,c_first_name ,c_last_name ,c_preferred_cust_flag ,c_birth_country ,c_login ,c_email_address ,d_year union all select c_customer_id customer_id ,c_first_name customer_first_name ,c_last_name customer_last_name ,c_preferred_cust_flag customer_preferred_cust_flag ,c_birth_country customer_birth_country ,c_login customer_login ,c_email_address customer_email_address ,d_year dyear ,sum((((cs_ext_list_price-cs_ext_wholesale_cost-cs_ext_discount_amt)+cs_ext_sales_price)/2) ) year_total ,'c' sale_type from customer ,catalog_sales ,date_dim where c_customer_sk = cs_bill_customer_sk and cs_sold_date_sk = d_date_sk group by c_customer_id ,c_first_name ,c_last_name ,c_preferred_cust_flag ,c_birth_country ,c_login ,c_email_address ,d_year union all select c_customer_id customer_id ,c_first_name customer_first_name ,c_last_name customer_last_name ,c_preferred_cust_flag customer_preferred_cust_flag ,c_birth_country customer_birth_country ,c_login customer_login ,c_email_address customer_email_address ,d_year dyear ,sum((((ws_ext_list_price-ws_ext_wholesale_cost-ws_ext_discount_amt)+ws_ext_sales_price)/2) ) year_total ,'w' sale_type from customer ,web_sales ,date_dim where c_customer_sk = ws_bill_customer_sk and ws_sold_date_sk = d_date_sk group by c_customer_id ,c_first_name ,c_last_name ,c_preferred_cust_flag ,c_birth_country ,c_login ,c_email_address ,d_year ) select t_s_secyear.customer_id ,t_s_secyear.customer_first_name ,t_s_secyear.customer_last_name ,t_s_secyear.customer_preferred_cust_flag from year_total t_s_firstyear ,year_total t_s_secyear ,year_total t_c_firstyear ,year_total t_c_secyear ,year_total t_w_firstyear ,year_total t_w_secyear where t_s_secyear.customer_id = t_s_firstyear.customer_id and t_s_firstyear.customer_id = t_c_secyear.customer_id and t_s_firstyear.customer_id = t_c_firstyear.customer_id and t_s_firstyear.customer_id = t_w_firstyear.customer_id and t_s_firstyear.customer_id = t_w_secyear.customer_id and t_s_firstyear.sale_type = 's' and t_c_firstyear.sale_type = 'c' and t_w_firstyear.sale_type = 'w' and t_s_secyear.sale_type = 's' and t_c_secyear.sale_type = 'c' and t_w_secyear.sale_type = 'w' and t_s_firstyear.dyear = 2001 and t_s_secyear.dyear = 2001+1 and t_c_firstyear.dyear = 2001 and t_c_secyear.dyear = 2001+1 and t_w_firstyear.dyear = 2001 and t_w_secyear.dyear = 2001+1 and t_s_firstyear.year_total > 0 and t_c_firstyear.year_total > 0 and t_w_firstyear.year_total > 0 and case when t_c_firstyear.year_total > 0 then t_c_secyear.year_total / t_c_firstyear.year_total else null end > case when t_s_firstyear.year_total > 0 then t_s_secyear.year_total / t_s_firstyear.year_total else null end and case when t_c_firstyear.year_total > 0 then t_c_secyear.year_total / t_c_firstyear.year_total else null end > case when t_w_firstyear.year_total > 0 then t_w_secyear.year_total / t_w_firstyear.year_total else null end order by t_s_secyear.customer_id ,t_s_secyear.customer_first_name ,t_s_secyear.customer_last_name ,t_s_secyear.customer_preferred_cust_flag limit 100; -- query 5 with ssr as (select s_store_id, sum(sales_price) as sales, sum(profit) as profit, sum(return_amt) as returns, sum(net_loss) as profit_loss from ( select ss_store_sk as store_sk, ss_sold_date_sk as date_sk, ss_ext_sales_price as sales_price, ss_net_profit as profit, cast(0 as decimal(7,2)) as return_amt, cast(0 as decimal(7,2)) as net_loss from store_sales union all select sr_store_sk as store_sk, sr_returned_date_sk as date_sk, cast(0 as decimal(7,2)) as sales_price, cast(0 as decimal(7,2)) as profit, sr_return_amt as return_amt, sr_net_loss as net_loss from store_returns ) salesreturns, date_dim, store where date_sk = d_date_sk and d_date between cast('2000-08-23' as date) and date_add(cast('2000-08-23' as date), 14) and store_sk = s_store_sk group by s_store_id) , csr as (select cp_catalog_page_id, sum(sales_price) as sales, sum(profit) as profit, sum(return_amt) as returns, sum(net_loss) as profit_loss from ( select cs_catalog_page_sk as page_sk, cs_sold_date_sk as date_sk, cs_ext_sales_price as sales_price, cs_net_profit as profit, cast(0 as decimal(7,2)) as return_amt, cast(0 as decimal(7,2)) as net_loss from catalog_sales union all select cr_catalog_page_sk as page_sk, cr_returned_date_sk as date_sk, cast(0 as decimal(7,2)) as sales_price, cast(0 as decimal(7,2)) as profit, cr_return_amount as return_amt, cr_net_loss as net_loss from catalog_returns ) salesreturns, date_dim, catalog_page where date_sk = d_date_sk and d_date between cast('2000-08-23' as date) and date_add(cast('2000-08-23' as date), 14) and page_sk = cp_catalog_page_sk group by cp_catalog_page_id) , wsr as (select web_site_id, sum(sales_price) as sales, sum(profit) as profit, sum(return_amt) as returns, sum(net_loss) as profit_loss from ( select ws_web_site_sk as wsr_web_site_sk, ws_sold_date_sk as date_sk, ws_ext_sales_price as sales_price, ws_net_profit as profit, cast(0 as decimal(7,2)) as return_amt, cast(0 as decimal(7,2)) as net_loss from web_sales union all select ws_web_site_sk as wsr_web_site_sk, wr_returned_date_sk as date_sk, cast(0 as decimal(7,2)) as sales_price, cast(0 as decimal(7,2)) as profit, wr_return_amt as return_amt, wr_net_loss as net_loss from web_returns left outer join web_sales on ( wr_item_sk = ws_item_sk and wr_order_number = ws_order_number) ) salesreturns, date_dim, web_site where date_sk = d_date_sk and d_date between cast('2000-08-23' as date) and date_add(cast('2000-08-23' as date), 14) and wsr_web_site_sk = web_site_sk group by web_site_id) select channel , id , sum(sales) as sales , sum(returns) as returns , sum(profit) as profit from (select 'store channel' as channel , 'store' || s_store_id as id , sales , returns , (profit - profit_loss) as profit from ssr union all select 'catalog channel' as channel , 'catalog_page' || cp_catalog_page_id as id , sales , returns , (profit - profit_loss) as profit from csr union all select 'web channel' as channel , 'web_site' || web_site_id as id , sales , returns , (profit - profit_loss) as profit from wsr ) x group by rollup (channel, id) order by channel ,id limit 100; -- query 6 select a.ca_state state, count(*) cnt from customer_address a ,customer c ,store_sales s ,date_dim d ,item i where a.ca_address_sk = c.c_current_addr_sk and c.c_customer_sk = s.ss_customer_sk and s.ss_sold_date_sk = d.d_date_sk and s.ss_item_sk = i.i_item_sk and d.d_month_seq = (select distinct (d_month_seq) from date_dim where d_year = 2001 and d_moy = 1 ) and i.i_current_price > 1.2 * (select avg(j.i_current_price) from item j where j.i_category = i.i_category) group by a.ca_state having count(*) >= 10 order by cnt, a.ca_state limit 100; -- query 7 select i_item_id, avg(ss_quantity) agg1, avg(ss_list_price) agg2, avg(ss_coupon_amt) agg3, avg(ss_sales_price) agg4 from store_sales, customer_demographics, date_dim, item, promotion where ss_sold_date_sk = d_date_sk and ss_item_sk = i_item_sk and ss_cdemo_sk = cd_demo_sk and ss_promo_sk = p_promo_sk and cd_gender = 'M' and cd_marital_status = 'S' and cd_education_status = 'College' and (p_channel_email = 'N' or p_channel_event = 'N') and d_year = 2000 group by i_item_id order by i_item_id limit 100; -- query 8 select s_store_name ,sum(ss_net_profit) from store_sales ,date_dim ,store, (select ca_zip from ( SELECT substr(ca_zip,1,5) ca_zip FROM customer_address WHERE substr(ca_zip,1,5) IN ( '24128','76232','65084','87816','83926','77556','20548','26231','43848','15126', '91137','61265','98294','25782','17920','18426','98235','40081','84093','28577', '55565','17183','54601','67897','22752','86284','18376','38607','45200','21756', '29741','96765','23932','89360','29839','25989','28898','91068','72550','10390', '18845','47770','82636','41367','76638','86198','81312','37126','39192','88424', '72175','81426','53672','10445','42666','66864','66708','41248','48583','82276', '18842','78890','49448','14089','38122','34425','79077','19849','43285','39861', '66162','77610','13695','99543','83444','83041','12305','57665','68341','25003', '57834','62878','49130','81096','18840','27700','23470','50412','21195','16021', '76107','71954','68309','18119','98359','64544','10336','86379','27068','39736', '98569','28915','24206','56529','57647','54917','42961','91110','63981','14922', '36420','23006','67467','32754','30903','20260','31671','51798','72325','85816', '68621','13955','36446','41766','68806','16725','15146','22744','35850','88086', '51649','18270','52867','39972','96976','63792','11376','94898','13595','10516', '90225','58943','39371','94945','28587','96576','57855','28488','26105','83933', '25858','34322','44438','73171','30122','34102','22685','71256','78451','54364', '13354','45375','40558','56458','28286','45266','47305','69399','83921','26233', '11101','15371','69913','35942','15882','25631','24610','44165','99076','33786', '70738','26653','14328','72305','62496','22152','10144','64147','48425','14663', '21076','18799','30450','63089','81019','68893','24996','51200','51211','45692', '92712','70466','79994','22437','25280','38935','71791','73134','56571','14060', '19505','72425','56575','74351','68786','51650','20004','18383','76614','11634', '18906','15765','41368','73241','76698','78567','97189','28545','76231','75691', '22246','51061','90578','56691','68014','51103','94167','57047','14867','73520', '15734','63435','25733','35474','24676','94627','53535','17879','15559','53268', '59166','11928','59402','33282','45721','43933','68101','33515','36634','71286', '19736','58058','55253','67473','41918','19515','36495','19430','22351','77191', '91393','49156','50298','87501','18652','53179','18767','63193','23968','65164', '68880','21286','72823','58470','67301','13394','31016','70372','67030','40604', '24317','45748','39127','26065','77721','31029','31880','60576','24671','45549', '13376','50016','33123','19769','22927','97789','46081','72151','15723','46136', '51949','68100','96888','64528','14171','79777','28709','11489','25103','32213', '78668','22245','15798','27156','37930','62971','21337','51622','67853','10567', '38415','15455','58263','42029','60279','37125','56240','88190','50308','26859', '64457','89091','82136','62377','36233','63837','58078','17043','30010','60099', '28810','98025','29178','87343','73273','30469','64034','39516','86057','21309', '90257','67875','40162','11356','73650','61810','72013','30431','22461','19512', '13375','55307','30625','83849','68908','26689','96451','38193','46820','88885', '84935','69035','83144','47537','56616','94983','48033','69952','25486','61547', '27385','61860','58048','56910','16807','17871','35258','31387','35458','35576') intersect select ca_zip from (SELECT substr(ca_zip,1,5) ca_zip,count(*) cnt FROM customer_address, customer WHERE ca_address_sk = c_current_addr_sk and c_preferred_cust_flag='Y' group by ca_zip having count(*) > 10)A1)A2) V1 where ss_store_sk = s_store_sk and ss_sold_date_sk = d_date_sk and d_qoy = 2 and d_year = 1998 and (substr(s_zip,1,2) = substr(V1.ca_zip,1,2)) group by s_store_name order by s_store_name limit 100; -- query 9 select case when (select count(*) from store_sales where ss_quantity between 1 and 20) > 74219 then (select avg(ss_ext_discount_amt) from store_sales where ss_quantity between 1 and 20) else (select avg(ss_net_paid) from store_sales where ss_quantity between 1 and 20) end bucket1 , case when (select count(*) from store_sales where ss_quantity between 21 and 40) > 122840 then (select avg(ss_ext_discount_amt) from store_sales where ss_quantity between 21 and 40) else (select avg(ss_net_paid) from store_sales where ss_quantity between 21 and 40) end bucket2, case when (select count(*) from store_sales where ss_quantity between 41 and 60) > 56580 then (select avg(ss_ext_discount_amt) from store_sales where ss_quantity between 41 and 60) else (select avg(ss_net_paid) from store_sales where ss_quantity between 41 and 60) end bucket3, case when (select count(*) from store_sales where ss_quantity between 61 and 80) > 10097 then (select avg(ss_ext_discount_amt) from store_sales where ss_quantity between 61 and 80) else (select avg(ss_net_paid) from store_sales where ss_quantity between 61 and 80) end bucket4, case when (select count(*) from store_sales where ss_quantity between 81 and 100) > 165306 then (select avg(ss_ext_discount_amt) from store_sales where ss_quantity between 81 and 100) else (select avg(ss_net_paid) from store_sales where ss_quantity between 81 and 100) end bucket5 from reason where r_reason_sk = 1 ; -- query 10 select cd_gender, cd_marital_status, cd_education_status, count(*) cnt1, cd_purchase_estimate, count(*) cnt2, cd_credit_rating, count(*) cnt3, cd_dep_count, count(*) cnt4, cd_dep_employed_count, count(*) cnt5, cd_dep_college_count, count(*) cnt6 from customer c,customer_address ca,customer_demographics where c.c_current_addr_sk = ca.ca_address_sk and ca_county in ('Rush County','Toole County','Jefferson County','Dona Ana County','La Porte County') and cd_demo_sk = c.c_current_cdemo_sk and exists (select * from store_sales,date_dim where c.c_customer_sk = ss_customer_sk and ss_sold_date_sk = d_date_sk and d_year = 2002 and d_moy between 1 and 1+3) and exists (select * from ( select ws_bill_customer_sk as customer_sk, d_year,d_moy from web_sales, date_dim where ws_sold_date_sk = d_date_sk and d_year = 2002 and d_moy between 1 and 1+3 union all select cs_ship_customer_sk as customer_sk, d_year, d_moy from catalog_sales, date_dim where cs_sold_date_sk = d_date_sk and d_year = 2002 and d_moy between 1 and 1+3 ) x where c.c_customer_sk = customer_sk) group by cd_gender, cd_marital_status, cd_education_status, cd_purchase_estimate, cd_credit_rating, cd_dep_count, cd_dep_employed_count, cd_dep_college_count order by cd_gender, cd_marital_status, cd_education_status, cd_purchase_estimate, cd_credit_rating, cd_dep_count, cd_dep_employed_count, cd_dep_college_count limit 100; -- query 11 with year_total as ( select c_customer_id customer_id ,c_first_name customer_first_name ,c_last_name customer_last_name ,c_preferred_cust_flag customer_preferred_cust_flag ,c_birth_country customer_birth_country ,c_login customer_login ,c_email_address customer_email_address ,d_year dyear ,sum(ss_ext_list_price-ss_ext_discount_amt) year_total ,'s' sale_type from customer ,store_sales ,date_dim where c_customer_sk = ss_customer_sk and ss_sold_date_sk = d_date_sk group by c_customer_id ,c_first_name ,c_last_name ,c_preferred_cust_flag ,c_birth_country ,c_login ,c_email_address ,d_year union all select c_customer_id customer_id ,c_first_name customer_first_name ,c_last_name customer_last_name ,c_preferred_cust_flag customer_preferred_cust_flag ,c_birth_country customer_birth_country ,c_login customer_login ,c_email_address customer_email_address ,d_year dyear ,sum(ws_ext_list_price-ws_ext_discount_amt) year_total ,'w' sale_type from customer ,web_sales ,date_dim where c_customer_sk = ws_bill_customer_sk and ws_sold_date_sk = d_date_sk group by c_customer_id ,c_first_name ,c_last_name ,c_preferred_cust_flag ,c_birth_country ,c_login ,c_email_address ,d_year ) select t_s_secyear.customer_id ,t_s_secyear.customer_first_name ,t_s_secyear.customer_last_name ,t_s_secyear.customer_preferred_cust_flag from year_total t_s_firstyear ,year_total t_s_secyear ,year_total t_w_firstyear ,year_total t_w_secyear where t_s_secyear.customer_id = t_s_firstyear.customer_id and t_s_firstyear.customer_id = t_w_secyear.customer_id and t_s_firstyear.customer_id = t_w_firstyear.customer_id and t_s_firstyear.sale_type = 's' and t_w_firstyear.sale_type = 'w' and t_s_secyear.sale_type = 's' and t_w_secyear.sale_type = 'w' and t_s_firstyear.dyear = 2001 and t_s_secyear.dyear = 2001+1 and t_w_firstyear.dyear = 2001 and t_w_secyear.dyear = 2001+1 and t_s_firstyear.year_total > 0 and t_w_firstyear.year_total > 0 and case when t_w_firstyear.year_total > 0 then t_w_secyear.year_total / t_w_firstyear.year_total else 0.0 end > case when t_s_firstyear.year_total > 0 then t_s_secyear.year_total / t_s_firstyear.year_total else 0.0 end order by t_s_secyear.customer_id ,t_s_secyear.customer_first_name ,t_s_secyear.customer_last_name ,t_s_secyear.customer_preferred_cust_flag limit 100; -- query 12 select i_item_id ,i_item_desc ,i_category ,i_class ,i_current_price ,sum(ws_ext_sales_price) as itemrevenue ,sum(ws_ext_sales_price)*100/sum(sum(ws_ext_sales_price)) over (partition by i_class) as revenueratio from web_sales ,item ,date_dim where ws_item_sk = i_item_sk and i_category in ('Sports', 'Books', 'Home') and ws_sold_date_sk = d_date_sk and d_date between cast('1999-02-22' as date) and date_add(cast('1999-02-22' as date), 30) group by i_item_id ,i_item_desc ,i_category ,i_class ,i_current_price order by i_category ,i_class ,i_item_id ,i_item_desc ,revenueratio limit 100; -- query 13 select avg(ss_quantity) ,avg(ss_ext_sales_price) ,avg(ss_ext_wholesale_cost) ,sum(ss_ext_wholesale_cost) from store_sales ,store ,customer_demographics ,household_demographics ,customer_address ,date_dim where s_store_sk = ss_store_sk and ss_sold_date_sk = d_date_sk and d_year = 2001 and((ss_hdemo_sk=hd_demo_sk and cd_demo_sk = ss_cdemo_sk and cd_marital_status = 'M' and cd_education_status = 'Advanced Degree' and ss_sales_price between 100.00 and 150.00 and hd_dep_count = 3 )or (ss_hdemo_sk=hd_demo_sk and cd_demo_sk = ss_cdemo_sk and cd_marital_status = 'S' and cd_education_status = 'College' and ss_sales_price between 50.00 and 100.00 and hd_dep_count = 1 ) or (ss_hdemo_sk=hd_demo_sk and cd_demo_sk = ss_cdemo_sk and cd_marital_status = 'W' and cd_education_status = '2 yr Degree' and ss_sales_price between 150.00 and 200.00 and hd_dep_count = 1 )) and((ss_addr_sk = ca_address_sk and ca_country = 'United States' and ca_state in ('TX', 'OH', 'TX') and ss_net_profit between 100 and 200 ) or (ss_addr_sk = ca_address_sk and ca_country = 'United States' and ca_state in ('OR', 'NM', 'KY') and ss_net_profit between 150 and 300 ) or (ss_addr_sk = ca_address_sk and ca_country = 'United States' and ca_state in ('VA', 'TX', 'MS') and ss_net_profit between 50 and 250 )) ; -- query 14 with cross_items as (select i_item_sk ss_item_sk from item, (select iss.i_brand_id brand_id ,iss.i_class_id class_id ,iss.i_category_id category_id from store_sales ,item iss ,date_dim d1 where ss_item_sk = iss.i_item_sk and ss_sold_date_sk = d1.d_date_sk and d1.d_year between 1999 AND 1999 + 2 intersect select ics.i_brand_id ,ics.i_class_id ,ics.i_category_id from catalog_sales ,item ics ,date_dim d2 where cs_item_sk = ics.i_item_sk and cs_sold_date_sk = d2.d_date_sk and d2.d_year between 1999 AND 1999 + 2 intersect select iws.i_brand_id ,iws.i_class_id ,iws.i_category_id from web_sales ,item iws ,date_dim d3 where ws_item_sk = iws.i_item_sk and ws_sold_date_sk = d3.d_date_sk and d3.d_year between 1999 AND 1999 + 2) t where i_brand_id = brand_id and i_class_id = class_id and i_category_id = category_id ), avg_sales as (select avg(quantity*list_price) average_sales from (select ss_quantity quantity ,ss_list_price list_price from store_sales ,date_dim where ss_sold_date_sk = d_date_sk and d_year between 1999 and 1999 + 2 union all select cs_quantity quantity ,cs_list_price list_price from catalog_sales ,date_dim where cs_sold_date_sk = d_date_sk and d_year between 1999 and 1999 + 2 union all select ws_quantity quantity ,ws_list_price list_price from web_sales ,date_dim where ws_sold_date_sk = d_date_sk and d_year between 1999 and 1999 + 2) x) select channel, i_brand_id,i_class_id,i_category_id,sum(sales), sum(number_sales) from( select 'store' channel, i_brand_id,i_class_id ,i_category_id,sum(ss_quantity*ss_list_price) sales , count(*) number_sales from store_sales ,item ,date_dim where ss_item_sk in (select ss_item_sk from cross_items) and ss_item_sk = i_item_sk and ss_sold_date_sk = d_date_sk and d_year = 1999+2 and d_moy = 11 group by i_brand_id,i_class_id,i_category_id having sum(ss_quantity*ss_list_price) > (select average_sales from avg_sales) union all select 'catalog' channel, i_brand_id,i_class_id,i_category_id, sum(cs_quantity*cs_list_price) sales, count(*) number_sales from catalog_sales ,item ,date_dim where cs_item_sk in (select ss_item_sk from cross_items) and cs_item_sk = i_item_sk and cs_sold_date_sk = d_date_sk and d_year = 1999+2 and d_moy = 11 group by i_brand_id,i_class_id,i_category_id having sum(cs_quantity*cs_list_price) > (select average_sales from avg_sales) union all select 'web' channel, i_brand_id,i_class_id,i_category_id, sum(ws_quantity*ws_list_price) sales , count(*) number_sales from web_sales ,item ,date_dim where ws_item_sk in (select ss_item_sk from cross_items) and ws_item_sk = i_item_sk and ws_sold_date_sk = d_date_sk and d_year = 1999+2 and d_moy = 11 group by i_brand_id,i_class_id,i_category_id having sum(ws_quantity*ws_list_price) > (select average_sales from avg_sales) ) y group by rollup (channel, i_brand_id,i_class_id,i_category_id) order by channel,i_brand_id,i_class_id,i_category_id limit 100; with cross_items as (select i_item_sk ss_item_sk from item, (select iss.i_brand_id brand_id ,iss.i_class_id class_id ,iss.i_category_id category_id from store_sales ,item iss ,date_dim d1 where ss_item_sk = iss.i_item_sk and ss_sold_date_sk = d1.d_date_sk and d1.d_year between 1999 AND 1999 + 2 intersect select ics.i_brand_id ,ics.i_class_id ,ics.i_category_id from catalog_sales ,item ics ,date_dim d2 where cs_item_sk = ics.i_item_sk and cs_sold_date_sk = d2.d_date_sk and d2.d_year between 1999 AND 1999 + 2 intersect select iws.i_brand_id ,iws.i_class_id ,iws.i_category_id from web_sales ,item iws ,date_dim d3 where ws_item_sk = iws.i_item_sk and ws_sold_date_sk = d3.d_date_sk and d3.d_year between 1999 AND 1999 + 2) x where i_brand_id = brand_id and i_class_id = class_id and i_category_id = category_id ), avg_sales as (select avg(quantity*list_price) average_sales from (select ss_quantity quantity ,ss_list_price list_price from store_sales ,date_dim where ss_sold_date_sk = d_date_sk and d_year between 1999 and 1999 + 2 union all select cs_quantity quantity ,cs_list_price list_price from catalog_sales ,date_dim where cs_sold_date_sk = d_date_sk and d_year between 1999 and 1999 + 2 union all select ws_quantity quantity ,ws_list_price list_price from web_sales ,date_dim where ws_sold_date_sk = d_date_sk and d_year between 1999 and 1999 + 2) x) select this_year.channel ty_channel ,this_year.i_brand_id ty_brand ,this_year.i_class_id ty_class ,this_year.i_category_id ty_category ,this_year.sales ty_sales ,this_year.number_sales ty_number_sales ,last_year.channel ly_channel ,last_year.i_brand_id ly_brand ,last_year.i_class_id ly_class ,last_year.i_category_id ly_category ,last_year.sales ly_sales ,last_year.number_sales ly_number_sales from (select 'store' channel, i_brand_id,i_class_id,i_category_id ,sum(ss_quantity*ss_list_price) sales, count(*) number_sales from store_sales ,item ,date_dim where ss_item_sk in (select ss_item_sk from cross_items) and ss_item_sk = i_item_sk and ss_sold_date_sk = d_date_sk and d_week_seq = (select d_week_seq from date_dim where d_year = 1999 + 1 and d_moy = 12 and d_dom = 11) group by i_brand_id,i_class_id,i_category_id having sum(ss_quantity*ss_list_price) > (select average_sales from avg_sales)) this_year, (select 'store' channel, i_brand_id,i_class_id ,i_category_id, sum(ss_quantity*ss_list_price) sales, count(*) number_sales from store_sales ,item ,date_dim where ss_item_sk in (select ss_item_sk from cross_items) and ss_item_sk = i_item_sk and ss_sold_date_sk = d_date_sk and d_week_seq = (select d_week_seq from date_dim where d_year = 1999 and d_moy = 12 and d_dom = 11) group by i_brand_id,i_class_id,i_category_id having sum(ss_quantity*ss_list_price) > (select average_sales from avg_sales)) last_year where this_year.i_brand_id= last_year.i_brand_id and this_year.i_class_id = last_year.i_class_id and this_year.i_category_id = last_year.i_category_id order by this_year.channel, this_year.i_brand_id, this_year.i_class_id, this_year.i_category_id limit 100; -- query 15 select ca_zip ,sum(cs_sales_price) from catalog_sales ,customer ,customer_address ,date_dim where cs_bill_customer_sk = c_customer_sk and c_current_addr_sk = ca_address_sk and ( substr(ca_zip,1,5) in ('85669', '86197','88274','83405','86475', '85392', '85460', '80348', '81792') or ca_state in ('CA','WA','GA') or cs_sales_price > 500) and cs_sold_date_sk = d_date_sk and d_qoy = 2 and d_year = 2001 group by ca_zip order by ca_zip limit 100; -- query 16 select count(distinct cs_order_number) as "order count" ,sum(cs_ext_ship_cost) as "total shipping cost" ,sum(cs_net_profit) as "total net profit" from catalog_sales cs1 ,date_dim ,customer_address ,call_center where d_date between '2002-2-01' and date_add(cast('2002-2-01' as date), 60) and cs1.cs_ship_date_sk = d_date_sk and cs1.cs_ship_addr_sk = ca_address_sk and ca_state = 'GA' and cs1.cs_call_center_sk = cc_call_center_sk and cc_county in ('Williamson County', 'Williamson County', 'Williamson County', 'Williamson County', 'Williamson County') and exists (select * from catalog_sales cs2 where cs1.cs_order_number = cs2.cs_order_number and cs1.cs_warehouse_sk <> cs2.cs_warehouse_sk) and not exists(select * from catalog_returns cr1 where cs1.cs_order_number = cr1.cr_order_number) order by count(distinct cs_order_number) limit 100; -- query 17 select i_item_id ,i_item_desc ,s_state ,count(ss_quantity) as store_sales_quantitycount ,avg(ss_quantity) as store_sales_quantityave ,stddev_samp(ss_quantity) as store_sales_quantitystdev ,stddev_samp(ss_quantity)/avg(ss_quantity) as store_sales_quantitycov ,count(sr_return_quantity) as store_returns_quantitycount ,avg(sr_return_quantity) as store_returns_quantityave ,stddev_samp(sr_return_quantity) as store_returns_quantitystdev ,stddev_samp(sr_return_quantity)/avg(sr_return_quantity) as store_returns_quantitycov ,count(cs_quantity) as catalog_sales_quantitycount ,avg(cs_quantity) as catalog_sales_quantityave ,stddev_samp(cs_quantity) as catalog_sales_quantitystdev ,stddev_samp(cs_quantity)/avg(cs_quantity) as catalog_sales_quantitycov from store_sales ,store_returns ,catalog_sales ,date_dim d1 ,date_dim d2 ,date_dim d3 ,store ,item where d1.d_quarter_name = '2001Q1' and d1.d_date_sk = ss_sold_date_sk and i_item_sk = ss_item_sk and s_store_sk = ss_store_sk and ss_customer_sk = sr_customer_sk and ss_item_sk = sr_item_sk and ss_ticket_number = sr_ticket_number and sr_returned_date_sk = d2.d_date_sk and d2.d_quarter_name in ('2001Q1','2001Q2','2001Q3') and sr_customer_sk = cs_bill_customer_sk and sr_item_sk = cs_item_sk and cs_sold_date_sk = d3.d_date_sk and d3.d_quarter_name in ('2001Q1','2001Q2','2001Q3') group by i_item_id ,i_item_desc ,s_state order by i_item_id ,i_item_desc ,s_state limit 100; -- query 18 select i_item_id, ca_country, ca_state, ca_county, avg( cast(cs_quantity as decimal(12,2))) agg1, avg( cast(cs_list_price as decimal(12,2))) agg2, avg( cast(cs_coupon_amt as decimal(12,2))) agg3, avg( cast(cs_sales_price as decimal(12,2))) agg4, avg( cast(cs_net_profit as decimal(12,2))) agg5, avg( cast(c_birth_year as decimal(12,2))) agg6, avg( cast(cd1.cd_dep_count as decimal(12,2))) agg7 from catalog_sales, customer_demographics cd1, customer_demographics cd2, customer, customer_address, date_dim, item where cs_sold_date_sk = d_date_sk and cs_item_sk = i_item_sk and cs_bill_cdemo_sk = cd1.cd_demo_sk and cs_bill_customer_sk = c_customer_sk and cd1.cd_gender = 'F' and cd1.cd_education_status = 'Unknown' and c_current_cdemo_sk = cd2.cd_demo_sk and c_current_addr_sk = ca_address_sk and c_birth_month in (1, 6, 8, 9, 12, 2) and d_year = 1998 and ca_state in ('MS','IN','ND' ,'OK','NM','VA','MS') group by rollup (i_item_id, ca_country, ca_state, ca_county) order by ca_country, ca_state, ca_county, i_item_id limit 100; -- query 19 select i_brand_id brand_id, i_brand brand, i_manufact_id, i_manufact, sum(ss_ext_sales_price) ext_price from date_dim, store_sales, item,customer,customer_address,store where d_date_sk = ss_sold_date_sk and ss_item_sk = i_item_sk and i_manager_id=8 and d_moy=11 and d_year=1998 and ss_customer_sk = c_customer_sk and c_current_addr_sk = ca_address_sk and substr(ca_zip,1,5) <> substr(s_zip,1,5) and ss_store_sk = s_store_sk group by i_brand ,i_brand_id ,i_manufact_id ,i_manufact order by ext_price desc ,i_brand ,i_brand_id ,i_manufact_id ,i_manufact limit 100 ; -- query 20 select i_item_id ,i_item_desc ,i_category ,i_class ,i_current_price ,sum(cs_ext_sales_price) as itemrevenue ,sum(cs_ext_sales_price)*100/sum(sum(cs_ext_sales_price)) over (partition by i_class) as revenueratio from catalog_sales ,item ,date_dim where cs_item_sk = i_item_sk and i_category in ('Sports','Books','Home') and cs_sold_date_sk = d_date_sk and d_date between cast('1999-02-22' as date) and date_add(cast('1999-02-22' as date), 30) group by i_item_id ,i_item_desc ,i_category ,i_class ,i_current_price order by i_category ,i_class ,i_item_id ,i_item_desc ,revenueratio limit 100; -- query 21 select * from(select w_warehouse_name ,i_item_id ,sum(case when (cast(d_date as date) < cast ('2000-03-11' as date)) then inv_quantity_on_hand else 0 end) as inv_before ,sum(case when (cast(d_date as date) >= cast ('2000-03-11' as date)) then inv_quantity_on_hand else 0 end) as inv_after from inventory ,warehouse ,item ,date_dim where i_current_price between 0.99 and 1.49 and i_item_sk = inv_item_sk and inv_warehouse_sk = w_warehouse_sk and inv_date_sk = d_date_sk and d_date between date_add(cast ('2000-03-11' as date), -30) and date_add(cast ('2000-03-11' as date), 30) group by w_warehouse_name, i_item_id) x where (case when inv_before > 0 then inv_after / inv_before else null end) between 2.0/3.0 and 3.0/2.0 order by w_warehouse_name ,i_item_id limit 100; -- query 22 select i_product_name ,i_brand ,i_class ,i_category ,avg(inv_quantity_on_hand) qoh from inventory ,date_dim ,item where inv_date_sk=d_date_sk and inv_item_sk=i_item_sk and d_month_seq between 1200 and 1200 + 11 group by rollup(i_product_name ,i_brand ,i_class ,i_category) order by qoh, i_product_name, i_brand, i_class, i_category limit 100; -- query 23 with frequent_ss_items as (select substr(i_item_desc,1,30) itemdesc,i_item_sk item_sk,d_date solddate,count(*) cnt from store_sales ,date_dim ,item where ss_sold_date_sk = d_date_sk and ss_item_sk = i_item_sk and d_year in (2000,2000+1,2000+2,2000+3) group by substr(i_item_desc,1,30),i_item_sk,d_date having count(*) >4), max_store_sales as (select max(csales) tpcds_cmax from (select c_customer_sk,sum(ss_quantity*ss_sales_price) csales from store_sales ,customer ,date_dim where ss_customer_sk = c_customer_sk and ss_sold_date_sk = d_date_sk and d_year in (2000,2000+1,2000+2,2000+3) group by c_customer_sk) t1), best_ss_customer as (select c_customer_sk,sum(ss_quantity*ss_sales_price) ssales from store_sales ,customer where ss_customer_sk = c_customer_sk group by c_customer_sk having sum(ss_quantity*ss_sales_price) > (50/100.0) * (select * from max_store_sales)) select sum(sales) from (select cs_quantity*cs_list_price sales from catalog_sales ,date_dim where d_year = 2000 and d_moy = 2 and cs_sold_date_sk = d_date_sk and cs_item_sk in (select item_sk from frequent_ss_items) and cs_bill_customer_sk in (select c_customer_sk from best_ss_customer) union all select ws_quantity*ws_list_price sales from web_sales ,date_dim where d_year = 2000 and d_moy = 2 and ws_sold_date_sk = d_date_sk and ws_item_sk in (select item_sk from frequent_ss_items) and ws_bill_customer_sk in (select c_customer_sk from best_ss_customer)) t2 limit 100; with frequent_ss_items as (select substr(i_item_desc,1,30) itemdesc,i_item_sk item_sk,d_date solddate,count(*) cnt from store_sales ,date_dim ,item where ss_sold_date_sk = d_date_sk and ss_item_sk = i_item_sk and d_year in (2000,2000 + 1,2000 + 2,2000 + 3) group by substr(i_item_desc,1,30),i_item_sk,d_date having count(*) >4), max_store_sales as (select max(csales) tpcds_cmax from (select c_customer_sk,sum(ss_quantity*ss_sales_price) csales from store_sales ,customer ,date_dim where ss_customer_sk = c_customer_sk and ss_sold_date_sk = d_date_sk and d_year in (2000,2000+1,2000+2,2000+3) group by c_customer_sk) t1), best_ss_customer as (select c_customer_sk,sum(ss_quantity*ss_sales_price) ssales from store_sales ,customer where ss_customer_sk = c_customer_sk group by c_customer_sk having sum(ss_quantity*ss_sales_price) > (50/100.0) * (select * from max_store_sales)) select c_last_name,c_first_name,sales from (select c_last_name,c_first_name,sum(cs_quantity*cs_list_price) sales from catalog_sales ,customer ,date_dim where d_year = 2000 and d_moy = 2 and cs_sold_date_sk = d_date_sk and cs_item_sk in (select item_sk from frequent_ss_items) and cs_bill_customer_sk in (select c_customer_sk from best_ss_customer) and cs_bill_customer_sk = c_customer_sk group by c_last_name,c_first_name union all select c_last_name,c_first_name,sum(ws_quantity*ws_list_price) sales from web_sales ,customer ,date_dim where d_year = 2000 and d_moy = 2 and ws_sold_date_sk = d_date_sk and ws_item_sk in (select item_sk from frequent_ss_items) and ws_bill_customer_sk in (select c_customer_sk from best_ss_customer) and ws_bill_customer_sk = c_customer_sk group by c_last_name,c_first_name) t2 order by c_last_name,c_first_name,sales limit 100; -- query 24 with ssales as (select c_last_name ,c_first_name ,s_store_name ,ca_state ,s_state ,i_color ,i_current_price ,i_manager_id ,i_units ,i_size ,sum(ss_net_paid) netpaid from store_sales ,store_returns ,store ,item ,customer ,customer_address where ss_ticket_number = sr_ticket_number and ss_item_sk = sr_item_sk and ss_customer_sk = c_customer_sk and ss_item_sk = i_item_sk and ss_store_sk = s_store_sk and c_current_addr_sk = ca_address_sk and c_birth_country <> upper(ca_country) and s_zip = ca_zip and s_market_id=8 group by c_last_name ,c_first_name ,s_store_name ,ca_state ,s_state ,i_color ,i_current_price ,i_manager_id ,i_units ,i_size) select c_last_name ,c_first_name ,s_store_name ,sum(netpaid) paid from ssales where i_color = 'peach' group by c_last_name ,c_first_name ,s_store_name having sum(netpaid) > (select 0.05*avg(netpaid) from ssales) order by c_last_name ,c_first_name ,s_store_name ; with ssales as (select c_last_name ,c_first_name ,s_store_name ,ca_state ,s_state ,i_color ,i_current_price ,i_manager_id ,i_units ,i_size ,sum(ss_net_paid) netpaid from store_sales ,store_returns ,store ,item ,customer ,customer_address where ss_ticket_number = sr_ticket_number and ss_item_sk = sr_item_sk and ss_customer_sk = c_customer_sk and ss_item_sk = i_item_sk and ss_store_sk = s_store_sk and c_current_addr_sk = ca_address_sk and c_birth_country <> upper(ca_country) and s_zip = ca_zip and s_market_id = 8 group by c_last_name ,c_first_name ,s_store_name ,ca_state ,s_state ,i_color ,i_current_price ,i_manager_id ,i_units ,i_size) select c_last_name ,c_first_name ,s_store_name ,sum(netpaid) paid from ssales where i_color = 'saddle' group by c_last_name ,c_first_name ,s_store_name having sum(netpaid) > (select 0.05*avg(netpaid) from ssales) order by c_last_name ,c_first_name ,s_store_name ; -- query 25 select i_item_id ,i_item_desc ,s_store_id ,s_store_name ,sum(ss_net_profit) as store_sales_profit ,sum(sr_net_loss) as store_returns_loss ,sum(cs_net_profit) as catalog_sales_profit from store_sales ,store_returns ,catalog_sales ,date_dim d1 ,date_dim d2 ,date_dim d3 ,store ,item where d1.d_moy = 4 and d1.d_year = 2001 and d1.d_date_sk = ss_sold_date_sk and i_item_sk = ss_item_sk and s_store_sk = ss_store_sk and ss_customer_sk = sr_customer_sk and ss_item_sk = sr_item_sk and ss_ticket_number = sr_ticket_number and sr_returned_date_sk = d2.d_date_sk and d2.d_moy between 4 and 10 and d2.d_year = 2001 and sr_customer_sk = cs_bill_customer_sk and sr_item_sk = cs_item_sk and cs_sold_date_sk = d3.d_date_sk and d3.d_moy between 4 and 10 and d3.d_year = 2001 group by i_item_id ,i_item_desc ,s_store_id ,s_store_name order by i_item_id ,i_item_desc ,s_store_id ,s_store_name limit 100; -- query 26 select i_item_id, avg(cs_quantity) agg1, avg(cs_list_price) agg2, avg(cs_coupon_amt) agg3, avg(cs_sales_price) agg4 from catalog_sales, customer_demographics, date_dim, item, promotion where cs_sold_date_sk = d_date_sk and cs_item_sk = i_item_sk and cs_bill_cdemo_sk = cd_demo_sk and cs_promo_sk = p_promo_sk and cd_gender = 'M' and cd_marital_status = 'S' and cd_education_status = 'College' and (p_channel_email = 'N' or p_channel_event = 'N') and d_year = 2000 group by i_item_id order by i_item_id limit 100; -- query 27 select i_item_id, s_state, grouping(s_state) g_state, avg(ss_quantity) agg1, avg(ss_list_price) agg2, avg(ss_coupon_amt) agg3, avg(ss_sales_price) agg4 from store_sales, customer_demographics, date_dim, store, item where ss_sold_date_sk = d_date_sk and ss_item_sk = i_item_sk and ss_store_sk = s_store_sk and ss_cdemo_sk = cd_demo_sk and cd_gender = 'M' and cd_marital_status = 'S' and cd_education_status = 'College' and d_year = 2002 and s_state in ('TN','TN', 'TN', 'TN', 'TN', 'TN') group by rollup (i_item_id, s_state) order by i_item_id ,s_state limit 100; -- query 28 select * from (select avg(ss_list_price) B1_LP ,count(ss_list_price) B1_CNT ,count(distinct ss_list_price) B1_CNTD from store_sales where ss_quantity between 0 and 5 and (ss_list_price between 8 and 8+10 or ss_coupon_amt between 459 and 459+1000 or ss_wholesale_cost between 57 and 57+20)) B1, (select avg(ss_list_price) B2_LP ,count(ss_list_price) B2_CNT ,count(distinct ss_list_price) B2_CNTD from store_sales where ss_quantity between 6 and 10 and (ss_list_price between 90 and 90+10 or ss_coupon_amt between 2323 and 2323+1000 or ss_wholesale_cost between 31 and 31+20)) B2, (select avg(ss_list_price) B3_LP ,count(ss_list_price) B3_CNT ,count(distinct ss_list_price) B3_CNTD from store_sales where ss_quantity between 11 and 15 and (ss_list_price between 142 and 142+10 or ss_coupon_amt between 12214 and 12214+1000 or ss_wholesale_cost between 79 and 79+20)) B3, (select avg(ss_list_price) B4_LP ,count(ss_list_price) B4_CNT ,count(distinct ss_list_price) B4_CNTD from store_sales where ss_quantity between 16 and 20 and (ss_list_price between 135 and 135+10 or ss_coupon_amt between 6071 and 6071+1000 or ss_wholesale_cost between 38 and 38+20)) B4, (select avg(ss_list_price) B5_LP ,count(ss_list_price) B5_CNT ,count(distinct ss_list_price) B5_CNTD from store_sales where ss_quantity between 21 and 25 and (ss_list_price between 122 and 122+10 or ss_coupon_amt between 836 and 836+1000 or ss_wholesale_cost between 17 and 17+20)) B5, (select avg(ss_list_price) B6_LP ,count(ss_list_price) B6_CNT ,count(distinct ss_list_price) B6_CNTD from store_sales where ss_quantity between 26 and 30 and (ss_list_price between 154 and 154+10 or ss_coupon_amt between 7326 and 7326+1000 or ss_wholesale_cost between 7 and 7+20)) B6 limit 100; -- query 29 select i_item_id ,i_item_desc ,s_store_id ,s_store_name ,sum(ss_quantity) as store_sales_quantity ,sum(sr_return_quantity) as store_returns_quantity ,sum(cs_quantity) as catalog_sales_quantity from store_sales ,store_returns ,catalog_sales ,date_dim d1 ,date_dim d2 ,date_dim d3 ,store ,item where d1.d_moy = 9 and d1.d_year = 1999 and d1.d_date_sk = ss_sold_date_sk and i_item_sk = ss_item_sk and s_store_sk = ss_store_sk and ss_customer_sk = sr_customer_sk and ss_item_sk = sr_item_sk and ss_ticket_number = sr_ticket_number and sr_returned_date_sk = d2.d_date_sk and d2.d_moy between 9 and 9 + 3 and d2.d_year = 1999 and sr_customer_sk = cs_bill_customer_sk and sr_item_sk = cs_item_sk and cs_sold_date_sk = d3.d_date_sk and d3.d_year in (1999,1999+1,1999+2) group by i_item_id ,i_item_desc ,s_store_id ,s_store_name order by i_item_id ,i_item_desc ,s_store_id ,s_store_name limit 100; -- query 30 with customer_total_return as (select wr_returning_customer_sk as ctr_customer_sk ,ca_state as ctr_state, sum(wr_return_amt) as ctr_total_return from web_returns ,date_dim ,customer_address where wr_returned_date_sk = d_date_sk and d_year =2002 and wr_returning_addr_sk = ca_address_sk group by wr_returning_customer_sk ,ca_state) select c_customer_id,c_salutation,c_first_name,c_last_name,c_preferred_cust_flag ,c_birth_day,c_birth_month,c_birth_year,c_birth_country,c_login,c_email_address ,c_last_review_date,ctr_total_return from customer_total_return ctr1 ,customer_address ,customer where ctr1.ctr_total_return > (select avg(ctr_total_return)*1.2 from customer_total_return ctr2 where ctr1.ctr_state = ctr2.ctr_state) and ca_address_sk = c_current_addr_sk and ca_state = 'GA' and ctr1.ctr_customer_sk = c_customer_sk order by c_customer_id,c_salutation,c_first_name,c_last_name,c_preferred_cust_flag ,c_birth_day,c_birth_month,c_birth_year,c_birth_country,c_login,c_email_address ,c_last_review_date,ctr_total_return limit 100; -- query 31 with ss as (select ca_county,d_qoy, d_year,sum(ss_ext_sales_price) as store_sales from store_sales,date_dim,customer_address where ss_sold_date_sk = d_date_sk and ss_addr_sk=ca_address_sk group by ca_county,d_qoy, d_year), ws as (select ca_county,d_qoy, d_year,sum(ws_ext_sales_price) as web_sales from web_sales,date_dim,customer_address where ws_sold_date_sk = d_date_sk and ws_bill_addr_sk=ca_address_sk group by ca_county,d_qoy, d_year) select ss1.ca_county ,ss1.d_year ,ws2.web_sales/ws1.web_sales web_q1_q2_increase ,ss2.store_sales/ss1.store_sales store_q1_q2_increase ,ws3.web_sales/ws2.web_sales web_q2_q3_increase ,ss3.store_sales/ss2.store_sales store_q2_q3_increase from ss ss1 ,ss ss2 ,ss ss3 ,ws ws1 ,ws ws2 ,ws ws3 where ss1.d_qoy = 1 and ss1.d_year = 2000 and ss1.ca_county = ss2.ca_county and ss2.d_qoy = 2 and ss2.d_year = 2000 and ss2.ca_county = ss3.ca_county and ss3.d_qoy = 3 and ss3.d_year = 2000 and ss1.ca_county = ws1.ca_county and ws1.d_qoy = 1 and ws1.d_year = 2000 and ws1.ca_county = ws2.ca_county and ws2.d_qoy = 2 and ws2.d_year = 2000 and ws1.ca_county = ws3.ca_county and ws3.d_qoy = 3 and ws3.d_year =2000 and case when ws1.web_sales > 0 then ws2.web_sales/ws1.web_sales else null end > case when ss1.store_sales > 0 then ss2.store_sales/ss1.store_sales else null end and case when ws2.web_sales > 0 then ws3.web_sales/ws2.web_sales else null end > case when ss2.store_sales > 0 then ss3.store_sales/ss2.store_sales else null end order by ss1.ca_county; -- query 32 select sum(cs_ext_discount_amt) as "excess discount amount" from catalog_sales ,item ,date_dim where i_manufact_id = 977 and i_item_sk = cs_item_sk and d_date between '2000-01-27' and date_add(cast('2000-01-27' as date), 90) and d_date_sk = cs_sold_date_sk and cs_ext_discount_amt > ( select 1.3 * avg(cs_ext_discount_amt) from catalog_sales ,date_dim where cs_item_sk = i_item_sk and d_date between '2000-01-27' and date_add(cast('2000-01-27' as date), 90) and d_date_sk = cs_sold_date_sk ) limit 100; -- query 33 with ss as ( select i_manufact_id,sum(ss_ext_sales_price) total_sales from store_sales, date_dim, customer_address, item where i_manufact_id in (select i_manufact_id from item where i_category in ('Electronics')) and ss_item_sk = i_item_sk and ss_sold_date_sk = d_date_sk and d_year = 1998 and d_moy = 5 and ss_addr_sk = ca_address_sk and ca_gmt_offset = -5 group by i_manufact_id), cs as ( select i_manufact_id,sum(cs_ext_sales_price) total_sales from catalog_sales, date_dim, customer_address, item where i_manufact_id in (select i_manufact_id from item where i_category in ('Electronics')) and cs_item_sk = i_item_sk and cs_sold_date_sk = d_date_sk and d_year = 1998 and d_moy = 5 and cs_bill_addr_sk = ca_address_sk and ca_gmt_offset = -5 group by i_manufact_id), ws as ( select i_manufact_id,sum(ws_ext_sales_price) total_sales from web_sales, date_dim, customer_address, item where i_manufact_id in (select i_manufact_id from item where i_category in ('Electronics')) and ws_item_sk = i_item_sk and ws_sold_date_sk = d_date_sk and d_year = 1998 and d_moy = 5 and ws_bill_addr_sk = ca_address_sk and ca_gmt_offset = -5 group by i_manufact_id) select i_manufact_id ,sum(total_sales) total_sales from (select * from ss union all select * from cs union all select * from ws) tmp1 group by i_manufact_id order by total_sales limit 100; -- query 34 select c_last_name ,c_first_name ,c_salutation ,c_preferred_cust_flag ,ss_ticket_number ,cnt from (select ss_ticket_number ,ss_customer_sk ,count(*) cnt from store_sales,date_dim,store,household_demographics where store_sales.ss_sold_date_sk = date_dim.d_date_sk and store_sales.ss_store_sk = store.s_store_sk and store_sales.ss_hdemo_sk = household_demographics.hd_demo_sk and (date_dim.d_dom between 1 and 3 or date_dim.d_dom between 25 and 28) and (household_demographics.hd_buy_potential = '>10000' or household_demographics.hd_buy_potential = 'Unknown') and household_demographics.hd_vehicle_count > 0 and (case when household_demographics.hd_vehicle_count > 0 then household_demographics.hd_dep_count/ household_demographics.hd_vehicle_count else null end) > 1.2 and date_dim.d_year in (1999,1999+1,1999+2) and store.s_county in ('Williamson County','Williamson County','Williamson County','Williamson County', 'Williamson County','Williamson County','Williamson County','Williamson County') group by ss_ticket_number,ss_customer_sk) dn,customer where ss_customer_sk = c_customer_sk and cnt between 15 and 20 order by c_last_name,c_first_name,c_salutation,c_preferred_cust_flag desc, ss_ticket_number; -- query 35 select ca_state, cd_gender, cd_marital_status, cd_dep_count, count(*) cnt1, min(cd_dep_count), max(cd_dep_count), avg(cd_dep_count), cd_dep_employed_count, count(*) cnt2, min(cd_dep_employed_count), max(cd_dep_employed_count), avg(cd_dep_employed_count), cd_dep_college_count, count(*) cnt3, min(cd_dep_college_count), max(cd_dep_college_count), avg(cd_dep_college_count) from customer c,customer_address ca,customer_demographics where c.c_current_addr_sk = ca.ca_address_sk and cd_demo_sk = c.c_current_cdemo_sk and exists (select * from store_sales,date_dim where c.c_customer_sk = ss_customer_sk and ss_sold_date_sk = d_date_sk and d_year = 2002 and d_qoy < 4) and exists (select * from (select ws_bill_customer_sk customsk from web_sales,date_dim where ws_sold_date_sk = d_date_sk and d_year = 2002 and d_qoy < 4 union all select cs_ship_customer_sk customsk from catalog_sales,date_dim where cs_sold_date_sk = d_date_sk and d_year = 2002 and d_qoy < 4)x where x.customsk = c.c_customer_sk) group by ca_state, cd_gender, cd_marital_status, cd_dep_count, cd_dep_employed_count, cd_dep_college_count order by ca_state, cd_gender, cd_marital_status, cd_dep_count, cd_dep_employed_count, cd_dep_college_count limit 100; -- query 36 select sum(ss_net_profit)/sum(ss_ext_sales_price) as gross_margin ,i_category ,i_class ,grouping(i_category)+grouping(i_class) as lochierarchy ,rank() over ( partition by grouping(i_category)+grouping(i_class), case when grouping(i_class) = 0 then i_category end order by sum(ss_net_profit)/sum(ss_ext_sales_price) asc) as rank_within_parent from store_sales ,date_dim d1 ,item ,store where d1.d_year = 2001 and d1.d_date_sk = ss_sold_date_sk and i_item_sk = ss_item_sk and s_store_sk = ss_store_sk and s_state in ('TN','TN','TN','TN', 'TN','TN','TN','TN') group by rollup(i_category,i_class) order by lochierarchy desc ,case when lochierarchy = 0 then i_category end ,rank_within_parent limit 100; -- query 37 select i_item_id ,i_item_desc ,i_current_price from item, inventory, date_dim, catalog_sales where i_current_price between 68 and 68 + 30 and inv_item_sk = i_item_sk and d_date_sk=inv_date_sk and d_date between cast('2000-02-01' as date) and date_add(cast('2000-02-01' as date), 60) and i_manufact_id in (677,940,694,808) and inv_quantity_on_hand between 100 and 500 and cs_item_sk = i_item_sk group by i_item_id,i_item_desc,i_current_price order by i_item_id limit 100; -- query 38 select count(*) from ( select distinct c_last_name, c_first_name, d_date from store_sales, date_dim, customer where store_sales.ss_sold_date_sk = date_dim.d_date_sk and store_sales.ss_customer_sk = customer.c_customer_sk and d_month_seq between 1200 and 1200 + 11 intersect select distinct c_last_name, c_first_name, d_date from catalog_sales, date_dim, customer where catalog_sales.cs_sold_date_sk = date_dim.d_date_sk and catalog_sales.cs_bill_customer_sk = customer.c_customer_sk and d_month_seq between 1200 and 1200 + 11 intersect select distinct c_last_name, c_first_name, d_date from web_sales, date_dim, customer where web_sales.ws_sold_date_sk = date_dim.d_date_sk and web_sales.ws_bill_customer_sk = customer.c_customer_sk and d_month_seq between 1200 and 1200 + 11 ) hot_cust limit 100; -- query 39 with inv as (select w_warehouse_name,w_warehouse_sk,i_item_sk,d_moy ,stdev,mean, case mean when 0 then null else stdev/mean end cov from(select w_warehouse_name,w_warehouse_sk,i_item_sk,d_moy ,stddev_samp(inv_quantity_on_hand) stdev,avg(inv_quantity_on_hand) mean from inventory ,item ,warehouse ,date_dim where inv_item_sk = i_item_sk and inv_warehouse_sk = w_warehouse_sk and inv_date_sk = d_date_sk and d_year =2001 group by w_warehouse_name,w_warehouse_sk,i_item_sk,d_moy) foo where case mean when 0 then 0 else stdev/mean end > 1) select inv1.w_warehouse_sk,inv1.i_item_sk,inv1.d_moy,inv1.mean, inv1.cov ,inv2.w_warehouse_sk,inv2.i_item_sk,inv2.d_moy,inv2.mean, inv2.cov from inv inv1,inv inv2 where inv1.i_item_sk = inv2.i_item_sk and inv1.w_warehouse_sk = inv2.w_warehouse_sk and inv1.d_moy=1 and inv2.d_moy=1+1 order by inv1.w_warehouse_sk,inv1.i_item_sk,inv1.d_moy,inv1.mean,inv1.cov ,inv2.d_moy,inv2.mean, inv2.cov ; with inv as (select w_warehouse_name,w_warehouse_sk,i_item_sk,d_moy ,stdev,mean, case mean when 0 then null else stdev/mean end cov from(select w_warehouse_name,w_warehouse_sk,i_item_sk,d_moy ,stddev_samp(inv_quantity_on_hand) stdev,avg(inv_quantity_on_hand) mean from inventory ,item ,warehouse ,date_dim where inv_item_sk = i_item_sk and inv_warehouse_sk = w_warehouse_sk and inv_date_sk = d_date_sk and d_year =2001 group by w_warehouse_name,w_warehouse_sk,i_item_sk,d_moy) foo where case mean when 0 then 0 else stdev/mean end > 1) select inv1.w_warehouse_sk,inv1.i_item_sk,inv1.d_moy,inv1.mean, inv1.cov ,inv2.w_warehouse_sk,inv2.i_item_sk,inv2.d_moy,inv2.mean, inv2.cov from inv inv1,inv inv2 where inv1.i_item_sk = inv2.i_item_sk and inv1.w_warehouse_sk = inv2.w_warehouse_sk and inv1.d_moy=1 and inv2.d_moy=1+1 and inv1.cov > 1.5 order by inv1.w_warehouse_sk,inv1.i_item_sk,inv1.d_moy,inv1.mean,inv1.cov ,inv2.d_moy,inv2.mean, inv2.cov ; -- query 40 select w_state ,i_item_id ,sum(case when (cast(d_date as date) < cast ('2000-03-11' as date)) then cs_sales_price - coalesce(cr_refunded_cash,0) else 0 end) as sales_before ,sum(case when (cast(d_date as date) >= cast ('2000-03-11' as date)) then cs_sales_price - coalesce(cr_refunded_cash,0) else 0 end) as sales_after from catalog_sales left outer join catalog_returns on (cs_order_number = cr_order_number and cs_item_sk = cr_item_sk) ,warehouse ,item ,date_dim where i_current_price between 0.99 and 1.49 and i_item_sk = cs_item_sk and cs_warehouse_sk = w_warehouse_sk and cs_sold_date_sk = d_date_sk and d_date between date_add(cast ('2000-03-11' as date), -30) and date_add(cast ('2000-03-11' as date), 30) group by w_state,i_item_id order by w_state,i_item_id limit 100; -- query 41 select distinct(i_product_name) from item i1 where i_manufact_id between 738 and 738+40 and (select count(*) as item_cnt from item where (i_manufact = i1.i_manufact and ((i_category = 'Women' and (i_color = 'powder' or i_color = 'khaki') and (i_units = 'Ounce' or i_units = 'Oz') and (i_size = 'medium' or i_size = 'extra large') ) or (i_category = 'Women' and (i_color = 'brown' or i_color = 'honeydew') and (i_units = 'Bunch' or i_units = 'Ton') and (i_size = 'N/A' or i_size = 'small') ) or (i_category = 'Men' and (i_color = 'floral' or i_color = 'deep') and (i_units = 'N/A' or i_units = 'Dozen') and (i_size = 'petite' or i_size = 'large') ) or (i_category = 'Men' and (i_color = 'light' or i_color = 'cornflower') and (i_units = 'Box' or i_units = 'Pound') and (i_size = 'medium' or i_size = 'extra large') ))) or (i_manufact = i1.i_manufact and ((i_category = 'Women' and (i_color = 'midnight' or i_color = 'snow') and (i_units = 'Pallet' or i_units = 'Gross') and (i_size = 'medium' or i_size = 'extra large') ) or (i_category = 'Women' and (i_color = 'cyan' or i_color = 'papaya') and (i_units = 'Cup' or i_units = 'Dram') and (i_size = 'N/A' or i_size = 'small') ) or (i_category = 'Men' and (i_color = 'orange' or i_color = 'frosted') and (i_units = 'Each' or i_units = 'Tbl') and (i_size = 'petite' or i_size = 'large') ) or (i_category = 'Men' and (i_color = 'forest' or i_color = 'ghost') and (i_units = 'Lb' or i_units = 'Bundle') and (i_size = 'medium' or i_size = 'extra large') )))) > 0 order by i_product_name limit 100; -- query 42 select dt.d_year ,item.i_category_id ,item.i_category ,sum(ss_ext_sales_price) from date_dim dt ,store_sales ,item where dt.d_date_sk = store_sales.ss_sold_date_sk and store_sales.ss_item_sk = item.i_item_sk and item.i_manager_id = 1 and dt.d_moy=11 and dt.d_year=2000 group by dt.d_year ,item.i_category_id ,item.i_category order by sum(ss_ext_sales_price) desc,dt.d_year ,item.i_category_id ,item.i_category limit 100 ; -- query 43 select s_store_name, s_store_id, sum(case when (d_day_name='Sunday') then ss_sales_price else null end) sun_sales, sum(case when (d_day_name='Monday') then ss_sales_price else null end) mon_sales, sum(case when (d_day_name='Tuesday') then ss_sales_price else null end) tue_sales, sum(case when (d_day_name='Wednesday') then ss_sales_price else null end) wed_sales, sum(case when (d_day_name='Thursday') then ss_sales_price else null end) thu_sales, sum(case when (d_day_name='Friday') then ss_sales_price else null end) fri_sales, sum(case when (d_day_name='Saturday') then ss_sales_price else null end) sat_sales from date_dim, store_sales, store where d_date_sk = ss_sold_date_sk and s_store_sk = ss_store_sk and s_gmt_offset = -5 and d_year = 2000 group by s_store_name, s_store_id order by s_store_name, s_store_id,sun_sales,mon_sales,tue_sales,wed_sales,thu_sales,fri_sales,sat_sales limit 100; -- query 44 select asceding.rnk, i1.i_product_name best_performing, i2.i_product_name worst_performing from(select * from (select item_sk,rank() over (order by rank_col asc) rnk from (select ss_item_sk item_sk,avg(ss_net_profit) rank_col from store_sales ss1 where ss_store_sk = 4 group by ss_item_sk having avg(ss_net_profit) > 0.9*(select avg(ss_net_profit) rank_col from store_sales where ss_store_sk = 4 and ss_addr_sk is null group by ss_store_sk))V1)V11 where rnk < 11) asceding, (select * from (select item_sk,rank() over (order by rank_col desc) rnk from (select ss_item_sk item_sk,avg(ss_net_profit) rank_col from store_sales ss1 where ss_store_sk = 4 group by ss_item_sk having avg(ss_net_profit) > 0.9*(select avg(ss_net_profit) rank_col from store_sales where ss_store_sk = 4 and ss_addr_sk is null group by ss_store_sk))V2)V21 where rnk < 11) descending, item i1, item i2 where asceding.rnk = descending.rnk and i1.i_item_sk=asceding.item_sk and i2.i_item_sk=descending.item_sk order by asceding.rnk limit 100; -- query 45 select ca_zip, ca_city, sum(ws_sales_price) from web_sales, customer, customer_address, date_dim, item where ws_bill_customer_sk = c_customer_sk and c_current_addr_sk = ca_address_sk and ws_item_sk = i_item_sk and ( substr(ca_zip,1,5) in ('85669', '86197','88274','83405','86475', '85392', '85460', '80348', '81792') or i_item_id in (select i_item_id from item where i_item_sk in (2, 3, 5, 7, 11, 13, 17, 19, 23, 29) ) ) and ws_sold_date_sk = d_date_sk and d_qoy = 2 and d_year = 2001 group by ca_zip, ca_city order by ca_zip, ca_city limit 100; -- query 46 select c_last_name ,c_first_name ,ca_city ,bought_city ,ss_ticket_number ,amt,profit from (select ss_ticket_number ,ss_customer_sk ,ca_city bought_city ,sum(ss_coupon_amt) amt ,sum(ss_net_profit) profit from store_sales,date_dim,store,household_demographics,customer_address where store_sales.ss_sold_date_sk = date_dim.d_date_sk and store_sales.ss_store_sk = store.s_store_sk and store_sales.ss_hdemo_sk = household_demographics.hd_demo_sk and store_sales.ss_addr_sk = customer_address.ca_address_sk and (household_demographics.hd_dep_count = 4 or household_demographics.hd_vehicle_count= 3) and date_dim.d_dow in (6,0) and date_dim.d_year in (1999,1999+1,1999+2) and store.s_city in ('Fairview','Midway','Fairview','Fairview','Fairview') group by ss_ticket_number,ss_customer_sk,ss_addr_sk,ca_city) dn,customer,customer_address current_addr where ss_customer_sk = c_customer_sk and customer.c_current_addr_sk = current_addr.ca_address_sk and current_addr.ca_city <> bought_city order by c_last_name ,c_first_name ,ca_city ,bought_city ,ss_ticket_number limit 100; -- query 47 with v1 as( select i_category, i_brand, s_store_name, s_company_name, d_year, d_moy, sum(ss_sales_price) sum_sales, avg(sum(ss_sales_price)) over (partition by i_category, i_brand, s_store_name, s_company_name, d_year) avg_monthly_sales, rank() over (partition by i_category, i_brand, s_store_name, s_company_name order by d_year, d_moy) rn from item, store_sales, date_dim, store where ss_item_sk = i_item_sk and ss_sold_date_sk = d_date_sk and ss_store_sk = s_store_sk and ( d_year = 1999 or ( d_year = 1999-1 and d_moy =12) or ( d_year = 1999+1 and d_moy =1) ) group by i_category, i_brand, s_store_name, s_company_name, d_year, d_moy), v2 as( select v1.i_category, v1.i_brand, v1.s_store_name, v1.s_company_name ,v1.d_year, v1.d_moy ,v1.avg_monthly_sales ,v1.sum_sales, v1_lag.sum_sales psum, v1_lead.sum_sales nsum from v1, v1 v1_lag, v1 v1_lead where v1.i_category = v1_lag.i_category and v1.i_category = v1_lead.i_category and v1.i_brand = v1_lag.i_brand and v1.i_brand = v1_lead.i_brand and v1.s_store_name = v1_lag.s_store_name and v1.s_store_name = v1_lead.s_store_name and v1.s_company_name = v1_lag.s_company_name and v1.s_company_name = v1_lead.s_company_name and v1.rn = v1_lag.rn + 1 and v1.rn = v1_lead.rn - 1) select * from v2 where d_year = 1999 and avg_monthly_sales > 0 and case when avg_monthly_sales > 0 then abs(sum_sales - avg_monthly_sales) / avg_monthly_sales else null end > 0.1 order by sum_sales - avg_monthly_sales, s_store_name limit 100; -- query 48 select sum (ss_quantity) from store_sales, store, customer_demographics, customer_address, date_dim where s_store_sk = ss_store_sk and ss_sold_date_sk = d_date_sk and d_year = 2000 and ( ( cd_demo_sk = ss_cdemo_sk and cd_marital_status = 'M' and cd_education_status = '4 yr Degree' and ss_sales_price between 100.00 and 150.00 ) or ( cd_demo_sk = ss_cdemo_sk and cd_marital_status = 'D' and cd_education_status = '2 yr Degree' and ss_sales_price between 50.00 and 100.00 ) or ( cd_demo_sk = ss_cdemo_sk and cd_marital_status = 'S' and cd_education_status = 'College' and ss_sales_price between 150.00 and 200.00 ) ) and ( ( ss_addr_sk = ca_address_sk and ca_country = 'United States' and ca_state in ('CO', 'OH', 'TX') and ss_net_profit between 0 and 2000 ) or (ss_addr_sk = ca_address_sk and ca_country = 'United States' and ca_state in ('OR', 'MN', 'KY') and ss_net_profit between 150 and 3000 ) or (ss_addr_sk = ca_address_sk and ca_country = 'United States' and ca_state in ('VA', 'CA', 'MS') and ss_net_profit between 50 and 25000 ) ) ; -- query 49 select channel, item, return_ratio, return_rank, currency_rank from (select 'web' as channel ,web.item ,web.return_ratio ,web.return_rank ,web.currency_rank from ( select item ,return_ratio ,currency_ratio ,rank() over (order by return_ratio) as return_rank ,rank() over (order by currency_ratio) as currency_rank from ( select ws.ws_item_sk as item ,(cast(sum(coalesce(wr.wr_return_quantity,0)) as decimal(15,4))/ cast(sum(coalesce(ws.ws_quantity,0)) as decimal(15,4) )) as return_ratio ,(cast(sum(coalesce(wr.wr_return_amt,0)) as decimal(15,4))/ cast(sum(coalesce(ws.ws_net_paid,0)) as decimal(15,4) )) as currency_ratio from web_sales ws left outer join web_returns wr on (ws.ws_order_number = wr.wr_order_number and ws.ws_item_sk = wr.wr_item_sk) ,date_dim where wr.wr_return_amt > 10000 and ws.ws_net_profit > 1 and ws.ws_net_paid > 0 and ws.ws_quantity > 0 and ws_sold_date_sk = d_date_sk and d_year = 2001 and d_moy = 12 group by ws.ws_item_sk ) in_web ) web where ( web.return_rank <= 10 or web.currency_rank <= 10 ) union select 'catalog' as channel ,catalog.item ,catalog.return_ratio ,catalog.return_rank ,catalog.currency_rank from ( select item ,return_ratio ,currency_ratio ,rank() over (order by return_ratio) as return_rank ,rank() over (order by currency_ratio) as currency_rank from ( select cs.cs_item_sk as item ,(cast(sum(coalesce(cr.cr_return_quantity,0)) as decimal(15,4))/ cast(sum(coalesce(cs.cs_quantity,0)) as decimal(15,4) )) as return_ratio ,(cast(sum(coalesce(cr.cr_return_amount,0)) as decimal(15,4))/ cast(sum(coalesce(cs.cs_net_paid,0)) as decimal(15,4) )) as currency_ratio from catalog_sales cs left outer join catalog_returns cr on (cs.cs_order_number = cr.cr_order_number and cs.cs_item_sk = cr.cr_item_sk) ,date_dim where cr.cr_return_amount > 10000 and cs.cs_net_profit > 1 and cs.cs_net_paid > 0 and cs.cs_quantity > 0 and cs_sold_date_sk = d_date_sk and d_year = 2001 and d_moy = 12 group by cs.cs_item_sk ) in_cat ) catalog where ( catalog.return_rank <= 10 or catalog.currency_rank <=10 ) union select 'store' as channel ,store.item ,store.return_ratio ,store.return_rank ,store.currency_rank from ( select item ,return_ratio ,currency_ratio ,rank() over (order by return_ratio) as return_rank ,rank() over (order by currency_ratio) as currency_rank from ( select sts.ss_item_sk as item ,(cast(sum(coalesce(sr.sr_return_quantity,0)) as decimal(15,4))/cast(sum(coalesce(sts.ss_quantity,0)) as decimal(15,4) )) as return_ratio ,(cast(sum(coalesce(sr.sr_return_amt,0)) as decimal(15,4))/cast(sum(coalesce(sts.ss_net_paid,0)) as decimal(15,4) )) as currency_ratio from store_sales sts left outer join store_returns sr on (sts.ss_ticket_number = sr.sr_ticket_number and sts.ss_item_sk = sr.sr_item_sk) ,date_dim where sr.sr_return_amt > 10000 and sts.ss_net_profit > 1 and sts.ss_net_paid > 0 and sts.ss_quantity > 0 and ss_sold_date_sk = d_date_sk and d_year = 2001 and d_moy = 12 group by sts.ss_item_sk ) in_store ) store where ( store.return_rank <= 10 or store.currency_rank <= 10 ) ) t1 order by 1,4,5,2 limit 100; -- query 50 select s_store_name ,s_company_id ,s_street_number ,s_street_name ,s_street_type ,s_suite_number ,s_city ,s_county ,s_state ,s_zip ,sum(case when (sr_returned_date_sk - ss_sold_date_sk <= 30 ) then 1 else 0 end) as "30 days" ,sum(case when (sr_returned_date_sk - ss_sold_date_sk > 30) and (sr_returned_date_sk - ss_sold_date_sk <= 60) then 1 else 0 end ) as "31-60 days" ,sum(case when (sr_returned_date_sk - ss_sold_date_sk > 60) and (sr_returned_date_sk - ss_sold_date_sk <= 90) then 1 else 0 end) as "61-90 days" ,sum(case when (sr_returned_date_sk - ss_sold_date_sk > 90) and (sr_returned_date_sk - ss_sold_date_sk <= 120) then 1 else 0 end) as "91-120 days" ,sum(case when (sr_returned_date_sk - ss_sold_date_sk > 120) then 1 else 0 end) as ">120 days" from store_sales ,store_returns ,store ,date_dim d1 ,date_dim d2 where d2.d_year = 2001 and d2.d_moy = 8 and ss_ticket_number = sr_ticket_number and ss_item_sk = sr_item_sk and ss_sold_date_sk = d1.d_date_sk and sr_returned_date_sk = d2.d_date_sk and ss_customer_sk = sr_customer_sk and ss_store_sk = s_store_sk group by s_store_name ,s_company_id ,s_street_number ,s_street_name ,s_street_type ,s_suite_number ,s_city ,s_county ,s_state ,s_zip order by s_store_name ,s_company_id ,s_street_number ,s_street_name ,s_street_type ,s_suite_number ,s_city ,s_county ,s_state ,s_zip limit 100; -- query 51 WITH web_v1 as ( select ws_item_sk item_sk, d_date, sum(sum(ws_sales_price)) over (partition by ws_item_sk order by d_date rows between unbounded preceding and current row) cume_sales from web_sales ,date_dim where ws_sold_date_sk=d_date_sk and d_month_seq between 1200 and 1200+11 and ws_item_sk is not NULL group by ws_item_sk, d_date), store_v1 as ( select ss_item_sk item_sk, d_date, sum(sum(ss_sales_price)) over (partition by ss_item_sk order by d_date rows between unbounded preceding and current row) cume_sales from store_sales ,date_dim where ss_sold_date_sk=d_date_sk and d_month_seq between 1200 and 1200+11 and ss_item_sk is not NULL group by ss_item_sk, d_date) select * from (select item_sk ,d_date ,web_sales ,store_sales ,max(web_sales) over (partition by item_sk order by d_date rows between unbounded preceding and current row) web_cumulative ,max(store_sales) over (partition by item_sk order by d_date rows between unbounded preceding and current row) store_cumulative from (select case when web.item_sk is not null then web.item_sk else store.item_sk end item_sk ,case when web.d_date is not null then web.d_date else store.d_date end d_date ,web.cume_sales web_sales ,store.cume_sales store_sales from web_v1 web full outer join store_v1 store on (web.item_sk = store.item_sk and web.d_date = store.d_date) )x )y where web_cumulative > store_cumulative order by item_sk ,d_date limit 100; -- query 52 select dt.d_year ,item.i_brand_id brand_id ,item.i_brand brand ,sum(ss_ext_sales_price) ext_price from date_dim dt ,store_sales ,item where dt.d_date_sk = store_sales.ss_sold_date_sk and store_sales.ss_item_sk = item.i_item_sk and item.i_manager_id = 1 and dt.d_moy=11 and dt.d_year=2000 group by dt.d_year ,item.i_brand ,item.i_brand_id order by dt.d_year ,ext_price desc ,brand_id limit 100 ; -- query 53 select * from (select i_manufact_id, sum(ss_sales_price) sum_sales, avg(sum(ss_sales_price)) over (partition by i_manufact_id) avg_quarterly_sales from item, store_sales, date_dim, store where ss_item_sk = i_item_sk and ss_sold_date_sk = d_date_sk and ss_store_sk = s_store_sk and d_month_seq in (1200,1200+1,1200+2,1200+3,1200+4,1200+5,1200+6,1200+7,1200+8,1200+9,1200+10,1200+11) and ((i_category in ('Books','Children','Electronics') and i_class in ('personal','portable','reference','self-help') and i_brand in ('scholaramalgamalg #14','scholaramalgamalg #7', 'exportiunivamalg #9','scholaramalgamalg #9')) or(i_category in ('Women','Music','Men') and i_class in ('accessories','classical','fragrances','pants') and i_brand in ('amalgimporto #1','edu packscholar #1','exportiimporto #1', 'importoamalg #1'))) group by i_manufact_id, d_qoy ) tmp1 where case when avg_quarterly_sales > 0 then abs (sum_sales - avg_quarterly_sales)/ avg_quarterly_sales else null end > 0.1 order by avg_quarterly_sales, sum_sales, i_manufact_id limit 100; -- query 54 with my_customers as ( select distinct c_customer_sk , c_current_addr_sk from ( select cs_sold_date_sk sold_date_sk, cs_bill_customer_sk customer_sk, cs_item_sk item_sk from catalog_sales union all select ws_sold_date_sk sold_date_sk, ws_bill_customer_sk customer_sk, ws_item_sk item_sk from web_sales ) cs_or_ws_sales, item, date_dim, customer where sold_date_sk = d_date_sk and item_sk = i_item_sk and i_category = 'Women' and i_class = 'maternity' and c_customer_sk = cs_or_ws_sales.customer_sk and d_moy = 12 and d_year = 1998 ) , my_revenue as ( select c_customer_sk, sum(ss_ext_sales_price) as revenue from my_customers, store_sales, customer_address, store, date_dim where c_current_addr_sk = ca_address_sk and ca_county = s_county and ca_state = s_state and ss_sold_date_sk = d_date_sk and c_customer_sk = ss_customer_sk and d_month_seq between (select distinct d_month_seq+1 from date_dim where d_year = 1998 and d_moy = 12) and (select distinct d_month_seq+3 from date_dim where d_year = 1998 and d_moy = 12) group by c_customer_sk ) , segments as (select cast((revenue/50) as int) as segment from my_revenue ) select segment, count(*) as num_customers, segment*50 as segment_base from segments group by segment order by segment, num_customers limit 100; -- query 55 select i_brand_id brand_id, i_brand brand, sum(ss_ext_sales_price) ext_price from date_dim, store_sales, item where d_date_sk = ss_sold_date_sk and ss_item_sk = i_item_sk and i_manager_id=28 and d_moy=11 and d_year=1999 group by i_brand, i_brand_id order by ext_price desc, i_brand_id limit 100 ; -- query 56 with ss as ( select i_item_id,sum(ss_ext_sales_price) total_sales from store_sales, date_dim, customer_address, item where i_item_id in (select i_item_id from item where i_color in ('slate', 'blanched', 'burnished')) and ss_item_sk = i_item_sk and ss_sold_date_sk = d_date_sk and d_year = 2001 and d_moy = 2 and ss_addr_sk = ca_address_sk and ca_gmt_offset = -5 group by i_item_id), cs as ( select i_item_id,sum(cs_ext_sales_price) total_sales from catalog_sales, date_dim, customer_address, item where i_item_id in (select i_item_id from item where i_color in ('slate', 'blanched', 'burnished')) and cs_item_sk = i_item_sk and cs_sold_date_sk = d_date_sk and d_year = 2001 and d_moy = 2 and cs_bill_addr_sk = ca_address_sk and ca_gmt_offset = -5 group by i_item_id), ws as ( select i_item_id,sum(ws_ext_sales_price) total_sales from web_sales, date_dim, customer_address, item where i_item_id in (select i_item_id from item where i_color in ('slate', 'blanched', 'burnished')) and ws_item_sk = i_item_sk and ws_sold_date_sk = d_date_sk and d_year = 2001 and d_moy = 2 and ws_bill_addr_sk = ca_address_sk and ca_gmt_offset = -5 group by i_item_id) select i_item_id ,sum(total_sales) total_sales from (select * from ss union all select * from cs union all select * from ws) tmp1 group by i_item_id order by total_sales, i_item_id limit 100; -- query 57 with v1 as( select i_category, i_brand, cc_name, d_year, d_moy, sum(cs_sales_price) sum_sales, avg(sum(cs_sales_price)) over (partition by i_category, i_brand, cc_name, d_year) avg_monthly_sales, rank() over (partition by i_category, i_brand, cc_name order by d_year, d_moy) rn from item, catalog_sales, date_dim, call_center where cs_item_sk = i_item_sk and cs_sold_date_sk = d_date_sk and cc_call_center_sk= cs_call_center_sk and ( d_year = 1999 or ( d_year = 1999-1 and d_moy =12) or ( d_year = 1999+1 and d_moy =1) ) group by i_category, i_brand, cc_name , d_year, d_moy), v2 as( select v1.i_category, v1.i_brand, v1.cc_name ,v1.d_year, v1.d_moy ,v1.avg_monthly_sales ,v1.sum_sales, v1_lag.sum_sales psum, v1_lead.sum_sales nsum from v1, v1 v1_lag, v1 v1_lead where v1.i_category = v1_lag.i_category and v1.i_category = v1_lead.i_category and v1.i_brand = v1_lag.i_brand and v1.i_brand = v1_lead.i_brand and v1. cc_name = v1_lag. cc_name and v1. cc_name = v1_lead. cc_name and v1.rn = v1_lag.rn + 1 and v1.rn = v1_lead.rn - 1) select * from v2 where d_year = 1999 and avg_monthly_sales > 0 and case when avg_monthly_sales > 0 then abs(sum_sales - avg_monthly_sales) / avg_monthly_sales else null end > 0.1 order by sum_sales - avg_monthly_sales, cc_name limit 100; -- query 58 with ss_items as (select i_item_id item_id ,sum(ss_ext_sales_price) ss_item_rev from store_sales ,item ,date_dim where ss_item_sk = i_item_sk and d_date in (select d_date from date_dim where d_week_seq = (select d_week_seq from date_dim where d_date = '2000-01-03')) and ss_sold_date_sk = d_date_sk group by i_item_id), cs_items as (select i_item_id item_id ,sum(cs_ext_sales_price) cs_item_rev from catalog_sales ,item ,date_dim where cs_item_sk = i_item_sk and d_date in (select d_date from date_dim where d_week_seq = (select d_week_seq from date_dim where d_date = '2000-01-03')) and cs_sold_date_sk = d_date_sk group by i_item_id), ws_items as (select i_item_id item_id ,sum(ws_ext_sales_price) ws_item_rev from web_sales ,item ,date_dim where ws_item_sk = i_item_sk and d_date in (select d_date from date_dim where d_week_seq =(select d_week_seq from date_dim where d_date = '2000-01-03')) and ws_sold_date_sk = d_date_sk group by i_item_id) select ss_items.item_id ,ss_item_rev ,ss_item_rev/((ss_item_rev+cs_item_rev+ws_item_rev)/3) * 100 ss_dev ,cs_item_rev ,cs_item_rev/((ss_item_rev+cs_item_rev+ws_item_rev)/3) * 100 cs_dev ,ws_item_rev ,ws_item_rev/((ss_item_rev+cs_item_rev+ws_item_rev)/3) * 100 ws_dev ,(ss_item_rev+cs_item_rev+ws_item_rev)/3 average from ss_items,cs_items,ws_items where ss_items.item_id=cs_items.item_id and ss_items.item_id=ws_items.item_id and ss_item_rev between 0.9 * cs_item_rev and 1.1 * cs_item_rev and ss_item_rev between 0.9 * ws_item_rev and 1.1 * ws_item_rev and cs_item_rev between 0.9 * ss_item_rev and 1.1 * ss_item_rev and cs_item_rev between 0.9 * ws_item_rev and 1.1 * ws_item_rev and ws_item_rev between 0.9 * ss_item_rev and 1.1 * ss_item_rev and ws_item_rev between 0.9 * cs_item_rev and 1.1 * cs_item_rev order by item_id ,ss_item_rev limit 100; -- query 59 with wss as (select d_week_seq, ss_store_sk, sum(case when (d_day_name='Sunday') then ss_sales_price else null end) sun_sales, sum(case when (d_day_name='Monday') then ss_sales_price else null end) mon_sales, sum(case when (d_day_name='Tuesday') then ss_sales_price else null end) tue_sales, sum(case when (d_day_name='Wednesday') then ss_sales_price else null end) wed_sales, sum(case when (d_day_name='Thursday') then ss_sales_price else null end) thu_sales, sum(case when (d_day_name='Friday') then ss_sales_price else null end) fri_sales, sum(case when (d_day_name='Saturday') then ss_sales_price else null end) sat_sales from store_sales,date_dim where d_date_sk = ss_sold_date_sk group by d_week_seq,ss_store_sk ) select s_store_name1,s_store_id1,d_week_seq1 ,sun_sales1/sun_sales2,mon_sales1/mon_sales2 ,tue_sales1/tue_sales2,wed_sales1/wed_sales2,thu_sales1/thu_sales2 ,fri_sales1/fri_sales2,sat_sales1/sat_sales2 from (select s_store_name s_store_name1,wss.d_week_seq d_week_seq1 ,s_store_id s_store_id1,sun_sales sun_sales1 ,mon_sales mon_sales1,tue_sales tue_sales1 ,wed_sales wed_sales1,thu_sales thu_sales1 ,fri_sales fri_sales1,sat_sales sat_sales1 from wss,store,date_dim d where d.d_week_seq = wss.d_week_seq and ss_store_sk = s_store_sk and d_month_seq between 1212 and 1212 + 11) y, (select s_store_name s_store_name2,wss.d_week_seq d_week_seq2 ,s_store_id s_store_id2,sun_sales sun_sales2 ,mon_sales mon_sales2,tue_sales tue_sales2 ,wed_sales wed_sales2,thu_sales thu_sales2 ,fri_sales fri_sales2,sat_sales sat_sales2 from wss,store,date_dim d where d.d_week_seq = wss.d_week_seq and ss_store_sk = s_store_sk and d_month_seq between 1212+ 12 and 1212 + 23) x where s_store_id1=s_store_id2 and d_week_seq1=d_week_seq2-52 order by s_store_name1,s_store_id1,d_week_seq1 limit 100; -- query 60 with ss as ( select i_item_id,sum(ss_ext_sales_price) total_sales from store_sales, date_dim, customer_address, item where i_item_id in (select i_item_id from item where i_category in ('Music')) and ss_item_sk = i_item_sk and ss_sold_date_sk = d_date_sk and d_year = 1998 and d_moy = 9 and ss_addr_sk = ca_address_sk and ca_gmt_offset = -5 group by i_item_id), cs as ( select i_item_id,sum(cs_ext_sales_price) total_sales from catalog_sales, date_dim, customer_address, item where i_item_id in (select i_item_id from item where i_category in ('Music')) and cs_item_sk = i_item_sk and cs_sold_date_sk = d_date_sk and d_year = 1998 and d_moy = 9 and cs_bill_addr_sk = ca_address_sk and ca_gmt_offset = -5 group by i_item_id), ws as ( select i_item_id,sum(ws_ext_sales_price) total_sales from web_sales, date_dim, customer_address, item where i_item_id in (select i_item_id from item where i_category in ('Music')) and ws_item_sk = i_item_sk and ws_sold_date_sk = d_date_sk and d_year = 1998 and d_moy = 9 and ws_bill_addr_sk = ca_address_sk and ca_gmt_offset = -5 group by i_item_id) select i_item_id ,sum(total_sales) total_sales from (select * from ss union all select * from cs union all select * from ws) tmp1 group by i_item_id order by i_item_id ,total_sales limit 100; -- query 61 select promotions,total,cast(promotions as decimal(15,4))/cast(total as decimal(15,4))*100 from (select sum(ss_ext_sales_price) promotions from store_sales ,store ,promotion ,date_dim ,customer ,customer_address ,item where ss_sold_date_sk = d_date_sk and ss_store_sk = s_store_sk and ss_promo_sk = p_promo_sk and ss_customer_sk= c_customer_sk and ca_address_sk = c_current_addr_sk and ss_item_sk = i_item_sk and ca_gmt_offset = -5 and i_category = 'Jewelry' and (p_channel_dmail = 'Y' or p_channel_email = 'Y' or p_channel_tv = 'Y') and s_gmt_offset = -5 and d_year = 1998 and d_moy = 11) promotional_sales, (select sum(ss_ext_sales_price) total from store_sales ,store ,date_dim ,customer ,customer_address ,item where ss_sold_date_sk = d_date_sk and ss_store_sk = s_store_sk and ss_customer_sk= c_customer_sk and ca_address_sk = c_current_addr_sk and ss_item_sk = i_item_sk and ca_gmt_offset = -5 and i_category = 'Jewelry' and s_gmt_offset = -5 and d_year = 1998 and d_moy = 11) all_sales order by promotions, total limit 100; -- query 62 select substr(w_warehouse_name,1,20) ,sm_type ,web_name ,sum(case when (ws_ship_date_sk - ws_sold_date_sk <= 30 ) then 1 else 0 end) as "30 days" ,sum(case when (ws_ship_date_sk - ws_sold_date_sk > 30) and (ws_ship_date_sk - ws_sold_date_sk <= 60) then 1 else 0 end ) as "31-60 days" ,sum(case when (ws_ship_date_sk - ws_sold_date_sk > 60) and (ws_ship_date_sk - ws_sold_date_sk <= 90) then 1 else 0 end) as "61-90 days" ,sum(case when (ws_ship_date_sk - ws_sold_date_sk > 90) and (ws_ship_date_sk - ws_sold_date_sk <= 120) then 1 else 0 end) as "91-120 days" ,sum(case when (ws_ship_date_sk - ws_sold_date_sk > 120) then 1 else 0 end) as ">120 days" from web_sales ,warehouse ,ship_mode ,web_site ,date_dim where d_month_seq between 1200 and 1200 + 11 and ws_ship_date_sk = d_date_sk and ws_warehouse_sk = w_warehouse_sk and ws_ship_mode_sk = sm_ship_mode_sk and ws_web_site_sk = web_site_sk group by substr(w_warehouse_name,1,20) ,sm_type ,web_name order by substr(w_warehouse_name,1,20) ,sm_type ,web_name limit 100; -- query 63 select * from (select i_manager_id ,sum(ss_sales_price) sum_sales ,avg(sum(ss_sales_price)) over (partition by i_manager_id) avg_monthly_sales from item ,store_sales ,date_dim ,store where ss_item_sk = i_item_sk and ss_sold_date_sk = d_date_sk and ss_store_sk = s_store_sk and d_month_seq in (1200,1200+1,1200+2,1200+3,1200+4,1200+5,1200+6,1200+7,1200+8,1200+9,1200+10,1200+11) and (( i_category in ('Books','Children','Electronics') and i_class in ('personal','portable','reference','self-help') and i_brand in ('scholaramalgamalg #14','scholaramalgamalg #7', 'exportiunivamalg #9','scholaramalgamalg #9')) or( i_category in ('Women','Music','Men') and i_class in ('accessories','classical','fragrances','pants') and i_brand in ('amalgimporto #1','edu packscholar #1','exportiimporto #1', 'importoamalg #1'))) group by i_manager_id, d_moy) tmp1 where case when avg_monthly_sales > 0 then abs (sum_sales - avg_monthly_sales) / avg_monthly_sales else null end > 0.1 order by i_manager_id ,avg_monthly_sales ,sum_sales limit 100; -- query 64 with cs_ui as (select cs_item_sk ,sum(cs_ext_list_price) as sale,sum(cr_refunded_cash+cr_reversed_charge+cr_store_credit) as refund from catalog_sales ,catalog_returns where cs_item_sk = cr_item_sk and cs_order_number = cr_order_number group by cs_item_sk having sum(cs_ext_list_price)>2*sum(cr_refunded_cash+cr_reversed_charge+cr_store_credit)), cross_sales as (select i_product_name product_name ,i_item_sk item_sk ,s_store_name store_name ,s_zip store_zip ,ad1.ca_street_number b_street_number ,ad1.ca_street_name b_street_name ,ad1.ca_city b_city ,ad1.ca_zip b_zip ,ad2.ca_street_number c_street_number ,ad2.ca_street_name c_street_name ,ad2.ca_city c_city ,ad2.ca_zip c_zip ,d1.d_year as syear ,d2.d_year as fsyear ,d3.d_year s2year ,count(*) cnt ,sum(ss_wholesale_cost) s1 ,sum(ss_list_price) s2 ,sum(ss_coupon_amt) s3 FROM store_sales ,store_returns ,cs_ui ,date_dim d1 ,date_dim d2 ,date_dim d3 ,store ,customer ,customer_demographics cd1 ,customer_demographics cd2 ,promotion ,household_demographics hd1 ,household_demographics hd2 ,customer_address ad1 ,customer_address ad2 ,income_band ib1 ,income_band ib2 ,item WHERE ss_store_sk = s_store_sk AND ss_sold_date_sk = d1.d_date_sk AND ss_customer_sk = c_customer_sk AND ss_cdemo_sk= cd1.cd_demo_sk AND ss_hdemo_sk = hd1.hd_demo_sk AND ss_addr_sk = ad1.ca_address_sk and ss_item_sk = i_item_sk and ss_item_sk = sr_item_sk and ss_ticket_number = sr_ticket_number and ss_item_sk = cs_ui.cs_item_sk and c_current_cdemo_sk = cd2.cd_demo_sk AND c_current_hdemo_sk = hd2.hd_demo_sk AND c_current_addr_sk = ad2.ca_address_sk and c_first_sales_date_sk = d2.d_date_sk and c_first_shipto_date_sk = d3.d_date_sk and ss_promo_sk = p_promo_sk and hd1.hd_income_band_sk = ib1.ib_income_band_sk and hd2.hd_income_band_sk = ib2.ib_income_band_sk and cd1.cd_marital_status <> cd2.cd_marital_status and i_color in ('purple','burlywood','indian','spring','floral','medium') and i_current_price between 64 and 64 + 10 and i_current_price between 64 + 1 and 64 + 15 group by i_product_name ,i_item_sk ,s_store_name ,s_zip ,ad1.ca_street_number ,ad1.ca_street_name ,ad1.ca_city ,ad1.ca_zip ,ad2.ca_street_number ,ad2.ca_street_name ,ad2.ca_city ,ad2.ca_zip ,d1.d_year ,d2.d_year ,d3.d_year ) select cs1.product_name ,cs1.store_name ,cs1.store_zip ,cs1.b_street_number ,cs1.b_street_name ,cs1.b_city ,cs1.b_zip ,cs1.c_street_number ,cs1.c_street_name ,cs1.c_city ,cs1.c_zip ,cs1.syear ,cs1.cnt ,cs1.s1 as s11 ,cs1.s2 as s21 ,cs1.s3 as s31 ,cs2.s1 as s12 ,cs2.s2 as s22 ,cs2.s3 as s32 ,cs2.syear ,cs2.cnt from cross_sales cs1,cross_sales cs2 where cs1.item_sk=cs2.item_sk and cs1.syear = 1999 and cs2.syear = 1999 + 1 and cs2.cnt <= cs1.cnt and cs1.store_name = cs2.store_name and cs1.store_zip = cs2.store_zip order by cs1.product_name ,cs1.store_name ,cs2.cnt ,cs1.s1 ,cs2.s1; -- query 65 select s_store_name, i_item_desc, sc.revenue, i_current_price, i_wholesale_cost, i_brand from store, item, (select ss_store_sk, avg(revenue) as ave from (select ss_store_sk, ss_item_sk, sum(ss_sales_price) as revenue from store_sales, date_dim where ss_sold_date_sk = d_date_sk and d_month_seq between 1176 and 1176+11 group by ss_store_sk, ss_item_sk) sa group by ss_store_sk) sb, (select ss_store_sk, ss_item_sk, sum(ss_sales_price) as revenue from store_sales, date_dim where ss_sold_date_sk = d_date_sk and d_month_seq between 1176 and 1176+11 group by ss_store_sk, ss_item_sk) sc where sb.ss_store_sk = sc.ss_store_sk and sc.revenue <= 0.1 * sb.ave and s_store_sk = sc.ss_store_sk and i_item_sk = sc.ss_item_sk order by s_store_name, i_item_desc limit 100; -- query 66 select w_warehouse_name ,w_warehouse_sq_ft ,w_city ,w_county ,w_state ,w_country ,ship_carriers ,year ,sum(jan_sales) as jan_sales ,sum(feb_sales) as feb_sales ,sum(mar_sales) as mar_sales ,sum(apr_sales) as apr_sales ,sum(may_sales) as may_sales ,sum(jun_sales) as jun_sales ,sum(jul_sales) as jul_sales ,sum(aug_sales) as aug_sales ,sum(sep_sales) as sep_sales ,sum(oct_sales) as oct_sales ,sum(nov_sales) as nov_sales ,sum(dec_sales) as dec_sales ,sum(jan_sales/w_warehouse_sq_ft) as jan_sales_per_sq_foot ,sum(feb_sales/w_warehouse_sq_ft) as feb_sales_per_sq_foot ,sum(mar_sales/w_warehouse_sq_ft) as mar_sales_per_sq_foot ,sum(apr_sales/w_warehouse_sq_ft) as apr_sales_per_sq_foot ,sum(may_sales/w_warehouse_sq_ft) as may_sales_per_sq_foot ,sum(jun_sales/w_warehouse_sq_ft) as jun_sales_per_sq_foot ,sum(jul_sales/w_warehouse_sq_ft) as jul_sales_per_sq_foot ,sum(aug_sales/w_warehouse_sq_ft) as aug_sales_per_sq_foot ,sum(sep_sales/w_warehouse_sq_ft) as sep_sales_per_sq_foot ,sum(oct_sales/w_warehouse_sq_ft) as oct_sales_per_sq_foot ,sum(nov_sales/w_warehouse_sq_ft) as nov_sales_per_sq_foot ,sum(dec_sales/w_warehouse_sq_ft) as dec_sales_per_sq_foot ,sum(jan_net) as jan_net ,sum(feb_net) as feb_net ,sum(mar_net) as mar_net ,sum(apr_net) as apr_net ,sum(may_net) as may_net ,sum(jun_net) as jun_net ,sum(jul_net) as jul_net ,sum(aug_net) as aug_net ,sum(sep_net) as sep_net ,sum(oct_net) as oct_net ,sum(nov_net) as nov_net ,sum(dec_net) as dec_net from ( select w_warehouse_name ,w_warehouse_sq_ft ,w_city ,w_county ,w_state ,w_country ,'DHL' || ',' || 'BARIAN' as ship_carriers ,d_year as year ,sum(case when d_moy = 1 then ws_ext_sales_price* ws_quantity else 0 end) as jan_sales ,sum(case when d_moy = 2 then ws_ext_sales_price* ws_quantity else 0 end) as feb_sales ,sum(case when d_moy = 3 then ws_ext_sales_price* ws_quantity else 0 end) as mar_sales ,sum(case when d_moy = 4 then ws_ext_sales_price* ws_quantity else 0 end) as apr_sales ,sum(case when d_moy = 5 then ws_ext_sales_price* ws_quantity else 0 end) as may_sales ,sum(case when d_moy = 6 then ws_ext_sales_price* ws_quantity else 0 end) as jun_sales ,sum(case when d_moy = 7 then ws_ext_sales_price* ws_quantity else 0 end) as jul_sales ,sum(case when d_moy = 8 then ws_ext_sales_price* ws_quantity else 0 end) as aug_sales ,sum(case when d_moy = 9 then ws_ext_sales_price* ws_quantity else 0 end) as sep_sales ,sum(case when d_moy = 10 then ws_ext_sales_price* ws_quantity else 0 end) as oct_sales ,sum(case when d_moy = 11 then ws_ext_sales_price* ws_quantity else 0 end) as nov_sales ,sum(case when d_moy = 12 then ws_ext_sales_price* ws_quantity else 0 end) as dec_sales ,sum(case when d_moy = 1 then ws_net_paid * ws_quantity else 0 end) as jan_net ,sum(case when d_moy = 2 then ws_net_paid * ws_quantity else 0 end) as feb_net ,sum(case when d_moy = 3 then ws_net_paid * ws_quantity else 0 end) as mar_net ,sum(case when d_moy = 4 then ws_net_paid * ws_quantity else 0 end) as apr_net ,sum(case when d_moy = 5 then ws_net_paid * ws_quantity else 0 end) as may_net ,sum(case when d_moy = 6 then ws_net_paid * ws_quantity else 0 end) as jun_net ,sum(case when d_moy = 7 then ws_net_paid * ws_quantity else 0 end) as jul_net ,sum(case when d_moy = 8 then ws_net_paid * ws_quantity else 0 end) as aug_net ,sum(case when d_moy = 9 then ws_net_paid * ws_quantity else 0 end) as sep_net ,sum(case when d_moy = 10 then ws_net_paid * ws_quantity else 0 end) as oct_net ,sum(case when d_moy = 11 then ws_net_paid * ws_quantity else 0 end) as nov_net ,sum(case when d_moy = 12 then ws_net_paid * ws_quantity else 0 end) as dec_net from web_sales ,warehouse ,date_dim ,time_dim ,ship_mode where ws_warehouse_sk = w_warehouse_sk and ws_sold_date_sk = d_date_sk and ws_sold_time_sk = t_time_sk and ws_ship_mode_sk = sm_ship_mode_sk and d_year = 2001 and t_time between 30838 and 30838+28800 and sm_carrier in ('DHL','BARIAN') group by w_warehouse_name ,w_warehouse_sq_ft ,w_city ,w_county ,w_state ,w_country ,d_year union all select w_warehouse_name ,w_warehouse_sq_ft ,w_city ,w_county ,w_state ,w_country ,'DHL' || ',' || 'BARIAN' as ship_carriers ,d_year as year ,sum(case when d_moy = 1 then cs_sales_price* cs_quantity else 0 end) as jan_sales ,sum(case when d_moy = 2 then cs_sales_price* cs_quantity else 0 end) as feb_sales ,sum(case when d_moy = 3 then cs_sales_price* cs_quantity else 0 end) as mar_sales ,sum(case when d_moy = 4 then cs_sales_price* cs_quantity else 0 end) as apr_sales ,sum(case when d_moy = 5 then cs_sales_price* cs_quantity else 0 end) as may_sales ,sum(case when d_moy = 6 then cs_sales_price* cs_quantity else 0 end) as jun_sales ,sum(case when d_moy = 7 then cs_sales_price* cs_quantity else 0 end) as jul_sales ,sum(case when d_moy = 8 then cs_sales_price* cs_quantity else 0 end) as aug_sales ,sum(case when d_moy = 9 then cs_sales_price* cs_quantity else 0 end) as sep_sales ,sum(case when d_moy = 10 then cs_sales_price* cs_quantity else 0 end) as oct_sales ,sum(case when d_moy = 11 then cs_sales_price* cs_quantity else 0 end) as nov_sales ,sum(case when d_moy = 12 then cs_sales_price* cs_quantity else 0 end) as dec_sales ,sum(case when d_moy = 1 then cs_net_paid_inc_tax * cs_quantity else 0 end) as jan_net ,sum(case when d_moy = 2 then cs_net_paid_inc_tax * cs_quantity else 0 end) as feb_net ,sum(case when d_moy = 3 then cs_net_paid_inc_tax * cs_quantity else 0 end) as mar_net ,sum(case when d_moy = 4 then cs_net_paid_inc_tax * cs_quantity else 0 end) as apr_net ,sum(case when d_moy = 5 then cs_net_paid_inc_tax * cs_quantity else 0 end) as may_net ,sum(case when d_moy = 6 then cs_net_paid_inc_tax * cs_quantity else 0 end) as jun_net ,sum(case when d_moy = 7 then cs_net_paid_inc_tax * cs_quantity else 0 end) as jul_net ,sum(case when d_moy = 8 then cs_net_paid_inc_tax * cs_quantity else 0 end) as aug_net ,sum(case when d_moy = 9 then cs_net_paid_inc_tax * cs_quantity else 0 end) as sep_net ,sum(case when d_moy = 10 then cs_net_paid_inc_tax * cs_quantity else 0 end) as oct_net ,sum(case when d_moy = 11 then cs_net_paid_inc_tax * cs_quantity else 0 end) as nov_net ,sum(case when d_moy = 12 then cs_net_paid_inc_tax * cs_quantity else 0 end) as dec_net from catalog_sales ,warehouse ,date_dim ,time_dim ,ship_mode where cs_warehouse_sk = w_warehouse_sk and cs_sold_date_sk = d_date_sk and cs_sold_time_sk = t_time_sk and cs_ship_mode_sk = sm_ship_mode_sk and d_year = 2001 and t_time between 30838 AND 30838+28800 and sm_carrier in ('DHL','BARIAN') group by w_warehouse_name ,w_warehouse_sq_ft ,w_city ,w_county ,w_state ,w_country ,d_year ) x group by w_warehouse_name ,w_warehouse_sq_ft ,w_city ,w_county ,w_state ,w_country ,ship_carriers ,year order by w_warehouse_name limit 100; -- query 67 select * from (select i_category ,i_class ,i_brand ,i_product_name ,d_year ,d_qoy ,d_moy ,s_store_id ,sumsales ,rank() over (partition by i_category order by sumsales desc) rk from (select i_category ,i_class ,i_brand ,i_product_name ,d_year ,d_qoy ,d_moy ,s_store_id ,sum(coalesce(ss_sales_price*ss_quantity,0)) sumsales from store_sales ,date_dim ,store ,item where ss_sold_date_sk=d_date_sk and ss_item_sk=i_item_sk and ss_store_sk = s_store_sk and d_month_seq between 1200 and 1200+11 group by rollup(i_category, i_class, i_brand, i_product_name, d_year, d_qoy, d_moy,s_store_id))dw1) dw2 where rk <= 100 order by i_category ,i_class ,i_brand ,i_product_name ,d_year ,d_qoy ,d_moy ,s_store_id ,sumsales ,rk limit 100; -- query 68 select c_last_name ,c_first_name ,ca_city ,bought_city ,ss_ticket_number ,extended_price ,extended_tax ,list_price from (select ss_ticket_number ,ss_customer_sk ,ca_city bought_city ,sum(ss_ext_sales_price) extended_price ,sum(ss_ext_list_price) list_price ,sum(ss_ext_tax) extended_tax from store_sales ,date_dim ,store ,household_demographics ,customer_address where store_sales.ss_sold_date_sk = date_dim.d_date_sk and store_sales.ss_store_sk = store.s_store_sk and store_sales.ss_hdemo_sk = household_demographics.hd_demo_sk and store_sales.ss_addr_sk = customer_address.ca_address_sk and date_dim.d_dom between 1 and 2 and (household_demographics.hd_dep_count = 4 or household_demographics.hd_vehicle_count= 3) and date_dim.d_year in (1999,1999+1,1999+2) and store.s_city in ('Fairview','Midway') group by ss_ticket_number ,ss_customer_sk ,ss_addr_sk,ca_city) dn ,customer ,customer_address current_addr where ss_customer_sk = c_customer_sk and customer.c_current_addr_sk = current_addr.ca_address_sk and current_addr.ca_city <> bought_city order by c_last_name ,ss_ticket_number limit 100; -- query 69 select cd_gender, cd_marital_status, cd_education_status, count(*) cnt1, cd_purchase_estimate, count(*) cnt2, cd_credit_rating, count(*) cnt3 from customer c,customer_address ca,customer_demographics where c.c_current_addr_sk = ca.ca_address_sk and ca_state in ('KY', 'GA', 'NM') and cd_demo_sk = c.c_current_cdemo_sk and exists (select * from store_sales,date_dim where c.c_customer_sk = ss_customer_sk and ss_sold_date_sk = d_date_sk and d_year = 2001 and d_moy between 4 and 4+2) and (not exists (select * from web_sales,date_dim where c.c_customer_sk = ws_bill_customer_sk and ws_sold_date_sk = d_date_sk and d_year = 2001 and d_moy between 4 and 4+2) and not exists (select * from catalog_sales,date_dim where c.c_customer_sk = cs_ship_customer_sk and cs_sold_date_sk = d_date_sk and d_year = 2001 and d_moy between 4 and 4+2)) group by cd_gender, cd_marital_status, cd_education_status, cd_purchase_estimate, cd_credit_rating order by cd_gender, cd_marital_status, cd_education_status, cd_purchase_estimate, cd_credit_rating limit 100; -- query 70 select sum(ss_net_profit) as total_sum ,s_state ,s_county ,grouping(s_state)+grouping(s_county) as lochierarchy ,rank() over ( partition by grouping(s_state)+grouping(s_county), case when grouping(s_county) = 0 then s_state end order by sum(ss_net_profit) desc) as rank_within_parent from store_sales ,date_dim d1 ,store where d1.d_month_seq between 1200 and 1200+11 and d1.d_date_sk = ss_sold_date_sk and s_store_sk = ss_store_sk and s_state in ( select s_state from (select s_state as s_state, rank() over ( partition by s_state order by sum(ss_net_profit) desc) as ranking from store_sales, store, date_dim where d_month_seq between 1200 and 1200+11 and d_date_sk = ss_sold_date_sk and s_store_sk = ss_store_sk group by s_state ) tmp1 where ranking <= 5 ) group by rollup(s_state,s_county) order by lochierarchy desc ,case when lochierarchy = 0 then s_state end ,rank_within_parent limit 100; -- query 71 select i_brand_id brand_id, i_brand brand,t_hour,t_minute, sum(ext_price) ext_price from item, (select ws_ext_sales_price as ext_price, ws_sold_date_sk as sold_date_sk, ws_item_sk as sold_item_sk, ws_sold_time_sk as time_sk from web_sales,date_dim where d_date_sk = ws_sold_date_sk and d_moy=11 and d_year=1999 union all select cs_ext_sales_price as ext_price, cs_sold_date_sk as sold_date_sk, cs_item_sk as sold_item_sk, cs_sold_time_sk as time_sk from catalog_sales,date_dim where d_date_sk = cs_sold_date_sk and d_moy=11 and d_year=1999 union all select ss_ext_sales_price as ext_price, ss_sold_date_sk as sold_date_sk, ss_item_sk as sold_item_sk, ss_sold_time_sk as time_sk from store_sales,date_dim where d_date_sk = ss_sold_date_sk and d_moy=11 and d_year=1999 ) tmp,time_dim where sold_item_sk = i_item_sk and i_manager_id=1 and time_sk = t_time_sk and (t_meal_time = 'breakfast' or t_meal_time = 'dinner') group by i_brand, i_brand_id,t_hour,t_minute order by ext_price desc, i_brand_id ; -- query 72 select i_item_desc ,w_warehouse_name ,d1.d_week_seq ,sum(case when p_promo_sk is null then 1 else 0 end) no_promo ,sum(case when p_promo_sk is not null then 1 else 0 end) promo ,count(*) total_cnt from catalog_sales join inventory on (cs_item_sk = inv_item_sk) join warehouse on (w_warehouse_sk=inv_warehouse_sk) join item on (i_item_sk = cs_item_sk) join customer_demographics on (cs_bill_cdemo_sk = cd_demo_sk) join household_demographics on (cs_bill_hdemo_sk = hd_demo_sk) join date_dim d1 on (cs_sold_date_sk = d1.d_date_sk) join date_dim d2 on (inv_date_sk = d2.d_date_sk) join date_dim d3 on (cs_ship_date_sk = d3.d_date_sk) left outer join promotion on (cs_promo_sk=p_promo_sk) left outer join catalog_returns on (cr_item_sk = cs_item_sk and cr_order_number = cs_order_number) where d1.d_week_seq = d2.d_week_seq and inv_quantity_on_hand < cs_quantity and d3.d_date > d1.d_date + 5 and hd_buy_potential = '>10000' and d1.d_year = 1999 and cd_marital_status = 'D' group by i_item_desc,w_warehouse_name,d1.d_week_seq order by total_cnt desc, i_item_desc, w_warehouse_name, d_week_seq limit 100; -- query 73 select c_last_name ,c_first_name ,c_salutation ,c_preferred_cust_flag ,ss_ticket_number ,cnt from (select ss_ticket_number ,ss_customer_sk ,count(*) cnt from store_sales,date_dim,store,household_demographics where store_sales.ss_sold_date_sk = date_dim.d_date_sk and store_sales.ss_store_sk = store.s_store_sk and store_sales.ss_hdemo_sk = household_demographics.hd_demo_sk and date_dim.d_dom between 1 and 2 and (household_demographics.hd_buy_potential = '>10000' or household_demographics.hd_buy_potential = 'Unknown') and household_demographics.hd_vehicle_count > 0 and case when household_demographics.hd_vehicle_count > 0 then household_demographics.hd_dep_count/ household_demographics.hd_vehicle_count else null end > 1 and date_dim.d_year in (1999,1999+1,1999+2) and store.s_county in ('Williamson County', 'Franklin Parish', 'Bronx County', 'Orange County') group by ss_ticket_number,ss_customer_sk) dj,customer where ss_customer_sk = c_customer_sk and cnt between 1 and 5 order by cnt desc, c_last_name asc; -- query 74 with year_total as ( select c_customer_id customer_id ,c_first_name customer_first_name ,c_last_name customer_last_name ,d_year as year ,sum(ss_net_paid) year_total ,'s' sale_type from customer ,store_sales ,date_dim where c_customer_sk = ss_customer_sk and ss_sold_date_sk = d_date_sk and d_year in (2001,2001+1) group by c_customer_id ,c_first_name ,c_last_name ,d_year union all select c_customer_id customer_id ,c_first_name customer_first_name ,c_last_name customer_last_name ,d_year as year ,sum(ws_net_paid) year_total ,'w' sale_type from customer ,web_sales ,date_dim where c_customer_sk = ws_bill_customer_sk and ws_sold_date_sk = d_date_sk and d_year in (2001,2001+1) group by c_customer_id ,c_first_name ,c_last_name ,d_year ) select t_s_secyear.customer_id, t_s_secyear.customer_first_name, t_s_secyear.customer_last_name from year_total t_s_firstyear ,year_total t_s_secyear ,year_total t_w_firstyear ,year_total t_w_secyear where t_s_secyear.customer_id = t_s_firstyear.customer_id and t_s_firstyear.customer_id = t_w_secyear.customer_id and t_s_firstyear.customer_id = t_w_firstyear.customer_id and t_s_firstyear.sale_type = 's' and t_w_firstyear.sale_type = 'w' and t_s_secyear.sale_type = 's' and t_w_secyear.sale_type = 'w' and t_s_firstyear.year = 2001 and t_s_secyear.year = 2001+1 and t_w_firstyear.year = 2001 and t_w_secyear.year = 2001+1 and t_s_firstyear.year_total > 0 and t_w_firstyear.year_total > 0 and case when t_w_firstyear.year_total > 0 then t_w_secyear.year_total / t_w_firstyear.year_total else null end > case when t_s_firstyear.year_total > 0 then t_s_secyear.year_total / t_s_firstyear.year_total else null end order by 1, 1, 1 limit 100; -- query 75 WITH all_sales AS ( SELECT d_year ,i_brand_id ,i_class_id ,i_category_id ,i_manufact_id ,SUM(sales_cnt) AS sales_cnt ,SUM(sales_amt) AS sales_amt FROM (SELECT d_year ,i_brand_id ,i_class_id ,i_category_id ,i_manufact_id ,cs_quantity - COALESCE(cr_return_quantity,0) AS sales_cnt ,cs_ext_sales_price - COALESCE(cr_return_amount,0.0) AS sales_amt FROM catalog_sales JOIN item ON i_item_sk=cs_item_sk JOIN date_dim ON d_date_sk=cs_sold_date_sk LEFT JOIN catalog_returns ON (cs_order_number=cr_order_number AND cs_item_sk=cr_item_sk) WHERE i_category='Books' UNION SELECT d_year ,i_brand_id ,i_class_id ,i_category_id ,i_manufact_id ,ss_quantity - COALESCE(sr_return_quantity,0) AS sales_cnt ,ss_ext_sales_price - COALESCE(sr_return_amt,0.0) AS sales_amt FROM store_sales JOIN item ON i_item_sk=ss_item_sk JOIN date_dim ON d_date_sk=ss_sold_date_sk LEFT JOIN store_returns ON (ss_ticket_number=sr_ticket_number AND ss_item_sk=sr_item_sk) WHERE i_category='Books' UNION SELECT d_year ,i_brand_id ,i_class_id ,i_category_id ,i_manufact_id ,ws_quantity - COALESCE(wr_return_quantity,0) AS sales_cnt ,ws_ext_sales_price - COALESCE(wr_return_amt,0.0) AS sales_amt FROM web_sales JOIN item ON i_item_sk=ws_item_sk JOIN date_dim ON d_date_sk=ws_sold_date_sk LEFT JOIN web_returns ON (ws_order_number=wr_order_number AND ws_item_sk=wr_item_sk) WHERE i_category='Books') sales_detail GROUP BY d_year, i_brand_id, i_class_id, i_category_id, i_manufact_id) SELECT prev_yr.d_year AS prev_year ,curr_yr.d_year AS year ,curr_yr.i_brand_id ,curr_yr.i_class_id ,curr_yr.i_category_id ,curr_yr.i_manufact_id ,prev_yr.sales_cnt AS prev_yr_cnt ,curr_yr.sales_cnt AS curr_yr_cnt ,curr_yr.sales_cnt-prev_yr.sales_cnt AS sales_cnt_diff ,curr_yr.sales_amt-prev_yr.sales_amt AS sales_amt_diff FROM all_sales curr_yr, all_sales prev_yr WHERE curr_yr.i_brand_id=prev_yr.i_brand_id AND curr_yr.i_class_id=prev_yr.i_class_id AND curr_yr.i_category_id=prev_yr.i_category_id AND curr_yr.i_manufact_id=prev_yr.i_manufact_id AND curr_yr.d_year=2002 AND prev_yr.d_year=2002-1 AND CAST(curr_yr.sales_cnt AS DECIMAL(17,2))/CAST(prev_yr.sales_cnt AS DECIMAL(17,2))<0.9 ORDER BY sales_cnt_diff,sales_amt_diff limit 100; -- query 76 select channel, col_name, d_year, d_qoy, i_category, COUNT(*) sales_cnt, SUM(ext_sales_price) sales_amt FROM ( SELECT 'store' as channel, 'ss_store_sk' col_name, d_year, d_qoy, i_category, ss_ext_sales_price ext_sales_price FROM store_sales, item, date_dim WHERE ss_store_sk IS NULL AND ss_sold_date_sk=d_date_sk AND ss_item_sk=i_item_sk UNION ALL SELECT 'web' as channel, 'ws_ship_customer_sk' col_name, d_year, d_qoy, i_category, ws_ext_sales_price ext_sales_price FROM web_sales, item, date_dim WHERE ws_ship_customer_sk IS NULL AND ws_sold_date_sk=d_date_sk AND ws_item_sk=i_item_sk UNION ALL SELECT 'catalog' as channel, 'cs_ship_addr_sk' col_name, d_year, d_qoy, i_category, cs_ext_sales_price ext_sales_price FROM catalog_sales, item, date_dim WHERE cs_ship_addr_sk IS NULL AND cs_sold_date_sk=d_date_sk AND cs_item_sk=i_item_sk) foo GROUP BY channel, col_name, d_year, d_qoy, i_category ORDER BY channel, col_name, d_year, d_qoy, i_category limit 100; -- query 77 with ss as (select s_store_sk, sum(ss_ext_sales_price) as sales, sum(ss_net_profit) as profit from store_sales, date_dim, store where ss_sold_date_sk = d_date_sk and d_date between cast('2000-08-23' as date) and date_add(cast('2000-08-23' as date), 30) and ss_store_sk = s_store_sk group by s_store_sk) , sr as (select s_store_sk, sum(sr_return_amt) as returns, sum(sr_net_loss) as profit_loss from store_returns, date_dim, store where sr_returned_date_sk = d_date_sk and d_date between cast('2000-08-23' as date) and date_add(cast('2000-08-23' as date), 30) and sr_store_sk = s_store_sk group by s_store_sk), cs as (select cs_call_center_sk, sum(cs_ext_sales_price) as sales, sum(cs_net_profit) as profit from catalog_sales, date_dim where cs_sold_date_sk = d_date_sk and d_date between cast('2000-08-23' as date) and date_add(cast('2000-08-23' as date), 30) group by cs_call_center_sk ), cr as (select cr_call_center_sk, sum(cr_return_amount) as returns, sum(cr_net_loss) as profit_loss from catalog_returns, date_dim where cr_returned_date_sk = d_date_sk and d_date between cast('2000-08-23' as date) and date_add(cast('2000-08-23' as date), 30) group by cr_call_center_sk ), ws as ( select wp_web_page_sk, sum(ws_ext_sales_price) as sales, sum(ws_net_profit) as profit from web_sales, date_dim, web_page where ws_sold_date_sk = d_date_sk and d_date between cast('2000-08-23' as date) and date_add(cast('2000-08-23' as date), 30) and ws_web_page_sk = wp_web_page_sk group by wp_web_page_sk), wr as (select wp_web_page_sk, sum(wr_return_amt) as returns, sum(wr_net_loss) as profit_loss from web_returns, date_dim, web_page where wr_returned_date_sk = d_date_sk and d_date between cast('2000-08-23' as date) and date_add(cast('2000-08-23' as date), 30) and wr_web_page_sk = wp_web_page_sk group by wp_web_page_sk) select channel , id , sum(sales) as sales , sum(returns) as returns , sum(profit) as profit from (select 'store channel' as channel , ss.s_store_sk as id , sales , coalesce(returns, 0) as returns , (profit - coalesce(profit_loss,0)) as profit from ss left join sr on ss.s_store_sk = sr.s_store_sk union all select 'catalog channel' as channel , cs_call_center_sk as id , sales , returns , (profit - profit_loss) as profit from cs , cr union all select 'web channel' as channel , ws.wp_web_page_sk as id , sales , coalesce(returns, 0) returns , (profit - coalesce(profit_loss,0)) as profit from ws left join wr on ws.wp_web_page_sk = wr.wp_web_page_sk ) x group by rollup (channel, id) order by channel ,id limit 100; -- query 78 with ws as (select d_year AS ws_sold_year, ws_item_sk, ws_bill_customer_sk ws_customer_sk, sum(ws_quantity) ws_qty, sum(ws_wholesale_cost) ws_wc, sum(ws_sales_price) ws_sp from web_sales left join web_returns on wr_order_number=ws_order_number and ws_item_sk=wr_item_sk join date_dim on ws_sold_date_sk = d_date_sk where wr_order_number is null group by d_year, ws_item_sk, ws_bill_customer_sk ), cs as (select d_year AS cs_sold_year, cs_item_sk, cs_bill_customer_sk cs_customer_sk, sum(cs_quantity) cs_qty, sum(cs_wholesale_cost) cs_wc, sum(cs_sales_price) cs_sp from catalog_sales left join catalog_returns on cr_order_number=cs_order_number and cs_item_sk=cr_item_sk join date_dim on cs_sold_date_sk = d_date_sk where cr_order_number is null group by d_year, cs_item_sk, cs_bill_customer_sk ), ss as (select d_year AS ss_sold_year, ss_item_sk, ss_customer_sk, sum(ss_quantity) ss_qty, sum(ss_wholesale_cost) ss_wc, sum(ss_sales_price) ss_sp from store_sales left join store_returns on sr_ticket_number=ss_ticket_number and ss_item_sk=sr_item_sk join date_dim on ss_sold_date_sk = d_date_sk where sr_ticket_number is null group by d_year, ss_item_sk, ss_customer_sk ) select ss_sold_year, ss_item_sk, ss_customer_sk, round(ss_qty/(coalesce(ws_qty,0)+coalesce(cs_qty,0)),2) ratio, ss_qty store_qty, ss_wc store_wholesale_cost, ss_sp store_sales_price, coalesce(ws_qty,0)+coalesce(cs_qty,0) other_chan_qty, coalesce(ws_wc,0)+coalesce(cs_wc,0) other_chan_wholesale_cost, coalesce(ws_sp,0)+coalesce(cs_sp,0) other_chan_sales_price from ss left join ws on (ws_sold_year=ss_sold_year and ws_item_sk=ss_item_sk and ws_customer_sk=ss_customer_sk) left join cs on (cs_sold_year=ss_sold_year and cs_item_sk=ss_item_sk and cs_customer_sk=ss_customer_sk) where (coalesce(ws_qty,0)>0 or coalesce(cs_qty, 0)>0) and ss_sold_year=2000 order by ss_sold_year, ss_item_sk, ss_customer_sk, ss_qty desc, ss_wc desc, ss_sp desc, other_chan_qty, other_chan_wholesale_cost, other_chan_sales_price, ratio limit 100; -- query 79 select c_last_name,c_first_name,substr(s_city,1,30),ss_ticket_number,amt,profit from (select ss_ticket_number ,ss_customer_sk ,store.s_city ,sum(ss_coupon_amt) amt ,sum(ss_net_profit) profit from store_sales,date_dim,store,household_demographics where store_sales.ss_sold_date_sk = date_dim.d_date_sk and store_sales.ss_store_sk = store.s_store_sk and store_sales.ss_hdemo_sk = household_demographics.hd_demo_sk and (household_demographics.hd_dep_count = 6 or household_demographics.hd_vehicle_count > 2) and date_dim.d_dow = 1 and date_dim.d_year in (1999,1999+1,1999+2) and store.s_number_employees between 200 and 295 group by ss_ticket_number,ss_customer_sk,ss_addr_sk,store.s_city) ms,customer where ss_customer_sk = c_customer_sk order by c_last_name,c_first_name,substr(s_city,1,30), profit limit 100; -- query 80 with ssr as (select s_store_id as store_id, sum(ss_ext_sales_price) as sales, sum(coalesce(sr_return_amt, 0)) as returns, sum(ss_net_profit - coalesce(sr_net_loss, 0)) as profit from store_sales left outer join store_returns on (ss_item_sk = sr_item_sk and ss_ticket_number = sr_ticket_number), date_dim, store, item, promotion where ss_sold_date_sk = d_date_sk and d_date between cast('2000-08-23' as date) and date_add(cast('2000-08-23' as date), 30) and ss_store_sk = s_store_sk and ss_item_sk = i_item_sk and i_current_price > 50 and ss_promo_sk = p_promo_sk and p_channel_tv = 'N' group by s_store_id) , csr as (select cp_catalog_page_id as catalog_page_id, sum(cs_ext_sales_price) as sales, sum(coalesce(cr_return_amount, 0)) as returns, sum(cs_net_profit - coalesce(cr_net_loss, 0)) as profit from catalog_sales left outer join catalog_returns on (cs_item_sk = cr_item_sk and cs_order_number = cr_order_number), date_dim, catalog_page, item, promotion where cs_sold_date_sk = d_date_sk and d_date between cast('2000-08-23' as date) and date_add(cast('2000-08-23' as date), 30) and cs_catalog_page_sk = cp_catalog_page_sk and cs_item_sk = i_item_sk and i_current_price > 50 and cs_promo_sk = p_promo_sk and p_channel_tv = 'N' group by cp_catalog_page_id) , wsr as (select web_site_id, sum(ws_ext_sales_price) as sales, sum(coalesce(wr_return_amt, 0)) as returns, sum(ws_net_profit - coalesce(wr_net_loss, 0)) as profit from web_sales left outer join web_returns on (ws_item_sk = wr_item_sk and ws_order_number = wr_order_number), date_dim, web_site, item, promotion where ws_sold_date_sk = d_date_sk and d_date between cast('2000-08-23' as date) and date_add(cast('2000-08-23' as date), 30) and ws_web_site_sk = web_site_sk and ws_item_sk = i_item_sk and i_current_price > 50 and ws_promo_sk = p_promo_sk and p_channel_tv = 'N' group by web_site_id) select channel , id , sum(sales) as sales , sum(returns) as returns , sum(profit) as profit from (select 'store channel' as channel , 'store' || store_id as id , sales , returns , profit from ssr union all select 'catalog channel' as channel , 'catalog_page' || catalog_page_id as id , sales , returns , profit from csr union all select 'web channel' as channel , 'web_site' || web_site_id as id , sales , returns , profit from wsr ) x group by rollup (channel, id) order by channel ,id limit 100; -- query 81 with customer_total_return as (select cr_returning_customer_sk as ctr_customer_sk ,ca_state as ctr_state, sum(cr_return_amt_inc_tax) as ctr_total_return from catalog_returns ,date_dim ,customer_address where cr_returned_date_sk = d_date_sk and d_year =2000 and cr_returning_addr_sk = ca_address_sk group by cr_returning_customer_sk ,ca_state ) select c_customer_id,c_salutation,c_first_name,c_last_name,ca_street_number,ca_street_name ,ca_street_type,ca_suite_number,ca_city,ca_county,ca_state,ca_zip,ca_country,ca_gmt_offset ,ca_location_type,ctr_total_return from customer_total_return ctr1 ,customer_address ,customer where ctr1.ctr_total_return > (select avg(ctr_total_return)*1.2 from customer_total_return ctr2 where ctr1.ctr_state = ctr2.ctr_state) and ca_address_sk = c_current_addr_sk and ca_state = 'GA' and ctr1.ctr_customer_sk = c_customer_sk order by c_customer_id,c_salutation,c_first_name,c_last_name,ca_street_number,ca_street_name ,ca_street_type,ca_suite_number,ca_city,ca_county,ca_state,ca_zip,ca_country,ca_gmt_offset ,ca_location_type,ctr_total_return limit 100; -- query 82 select i_item_id ,i_item_desc ,i_current_price from item, inventory, date_dim, store_sales where i_current_price between 62 and 62+30 and inv_item_sk = i_item_sk and d_date_sk=inv_date_sk and d_date between cast('2000-05-25' as date) and date_add(cast('2000-05-25' as date), 60) and i_manufact_id in (129,270,821,423) and inv_quantity_on_hand between 100 and 500 and ss_item_sk = i_item_sk group by i_item_id,i_item_desc,i_current_price order by i_item_id limit 100; -- query 83 with sr_items as (select i_item_id item_id, sum(sr_return_quantity) sr_item_qty from store_returns, item, date_dim where sr_item_sk = i_item_sk and d_date in (select d_date from date_dim where d_week_seq in (select d_week_seq from date_dim where d_date in ('2000-06-30','2000-09-27','2000-11-17'))) and sr_returned_date_sk = d_date_sk group by i_item_id), cr_items as (select i_item_id item_id, sum(cr_return_quantity) cr_item_qty from catalog_returns, item, date_dim where cr_item_sk = i_item_sk and d_date in (select d_date from date_dim where d_week_seq in (select d_week_seq from date_dim where d_date in ('2000-06-30','2000-09-27','2000-11-17'))) and cr_returned_date_sk = d_date_sk group by i_item_id), wr_items as (select i_item_id item_id, sum(wr_return_quantity) wr_item_qty from web_returns, item, date_dim where wr_item_sk = i_item_sk and d_date in (select d_date from date_dim where d_week_seq in (select d_week_seq from date_dim where d_date in ('2000-06-30','2000-09-27','2000-11-17'))) and wr_returned_date_sk = d_date_sk group by i_item_id) select sr_items.item_id ,sr_item_qty ,sr_item_qty/(sr_item_qty+cr_item_qty+wr_item_qty)/3.0 * 100 sr_dev ,cr_item_qty ,cr_item_qty/(sr_item_qty+cr_item_qty+wr_item_qty)/3.0 * 100 cr_dev ,wr_item_qty ,wr_item_qty/(sr_item_qty+cr_item_qty+wr_item_qty)/3.0 * 100 wr_dev ,(sr_item_qty+cr_item_qty+wr_item_qty)/3.0 average from sr_items ,cr_items ,wr_items where sr_items.item_id=cr_items.item_id and sr_items.item_id=wr_items.item_id order by sr_items.item_id ,sr_item_qty limit 100; -- query 84 select c_customer_id as customer_id , coalesce(c_last_name,'') || ', ' || coalesce(c_first_name,'') as customername from customer ,customer_address ,customer_demographics ,household_demographics ,income_band ,store_returns where ca_city = 'Edgewood' and c_current_addr_sk = ca_address_sk and ib_lower_bound >= 38128 and ib_upper_bound <= 38128 + 50000 and ib_income_band_sk = hd_income_band_sk and cd_demo_sk = c_current_cdemo_sk and hd_demo_sk = c_current_hdemo_sk and sr_cdemo_sk = cd_demo_sk order by c_customer_id limit 100; -- query 85 select substr(r_reason_desc,1,20) ,avg(ws_quantity) ,avg(wr_refunded_cash) ,avg(wr_fee) from web_sales, web_returns, web_page, customer_demographics cd1, customer_demographics cd2, customer_address, date_dim, reason where ws_web_page_sk = wp_web_page_sk and ws_item_sk = wr_item_sk and ws_order_number = wr_order_number and ws_sold_date_sk = d_date_sk and d_year = 2000 and cd1.cd_demo_sk = wr_refunded_cdemo_sk and cd2.cd_demo_sk = wr_returning_cdemo_sk and ca_address_sk = wr_refunded_addr_sk and r_reason_sk = wr_reason_sk and ( ( cd1.cd_marital_status = 'M' and cd1.cd_marital_status = cd2.cd_marital_status and cd1.cd_education_status = 'Advanced Degree' and cd1.cd_education_status = cd2.cd_education_status and ws_sales_price between 100.00 and 150.00 ) or ( cd1.cd_marital_status = 'S' and cd1.cd_marital_status = cd2.cd_marital_status and cd1.cd_education_status = 'College' and cd1.cd_education_status = cd2.cd_education_status and ws_sales_price between 50.00 and 100.00 ) or ( cd1.cd_marital_status = 'W' and cd1.cd_marital_status = cd2.cd_marital_status and cd1.cd_education_status = '2 yr Degree' and cd1.cd_education_status = cd2.cd_education_status and ws_sales_price between 150.00 and 200.00 ) ) and ( ( ca_country = 'United States' and ca_state in ('IN', 'OH', 'NJ') and ws_net_profit between 100 and 200 ) or ( ca_country = 'United States' and ca_state in ('WI', 'CT', 'KY') and ws_net_profit between 150 and 300 ) or ( ca_country = 'United States' and ca_state in ('LA', 'IA', 'AR') and ws_net_profit between 50 and 250 ) ) group by r_reason_desc order by substr(r_reason_desc,1,20) ,avg(ws_quantity) ,avg(wr_refunded_cash) ,avg(wr_fee) limit 100; -- query 86 select sum(ws_net_paid) as total_sum ,i_category ,i_class ,grouping(i_category)+grouping(i_class) as lochierarchy ,rank() over ( partition by grouping(i_category)+grouping(i_class), case when grouping(i_class) = 0 then i_category end order by sum(ws_net_paid) desc) as rank_within_parent from web_sales ,date_dim d1 ,item where d1.d_month_seq between 1200 and 1200+11 and d1.d_date_sk = ws_sold_date_sk and i_item_sk = ws_item_sk group by rollup(i_category,i_class) order by lochierarchy desc, case when lochierarchy = 0 then i_category end, rank_within_parent limit 100; -- query 87 select count(*) from ((select distinct c_last_name, c_first_name, d_date from store_sales, date_dim, customer where store_sales.ss_sold_date_sk = date_dim.d_date_sk and store_sales.ss_customer_sk = customer.c_customer_sk and d_month_seq between 1200 and 1200+11) except (select distinct c_last_name, c_first_name, d_date from catalog_sales, date_dim, customer where catalog_sales.cs_sold_date_sk = date_dim.d_date_sk and catalog_sales.cs_bill_customer_sk = customer.c_customer_sk and d_month_seq between 1200 and 1200+11) except (select distinct c_last_name, c_first_name, d_date from web_sales, date_dim, customer where web_sales.ws_sold_date_sk = date_dim.d_date_sk and web_sales.ws_bill_customer_sk = customer.c_customer_sk and d_month_seq between 1200 and 1200+11) ) cool_cust ; -- query 88 select * from (select count(*) h8_30_to_9 from store_sales, household_demographics , time_dim, store where ss_sold_time_sk = time_dim.t_time_sk and ss_hdemo_sk = household_demographics.hd_demo_sk and ss_store_sk = s_store_sk and time_dim.t_hour = 8 and time_dim.t_minute >= 30 and ((household_demographics.hd_dep_count = 4 and household_demographics.hd_vehicle_count<=4+2) or (household_demographics.hd_dep_count = 2 and household_demographics.hd_vehicle_count<=2+2) or (household_demographics.hd_dep_count = 0 and household_demographics.hd_vehicle_count<=0+2)) and store.s_store_name = 'ese') s1, (select count(*) h9_to_9_30 from store_sales, household_demographics , time_dim, store where ss_sold_time_sk = time_dim.t_time_sk and ss_hdemo_sk = household_demographics.hd_demo_sk and ss_store_sk = s_store_sk and time_dim.t_hour = 9 and time_dim.t_minute < 30 and ((household_demographics.hd_dep_count = 4 and household_demographics.hd_vehicle_count<=4+2) or (household_demographics.hd_dep_count = 2 and household_demographics.hd_vehicle_count<=2+2) or (household_demographics.hd_dep_count = 0 and household_demographics.hd_vehicle_count<=0+2)) and store.s_store_name = 'ese') s2, (select count(*) h9_30_to_10 from store_sales, household_demographics , time_dim, store where ss_sold_time_sk = time_dim.t_time_sk and ss_hdemo_sk = household_demographics.hd_demo_sk and ss_store_sk = s_store_sk and time_dim.t_hour = 9 and time_dim.t_minute >= 30 and ((household_demographics.hd_dep_count = 4 and household_demographics.hd_vehicle_count<=4+2) or (household_demographics.hd_dep_count = 2 and household_demographics.hd_vehicle_count<=2+2) or (household_demographics.hd_dep_count = 0 and household_demographics.hd_vehicle_count<=0+2)) and store.s_store_name = 'ese') s3, (select count(*) h10_to_10_30 from store_sales, household_demographics , time_dim, store where ss_sold_time_sk = time_dim.t_time_sk and ss_hdemo_sk = household_demographics.hd_demo_sk and ss_store_sk = s_store_sk and time_dim.t_hour = 10 and time_dim.t_minute < 30 and ((household_demographics.hd_dep_count = 4 and household_demographics.hd_vehicle_count<=4+2) or (household_demographics.hd_dep_count = 2 and household_demographics.hd_vehicle_count<=2+2) or (household_demographics.hd_dep_count = 0 and household_demographics.hd_vehicle_count<=0+2)) and store.s_store_name = 'ese') s4, (select count(*) h10_30_to_11 from store_sales, household_demographics , time_dim, store where ss_sold_time_sk = time_dim.t_time_sk and ss_hdemo_sk = household_demographics.hd_demo_sk and ss_store_sk = s_store_sk and time_dim.t_hour = 10 and time_dim.t_minute >= 30 and ((household_demographics.hd_dep_count = 4 and household_demographics.hd_vehicle_count<=4+2) or (household_demographics.hd_dep_count = 2 and household_demographics.hd_vehicle_count<=2+2) or (household_demographics.hd_dep_count = 0 and household_demographics.hd_vehicle_count<=0+2)) and store.s_store_name = 'ese') s5, (select count(*) h11_to_11_30 from store_sales, household_demographics , time_dim, store where ss_sold_time_sk = time_dim.t_time_sk and ss_hdemo_sk = household_demographics.hd_demo_sk and ss_store_sk = s_store_sk and time_dim.t_hour = 11 and time_dim.t_minute < 30 and ((household_demographics.hd_dep_count = 4 and household_demographics.hd_vehicle_count<=4+2) or (household_demographics.hd_dep_count = 2 and household_demographics.hd_vehicle_count<=2+2) or (household_demographics.hd_dep_count = 0 and household_demographics.hd_vehicle_count<=0+2)) and store.s_store_name = 'ese') s6, (select count(*) h11_30_to_12 from store_sales, household_demographics , time_dim, store where ss_sold_time_sk = time_dim.t_time_sk and ss_hdemo_sk = household_demographics.hd_demo_sk and ss_store_sk = s_store_sk and time_dim.t_hour = 11 and time_dim.t_minute >= 30 and ((household_demographics.hd_dep_count = 4 and household_demographics.hd_vehicle_count<=4+2) or (household_demographics.hd_dep_count = 2 and household_demographics.hd_vehicle_count<=2+2) or (household_demographics.hd_dep_count = 0 and household_demographics.hd_vehicle_count<=0+2)) and store.s_store_name = 'ese') s7, (select count(*) h12_to_12_30 from store_sales, household_demographics , time_dim, store where ss_sold_time_sk = time_dim.t_time_sk and ss_hdemo_sk = household_demographics.hd_demo_sk and ss_store_sk = s_store_sk and time_dim.t_hour = 12 and time_dim.t_minute < 30 and ((household_demographics.hd_dep_count = 4 and household_demographics.hd_vehicle_count<=4+2) or (household_demographics.hd_dep_count = 2 and household_demographics.hd_vehicle_count<=2+2) or (household_demographics.hd_dep_count = 0 and household_demographics.hd_vehicle_count<=0+2)) and store.s_store_name = 'ese') s8 ; -- query 89 select * from( select i_category, i_class, i_brand, s_store_name, s_company_name, d_moy, sum(ss_sales_price) sum_sales, avg(sum(ss_sales_price)) over (partition by i_category, i_brand, s_store_name, s_company_name) avg_monthly_sales from item, store_sales, date_dim, store where ss_item_sk = i_item_sk and ss_sold_date_sk = d_date_sk and ss_store_sk = s_store_sk and d_year in (1999) and ((i_category in ('Books','Electronics','Sports') and i_class in ('computers','stereo','football') ) or (i_category in ('Men','Jewelry','Women') and i_class in ('shirts','birdal','dresses') )) group by i_category, i_class, i_brand, s_store_name, s_company_name, d_moy) tmp1 where case when (avg_monthly_sales <> 0) then (abs(sum_sales - avg_monthly_sales) / avg_monthly_sales) else null end > 0.1 order by sum_sales - avg_monthly_sales, s_store_name limit 100; -- query 90 select cast(amc as decimal(15,4))/cast(pmc as decimal(15,4)) am_pm_ratio from ( select count(*) amc from web_sales, household_demographics , time_dim, web_page where ws_sold_time_sk = time_dim.t_time_sk and ws_ship_hdemo_sk = household_demographics.hd_demo_sk and ws_web_page_sk = web_page.wp_web_page_sk and time_dim.t_hour between 8 and 8+1 and household_demographics.hd_dep_count = 6 and web_page.wp_char_count between 5000 and 5200) at, ( select count(*) pmc from web_sales, household_demographics , time_dim, web_page where ws_sold_time_sk = time_dim.t_time_sk and ws_ship_hdemo_sk = household_demographics.hd_demo_sk and ws_web_page_sk = web_page.wp_web_page_sk and time_dim.t_hour between 19 and 19+1 and household_demographics.hd_dep_count = 6 and web_page.wp_char_count between 5000 and 5200) pt order by am_pm_ratio limit 100; -- query 91 select cc_call_center_id Call_Center, cc_name Call_Center_Name, cc_manager Manager, sum(cr_net_loss) Returns_Loss from call_center, catalog_returns, date_dim, customer, customer_address, customer_demographics, household_demographics where cr_call_center_sk = cc_call_center_sk and cr_returned_date_sk = d_date_sk and cr_returning_customer_sk= c_customer_sk and cd_demo_sk = c_current_cdemo_sk and hd_demo_sk = c_current_hdemo_sk and ca_address_sk = c_current_addr_sk and d_year = 1998 and d_moy = 11 and ( (cd_marital_status = 'M' and cd_education_status = 'Unknown') or(cd_marital_status = 'W' and cd_education_status = 'Advanced Degree')) and hd_buy_potential like 'Unknown%' and ca_gmt_offset = -7 group by cc_call_center_id,cc_name,cc_manager,cd_marital_status,cd_education_status order by sum(cr_net_loss) desc; -- query 92 select sum(ws_ext_discount_amt) as "Excess Discount Amount" from web_sales ,item ,date_dim where i_manufact_id = 350 and i_item_sk = ws_item_sk and d_date between '2000-01-27' and date_add(cast('2000-01-27' as date), 90) and d_date_sk = ws_sold_date_sk and ws_ext_discount_amt > ( SELECT 1.3 * avg(ws_ext_discount_amt) FROM web_sales ,date_dim WHERE ws_item_sk = i_item_sk and d_date between '2000-01-27' and date_add(cast('2000-01-27' as date), 90) and d_date_sk = ws_sold_date_sk ) order by sum(ws_ext_discount_amt) limit 100; -- query 93 select ss_customer_sk ,sum(act_sales) sumsales from (select ss_item_sk ,ss_ticket_number ,ss_customer_sk ,case when sr_return_quantity is not null then (ss_quantity-sr_return_quantity)*ss_sales_price else (ss_quantity*ss_sales_price) end act_sales from store_sales left outer join store_returns on (sr_item_sk = ss_item_sk and sr_ticket_number = ss_ticket_number) ,reason where sr_reason_sk = r_reason_sk and r_reason_desc = 'reason 28') t group by ss_customer_sk order by sumsales, ss_customer_sk limit 100; -- query 94 select count(distinct ws_order_number) as "order count" ,sum(ws_ext_ship_cost) as "total shipping cost" ,sum(ws_net_profit) as "total net profit" from web_sales ws1 ,date_dim ,customer_address ,web_site where d_date between '1999-2-01' and date_add(cast('1999-2-01' as date), 60) and ws1.ws_ship_date_sk = d_date_sk and ws1.ws_ship_addr_sk = ca_address_sk and ca_state = 'IL' and ws1.ws_web_site_sk = web_site_sk and web_company_name = 'pri' and exists (select * from web_sales ws2 where ws1.ws_order_number = ws2.ws_order_number and ws1.ws_warehouse_sk <> ws2.ws_warehouse_sk) and not exists(select * from web_returns wr1 where ws1.ws_order_number = wr1.wr_order_number) order by count(distinct ws_order_number) limit 100; -- query 95 with ws_wh as (select ws1.ws_order_number,ws1.ws_warehouse_sk wh1,ws2.ws_warehouse_sk wh2 from web_sales ws1,web_sales ws2 where ws1.ws_order_number = ws2.ws_order_number and ws1.ws_warehouse_sk <> ws2.ws_warehouse_sk) select count(distinct ws_order_number) as "order count" ,sum(ws_ext_ship_cost) as "total shipping cost" ,sum(ws_net_profit) as "total net profit" from web_sales ws1 ,date_dim ,customer_address ,web_site where d_date between '1999-2-01' and date_add(cast('1999-2-01' as date), 60) and ws1.ws_ship_date_sk = d_date_sk and ws1.ws_ship_addr_sk = ca_address_sk and ca_state = 'IL' and ws1.ws_web_site_sk = web_site_sk and web_company_name = 'pri' and ws1.ws_order_number in (select ws_order_number from ws_wh) and ws1.ws_order_number in (select wr_order_number from web_returns,ws_wh where wr_order_number = ws_wh.ws_order_number) order by count(distinct ws_order_number) limit 100; -- query 96 select count(*) from store_sales ,household_demographics ,time_dim, store where ss_sold_time_sk = time_dim.t_time_sk and ss_hdemo_sk = household_demographics.hd_demo_sk and ss_store_sk = s_store_sk and time_dim.t_hour = 20 and time_dim.t_minute >= 30 and household_demographics.hd_dep_count = 7 and store.s_store_name = 'ese' order by count(*) limit 100; -- query 97 with ssci as ( select ss_customer_sk customer_sk ,ss_item_sk item_sk from store_sales,date_dim where ss_sold_date_sk = d_date_sk and d_month_seq between 1200 and 1200 + 11 group by ss_customer_sk ,ss_item_sk), csci as( select cs_bill_customer_sk customer_sk ,cs_item_sk item_sk from catalog_sales,date_dim where cs_sold_date_sk = d_date_sk and d_month_seq between 1200 and 1200 + 11 group by cs_bill_customer_sk ,cs_item_sk) select sum(case when ssci.customer_sk is not null and csci.customer_sk is null then 1 else 0 end) store_only ,sum(case when ssci.customer_sk is null and csci.customer_sk is not null then 1 else 0 end) catalog_only ,sum(case when ssci.customer_sk is not null and csci.customer_sk is not null then 1 else 0 end) store_and_catalog from ssci full outer join csci on (ssci.customer_sk=csci.customer_sk and ssci.item_sk = csci.item_sk) limit 100; -- query 98 select i_item_id ,i_item_desc ,i_category ,i_class ,i_current_price ,sum(ss_ext_sales_price) as itemrevenue ,sum(ss_ext_sales_price)*100/sum(sum(ss_ext_sales_price)) over (partition by i_class) as revenueratio from store_sales ,item ,date_dim where ss_item_sk = i_item_sk and i_category in ('Sports', 'Books', 'Home') and ss_sold_date_sk = d_date_sk and d_date between cast('1999-02-22' as date) and date_add(cast('1999-02-22' as date), 30) group by i_item_id ,i_item_desc ,i_category ,i_class ,i_current_price order by i_category ,i_class ,i_item_id ,i_item_desc ,revenueratio; -- query 99 select substr(w_warehouse_name,1,20) ,sm_type ,cc_name ,sum(case when (cs_ship_date_sk - cs_sold_date_sk <= 30 ) then 1 else 0 end) as "30 days" ,sum(case when (cs_ship_date_sk - cs_sold_date_sk > 30) and (cs_ship_date_sk - cs_sold_date_sk <= 60) then 1 else 0 end ) as "31-60 days" ,sum(case when (cs_ship_date_sk - cs_sold_date_sk > 60) and (cs_ship_date_sk - cs_sold_date_sk <= 90) then 1 else 0 end) as "61-90 days" ,sum(case when (cs_ship_date_sk - cs_sold_date_sk > 90) and (cs_ship_date_sk - cs_sold_date_sk <= 120) then 1 else 0 end) as "91-120 days" ,sum(case when (cs_ship_date_sk - cs_sold_date_sk > 120) then 1 else 0 end) as ">120 days" from catalog_sales ,warehouse ,ship_mode ,call_center ,date_dim where d_month_seq between 1200 and 1200 + 11 and cs_ship_date_sk = d_date_sk and cs_warehouse_sk = w_warehouse_sk and cs_ship_mode_sk = sm_ship_mode_sk and cs_call_center_sk = cc_call_center_sk group by substr(w_warehouse_name,1,20) ,sm_type ,cc_name order by substr(w_warehouse_name,1,20) ,sm_type ,cc_name limit 100; ``` --- ### TPC-DS Benchmarking TPC-DS is a decision support benchmark developed by the Transaction Processing Performance Council (TPC). It uses more comprehensive test datasets and complex SQL queries than TPC-H. TPC-DS models several generally applicable aspects of a decision support system, including queries and data maintenance. TPC-DS aims to provide a comprehensive and realistic workload for testing and evaluating the performance of database systems in a retail environment. The TPC-DS benchmark simulates the sales and return data of three sales channels (stores, Internet, and catalog) in a retail enterprise. In addition to creating tables for sales and return data models, it also includes a simple inventory system and a promotion system. This benchmark tests a total of 99 complex SQL queries against 24 tables whose data size ranges from 1 GB to 3 TB. The main performance metric is the response time of each query, which is the duration between the time a query is submitted to the time the result is returned. #### Test Conclusions[​](#test-conclusions "Direct link to Test Conclusions") The test is performed against the TPC-DS 1 TB dataset on StarRocks and Trino. The unit of the following results are milliseconds. ![TPC-DS-SR](/assets/images/TPC-DS-SR-298c467caf034bdc5c5de30658886659.png) For StarRocks, the test queries are performed on its native tables (under both shared-nothing and shared-data architecture) and Iceberg Catalog (under shared-data architecture). For Trino, the test queries are performed on the same Iceberg Catalog. Both tests for StarRocks and Trino on the Iceberg Catalog use AWS Glue as the metastore, and Parquet-formatted data with ZSTD compression. The test results are: * Queries performed on the OLAP table in the StarRocks shared-nothing cluster took 314 seconds. * Queries performed on the OLAP table in the StarRocks shared-data cluster took 326 seconds. * Queries performed on Iceberg Catalog in the StarRocks shared-data cluster took 368 seconds. * Queries performed on Iceberg Catalog in Trino took 2552 seconds. The conclusions are: * **StarRocks delivers comparable performance in both shared-nothing and shared-data architecture** In OLAP table query scenarios, StarRocks in the shared-nothing mode achieves 1.04× the performance of the shared-data mode, indicating minimal performance overhead from storage-compute separation. * **StarRocks significantly outperforms Trino in Lakehouse query scenarios out of the box, with even greater advantages in OLAP table queries** * In Iceberg Catalog query scenarios, StarRocks delivers 6.93× the performance of Trino. * In OLAP table queries, the performance gap widens further—StarRocks achieves 8.13× (under shared-nothing mode) and 7.82× (under shared-data mode) the performance of Trino. #### Test Preparation[​](#test-preparation "Direct link to Test Preparation") ##### Hardware[​](#hardware "Direct link to Hardware") StarRocks and Trino are deployed on hosts of the same configurations - [AWS m6id.4xlarge](https://aws.amazon.com/ec2/instance-types/m6i/?nc1=h_ls). | | **Spec** | | ------------------------ | -------------- | | Instance Number | 5 | | vCPU | 16 | | Memory (GiB) | 64 | | Instance Storage (GB) | 1x950 NVMe SSD | | Network Bandwidth (Gbps) | Up to 12.5 | | EBS Bandwidth (Gbps) | Up to 10 | ##### Software[​](#software "Direct link to Software") | | **StarRocks** | **Trino** | | ----------------- | ------------------------------------------ | -------------------------------------------------------------- | | **Cluster Size** | One FE, four BE | One Coordinator, four Workers | | **Version** | 3.5.0 | 475 | | **Release Date** | 2025.6.13 | 2025.4.23 | | **Configuration** | `set query_mem_limit=66571993088;` (62 GB) | `query.max-memory=224GB`
`query.max-memory-per-node=35GB` | #### Test results[​](#test-results "Direct link to Test results") The following table shows the performance test results on 99 queries. The unit of query latency is milliseconds. All queries are warmed up 1 time and then executed 3 times to take the average value as the result. `Trino vs StarRocks on Iceberg` in the table header means using the query response time of Trino to divide the query response time of StarRocks. A larger value indicates better performance of StarRocks. Before the test on Iceberg Catalog, `ANALYZE TABLE` and `EXPLAIN COST` statements are executed on StarRocks to collect statistics. | Query | StarRocks | | | Trino | Trino vs StarRocks on Iceberg | | ----- | -------------- | ----------- | --------------- | --------------- | ----------------------------- | | | OLAP Table | | Iceberg Catalog | Iceberg Catalog | | | | Shared-nothing | Shared-data | Share-data | | | | SUM | 313582 | 326163 | 368404 | 2552076 | 6.93 | | Q01 | 675 | 636 | 727 | 2276 | 3.13 | | Q02 | 935 | 925 | 1949 | 21289 | 10.92 | | Q03 | 220 | 140 | 399 | 1764 | 4.42 | | Q04 | 17185 | 16206 | 16744 | 200249 | 11.96 | | Q05 | 417 | 543 | 1724 | 5574 | 3.23 | | Q06 | 200 | 180 | 343 | 2909 | 8.48 | | Q07 | 814 | 959 | 1795 | 3638 | 2.03 | | Q08 | 235 | 181 | 319 | 2534 | 7.94 | | Q09 | 7097 | 6503 | 23153 | 36125 | 1.56 | | Q10 | 297 | 343 | 497 | 2471 | 4.97 | | Q11 | 11229 | 10249 | 9777 | 133230 | 13.63 | | Q12 | 128 | 130 | 207 | 1067 | 5.15 | | Q13 | 417 | 447 | 2218 | 18297 | 8.25 | | Q14 | 15112 | 15055 | 15310 | 143172 | 9.35 | | Q15 | 455 | 474 | 525 | 2158 | 4.11 | | Q16 | 476 | 523 | 975 | 14035 | 14.39 | | Q17 | 895 | 948 | 1645 | 5487 | 3.34 | | Q18 | 802 | 973 | 1544 | 4736 | 3.07 | | Q19 | 209 | 268 | 507 | 1870 | 3.69 | | Q20 | 147 | 176 | 253 | 1257 | 4.97 | | Q21 | 83 | 80 | 405 | 945 | 2.33 | | Q22 | 1462 | 1823 | 2384 | 7871 | 3.30 | | Q23 | 82887 | 84426 | 50719 | 488653 | 9.63 | | Q24 | 7649 | 9064 | 11902 | 43436 | 3.65 | | Q25 | 795 | 919 | 1404 | 4347 | 3.10 | | Q26 | 376 | 511 | 868 | 2961 | 3.41 | | Q27 | 458 | 532 | 1397 | 4219 | 3.02 | | Q28 | 6489 | 6004 | 18727 | 24678 | 1.32 | | Q29 | 1296 | 2168 | 2129 | 10640 | 5.00 | | Q30 | 295 | 330 | 529 | 2770 | 5.24 | | Q31 | 2172 | 2162 | 2535 | 10260 | 4.05 | | Q32 | 110 | 225 | 277 | 1035 | 3.74 | | Q33 | 235 | 245 | 734 | 2743 | 3.74 | | Q34 | 402 | 422 | 696 | 2455 | 3.53 | | Q35 | 1353 | 1393 | 1562 | 8023 | 5.14 | | Q36 | 553 | 559 | 1193 | 3342 | 2.80 | | Q37 | 263 | 243 | 217 | 6178 | 28.47 | | Q38 | 5002 | 5207 | 5519 | 38408 | 6.96 | | Q39 | 356 | 367 | 820 | 5696 | 6.95 | | Q40 | 124 | 134 | 575 | 1554 | 2.70 | | Q41 | 43 | 50 | 76 | 271 | 3.57 | | Q42 | 81 | 104 | 197 | 996 | 5.06 | | Q43 | 487 | 733 | 722 | 4108 | 5.69 | | Q44 | 2087 | 2102 | 9364 | 16468 | 1.76 | | Q45 | 431 | 470 | 514 | 2212 | 4.30 | | Q46 | 1468 | 1829 | 1976 | 4780 | 2.42 | | Q47 | 2921 | 3120 | 3130 | 42704 | 13.64 | | Q48 | 467 | 573 | 1441 | 12464 | 8.65 | | Q49 | 496 | 547 | 1715 | 3827 | 2.23 | | Q50 | 3768 | 5135 | 3899 | 21771 | 5.58 | | Q51 | 4826 | 4633 | 3244 | 12247 | 3.78 | | Q52 | 87 | 106 | 219 | 1007 | 4.60 | | Q53 | 446 | 635 | 955 | 1768 | 1.85 | | Q54 | 198 | 266 | 763 | 31864 | 41.76 | | Q55 | 82 | 105 | 221 | 1187 | 5.37 | | Q56 | 175 | 186 | 499 | 2133 | 4.27 | | Q57 | 1848 | 1787 | 1916 | 26042 | 13.59 | | Q58 | 241 | 245 | 503 | 3032 | 6.03 | | Q59 | 3935 | 3908 | 3160 | 20819 | 6.59 | | Q60 | 262 | 292 | 643 | 2749 | 4.28 | | Q61 | 330 | 344 | 913 | 2054 | 2.25 | | Q62 | 500 | 519 | 840 | 5104 | 6.08 | | Q63 | 448 | 629 | 937 | 1719 | 1.83 | | Q64 | 4508 | 8886 | 8602 | 16990 | 1.98 | | Q65 | 5690 | 5617 | 5445 | 20616 | 3.79 | | Q66 | 387 | 390 | 1335 | 3511 | 2.63 | | Q67 | 31286 | 29240 | 21439 | 89430 | 4.17 | | Q68 | 240 | 341 | 683 | 2883 | 4.22 | | Q69 | 260 | 292 | 501 | 2181 | 4.35 | | Q70 | 2904 | 3012 | 3751 | 21499 | 5.73 | | Q71 | 272 | 1111 | 631 | 2578 | 4.09 | | Q72 | 1642 | 1899 | 4969 | 504573 | 101.54 | | Q73 | 186 | 228 | 388 | 1933 | 4.98 | | Q74 | 9433 | 9133 | 8890 | 78071 | 8.78 | | Q75 | 6446 | 7718 | 9082 | 26270 | 2.89 | | Q76 | 2356 | 2331 | 5028 | 11266 | 2.24 | | Q77 | 238 | 241 | 701 | 2785 | 3.97 | | Q78 | 17583 | 16590 | 18614 | 51497 | 2.77 | | Q79 | 2201 | 2474 | 1597 | 5572 | 3.49 | | Q80 | 800 | 855 | 2672 | 4762 | 1.78 | | Q81 | 518 | 533 | 769 | 4040 | 5.25 | | Q82 | 623 | 613 | 854 | 11622 | 13.61 | | Q83 | 158 | 217 | 404 | 2274 | 5.63 | | Q84 | 195 | 434 | 351 | 3086 | 8.79 | | Q85 | 455 | 733 | 1087 | 5424 | 4.99 | | Q86 | 725 | 736 | 819 | 3422 | 4.18 | | Q87 | 4816 | 5004 | 5535 | 41300 | 7.46 | | Q88 | 8712 | 8444 | 19465 | 30738 | 1.58 | | Q89 | 530 | 730 | 915 | 2227 | 2.43 | | Q90 | 587 | 581 | 1150 | 3890 | 3.38 | | Q91 | 95 | 96 | 227 | 2035 | 8.96 | | Q92 | 78 | 140 | 228 | 994 | 4.36 | | Q93 | 3723 | 4705 | 5781 | 22926 | 3.97 | | Q94 | 601 | 963 | 962 | 5884 | 6.12 | | Q95 | 1747 | 3106 | 2575 | 29607 | 11.50 | | Q96 | 1444 | 1428 | 2562 | 4905 | 1.91 | | Q97 | 4795 | 4844 | 5383 | 22726 | 4.22 | | Q98 | 284 | 323 | 365 | 1911 | 5.24 | | Q99 | 1132 | 1184 | 1594 | 10970 | 6.88 | --- ### TPC-H Benchmark TPC-H is a decision support benchmark developed by the Transaction Processing Performance Council (TPC). It consists of a suite of business oriented ad-hoc queries and concurrent data modifications. TPC-H can be used to build models based on real production environments to simulate the data warehouse of a sales system. This test uses eight tables with a data size ranging from 1 GB to 3 TB. A total of 22 queries are tested and the main performance metrics are the response time of each query, which is the duration between the time a query is submitted to the time the result is returned. #### 1. Test conclusion[​](#1-test-conclusion "Direct link to 1. Test conclusion") A comparative test was conducted on the TPC-H 100G scale dataset, with a total of 22 queries. The results are as follows: ![TPCH 100G results](/assets/images/tpch-c2ac4e648428201f0083970d29fb34ad.png) StarRocks tested two methods using local storage and Hive table queries. StarRocks Hive table and Trino query the same data. The data is stored in ORC format and compressed in zlib format. The latency for StarRocks to query data from its native storage is 21s, that for StarRocks to query Hive external tables is 92s, and that for Trino to query Hive external tables was 187s. #### 2. Test Preparation[​](#2-test-preparation "Direct link to 2. Test Preparation") ##### 2.1 Hardware environment[​](#21-hardware-environment "Direct link to 2.1 Hardware environment") | Machine | 4 Cloud hosts | | ----------------- | ----------------------------------------------------- | | CPU | 16core Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz | | Memory | 64 GB | | Network Bandwidth | 5 Gbits/s | | Disk | ESSD cloud disk | ##### 2.2 Software environment[​](#22-software-environment "Direct link to 2.2 Software environment") StarRocks and Trino are deployed on machines with the same configuration. StarRocks has 1 FE and 3 BEs deployed. Trino has 1 Coordinator and 3 Workers deployed. * Kernel version: Linux 3.10.0-1127.13.1.el7.x86\_64 * Operating system version: CentOS Linux release 7.8.2003 * Software version: StarRocks Community Edition 3.0, Trino-419, Hive-2.3.9 #### 3 Test data and results[​](#3-test-data-and-results "Direct link to 3 Test data and results") ##### 3.1 Test data[​](#31-test-data "Direct link to 3.1 Test data") | table | number of rows | | -------- | -------------- | | customer | 15 million | | lineitem | 600 million | | nation | 25 | | orders | 150 million | | part | 20 million | | partsupp | 80 million | | region | 5 | | supplier | 1 million | ##### 3.2 Test results[​](#32-test-results "Direct link to 3.2 Test results") note The unit of query results is ms. Lower is better. All queries are warmed up 1 time and then executed 3 times to take the average value as the result. | Query | StarRocks-native-3.0 | StarRocks-3.0-Hive external | Trino-419 | | ----- | -------------------- | --------------------------- | --------- | | Q1 | 1540 | 5660 | 8811 | | Q2 | 100 | 1593 | 3009 | | Q3 | 700 | 5286 | 7891 | | Q4 | 423 | 2110 | 5760 | | Q5 | 1180 | 4453 | 9181 | | Q6 | 56 | 2806 | 4029 | | Q7 | 903 | 4910 | 7158 | | Q8 | 546 | 4766 | 8014 | | Q9 | 2553 | 8010 | 18460 | | Q10 | 776 | 8190 | 9997 | | Q11 | 206 | 920 | 2088 | | Q12 | 166 | 2916 | 4852 | | Q13 | 1663 | 3420 | 7203 | | Q14 | 146 | 3286 | 4995 | | Q15 | 123 | 4173 | 9688 | | Q16 | 353 | 1126 | 2545 | | Q17 | 296 | 3426 | 18970 | | Q18 | 2713 | 7960 | 21763 | | Q19 | 246 | 4406 | 6586 | | Q20 | 176 | 3280 | 6632 | | Q21 | 1410 | 7933 | 16873 | | Q22 | 350 | 1190 | 2788 | | SUM | 16625 | 91820 | 187293 | #### 4. Test process[​](#4-test-process "Direct link to 4. Test process") ##### 4.1 Query StarRocks Native Table[​](#41-query-starrocks-native-table "Direct link to 4.1 Query StarRocks Native Table") ###### 4.1.1 Generate data[​](#411-generate-data "Direct link to 4.1.1 Generate data") Download the tpch-poc tool package to generate data for the TPC-H standard test set `scale factor=100`. ```bash wget https://starrocks-public.oss-cn-zhangjiakou.aliyuncs.com/tpch-poc-1.0.zip unzip tpch-poc-1.0 cd tpch-poc-1.0 sh bin/gen_data/gen-tpch.sh 100 data_100 ``` ###### 4.1.2 Create table structure[​](#412-create-table-structure "Direct link to 4.1.2 Create table structure") Modify the configuration file `conf/starrocks.conf`, specify the cluster address (host and port), and then perform the table creation operation. ```sql sh bin/create_db_table.sh ddl_100 ``` ###### 4.1.3 Import data[​](#413-import-data "Direct link to 4.1.3 Import data") ```python sh bin/stream_load.sh data_100 ``` ###### 4.1.4 Query data[​](#414-query-data "Direct link to 4.1.4 Query data") ```python sh bin/benchmark.sh ``` ##### 4.2 Query Hive external tables with StarRocks[​](#42-query-hive-external-tables-with-starrocks "Direct link to 4.2 Query Hive external tables with StarRocks") ###### 4.2.1 Create table structure[​](#421-create-table-structure "Direct link to 4.2.1 Create table structure") Create an external table in Hive. The external table storage format is ORC and the compression format is zlib. For detailed table creation statements, see [5.3](#53-hive-external-table-creation-orc-storage-format). This Hive external table is used for both StarRocks and Trino queries. ###### 4.2.2 Import data[​](#422-import-data "Direct link to 4.2.2 Import data") Upload the TPC-H CSV original data generated in step 4.1.1 to the HDFS specified path (this article uses the path `/user/tmp/csv/`), and then create an external table in Hive. For detailed table creation statements, see [5.4](#54-hive-external-table-creation-csv-storage-format). The storage format of this Hive external table is CSV, and the storage path is `/user/tmp/csv/`, the path where the CSV original data is uploaded. Import the data of the external table in CSV format into the external table in ORC format using `INSERT INTO`, so that the data is obtained in the storage format of ORC and the compression format of zlib. The import command is as follows: ```sql use tpch_hive_csv; insert into tpch_hive_orc.customer select * from customer; insert into tpch_hive_orc.lineitem select * from lineitem; insert into tpch_hive_orc.nation select * from nation; insert into tpch_hive_orc.orders select * from orders; insert into tpch_hive_orc.part select * from part; insert into tpch_hive_orc.partsupp select * from partsupp; insert into tpch_hive_orc.region select * from region; insert into tpch_hive_orc.supplier select * from supplier; ``` ###### 4.2.3 Query data[​](#423-query-data "Direct link to 4.2.3 Query data") StarRocks uses the Catalog function to query Hive table data. For detailed operations, see [Hive catalog](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md). #### 5. Query SQL and create table statements[​](#5-query-sql-and-create-table-statements "Direct link to 5. Query SQL and create table statements") ##### 5.1 TPC-H query SQL[​](#51-tpc-h-query-sql "Direct link to 5.1 TPC-H query SQL") ```sql --Q1 select l_returnflag, l_linestatus, sum(l_quantity) as sum_qty, sum(l_extendedprice) as sum_base_price, sum(l_extendedprice * (1 - l_discount)) as sum_disc_price, sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) as sum_charge, avg(l_quantity) as avg_qty, avg(l_extendedprice) as avg_price, avg(l_discount) as avg_disc, count(*) as count_order from lineitem where l_shipdate <= date '1998-12-01' - interval '90' day group by l_returnflag, l_linestatus order by l_returnflag, l_linestatus; --Q2 select s_acctbal, s_name, n_name, p_partkey, p_mfgr, s_address, s_phone, s_comment from part, supplier, partsupp, nation, region where p_partkey = ps_partkey and s_suppkey = ps_suppkey and p_size = 15 and p_type like '%BRASS' and s_nationkey = n_nationkey and n_regionkey = r_regionkey and r_name = 'EUROPE' and ps_supplycost = ( select min(ps_supplycost) from partsupp, supplier, nation, region where p_partkey = ps_partkey and s_suppkey = ps_suppkey and s_nationkey = n_nationkey and n_regionkey = r_regionkey and r_name = 'EUROPE' ) order by s_acctbal desc, n_name, s_name, p_partkey limit 100; --Q3 select l_orderkey, sum(l_extendedprice * (1 - l_discount)) as revenue, o_orderdate, o_shippriority from customer, orders, lineitem where c_mktsegment = 'BUILDING' and c_custkey = o_custkey and l_orderkey = o_orderkey and o_orderdate < date '1995-03-15' and l_shipdate > date '1995-03-15' group by l_orderkey, o_orderdate, o_shippriority order by revenue desc, o_orderdate limit 10; --Q4 select o_orderpriority, count(*) as order_count from orders where o_orderdate >= date '1993-07-01' and o_orderdate < date '1993-07-01' + interval '3' month and exists ( select * from lineitem where l_orderkey = o_orderkey and l_commitdate < l_receiptdate ) group by o_orderpriority order by o_orderpriority; --Q5 select n_name, sum(l_extendedprice * (1 - l_discount)) as revenue from customer, orders, lineitem, supplier, nation, region where c_custkey = o_custkey and l_orderkey = o_orderkey and l_suppkey = s_suppkey and c_nationkey = s_nationkey and s_nationkey = n_nationkey and n_regionkey = r_regionkey and r_name = 'ASIA' and o_orderdate >= date '1994-01-01' and o_orderdate < date '1994-01-01' + interval '1' year group by n_name order by revenue desc; --Q6 select sum(l_extendedprice * l_discount) as revenue from lineitem where l_shipdate >= date '1994-01-01' and l_shipdate < date '1994-01-01' + interval '1' year and l_discount between .06 - 0.01 and .06 + 0.01 and l_quantity < 24; --Q7 select supp_nation, cust_nation, l_year, sum(volume) as revenue from ( select n1.n_name as supp_nation, n2.n_name as cust_nation, extract(year from l_shipdate) as l_year, l_extendedprice * (1 - l_discount) as volume from supplier, lineitem, orders, customer, nation n1, nation n2 where s_suppkey = l_suppkey and o_orderkey = l_orderkey and c_custkey = o_custkey and s_nationkey = n1.n_nationkey and c_nationkey = n2.n_nationkey and ( (n1.n_name = 'FRANCE' and n2.n_name = 'GERMANY') or (n1.n_name = 'GERMANY' and n2.n_name = 'FRANCE') ) and l_shipdate between date '1995-01-01' and date '1996-12-31' ) as shipping group by supp_nation, cust_nation, l_year order by supp_nation, cust_nation, l_year; --Q8 select o_year, sum(case when nation = 'BRAZIL' then volume else 0 end) / sum(volume) as mkt_share from ( select extract(year from o_orderdate) as o_year, l_extendedprice * (1 - l_discount) as volume, n2.n_name as nation from part, supplier, lineitem, orders, customer, nation n1, nation n2, region where p_partkey = l_partkey and s_suppkey = l_suppkey and l_orderkey = o_orderkey and o_custkey = c_custkey and c_nationkey = n1.n_nationkey and n1.n_regionkey = r_regionkey and r_name = 'AMERICA' and s_nationkey = n2.n_nationkey and o_orderdate between date '1995-01-01' and date '1996-12-31' and p_type = 'ECONOMY ANODIZED STEEL' ) as all_nations group by o_year order by o_year; --Q9 select nation, o_year, sum(amount) as sum_profit from ( select n_name as nation, extract(year from o_orderdate) as o_year, l_extendedprice * (1 - l_discount) - ps_supplycost * l_quantity as amount from part, supplier, lineitem, partsupp, orders, nation where s_suppkey = l_suppkey and ps_suppkey = l_suppkey and ps_partkey = l_partkey and p_partkey = l_partkey and o_orderkey = l_orderkey and s_nationkey = n_nationkey and p_name like '%green%' ) as profit group by nation, o_year order by nation, o_year desc; --Q10 select c_custkey, c_name, sum(l_extendedprice * (1 - l_discount)) as revenue, c_acctbal, n_name, c_address, c_phone, c_comment from customer, orders, lineitem, nation where c_custkey = o_custkey and l_orderkey = o_orderkey and o_orderdate >= date '1993-10-01' and o_orderdate < date '1993-10-01' + interval '3' month and l_returnflag = 'R' and c_nationkey = n_nationkey group by c_custkey, c_name, c_acctbal, c_phone, n_name, c_address, c_comment order by revenue desc limit 20; --Q11 select ps_partkey, sum(ps_supplycost * ps_availqty) as value from partsupp, supplier, nation where ps_suppkey = s_suppkey and s_nationkey = n_nationkey and n_name = 'GERMANY' group by ps_partkey having sum(ps_supplycost * ps_availqty) > ( select sum(ps_supplycost * ps_availqty) * 0.000001 from partsupp, supplier, nation where ps_suppkey = s_suppkey and s_nationkey = n_nationkey and n_name = 'GERMANY' ) order by value desc; --Q12 select l_shipmode, sum(case when o_orderpriority = '1-URGENT' or o_orderpriority = '2-HIGH' then 1 else 0 end) as high_line_count, sum(case when o_orderpriority <> '1-URGENT' and o_orderpriority <> '2-HIGH' then 1 else 0 end) as low_line_count from orders, lineitem where o_orderkey = l_orderkey and l_shipmode in ('MAIL', 'SHIP') and l_commitdate < l_receiptdate and l_shipdate < l_commitdate and l_receiptdate >= date '1994-01-01' and l_receiptdate < date '1994-01-01' + interval '1' year group by l_shipmode order by l_shipmode; --Q13 select c_count, count(*) as custdist from ( select c_custkey, count(o_orderkey) as c_count from customer left outer join orders on c_custkey = o_custkey and o_comment not like '%special%requests%' group by c_custkey ) as c_orders group by c_count order by custdist desc, c_count desc; --Q14 select 100.00 * sum(case when p_type like 'PROMO%' then l_extendedprice * (1 - l_discount) else 0 end) / sum(l_extendedprice * (1 - l_discount)) as promo_revenue from lineitem, part where l_partkey = p_partkey and l_shipdate >= date '1995-09-01' and l_shipdate < date '1995-09-01' + interval '1' month; --Q15 select s_suppkey, s_name, s_address, s_phone, total_revenue from supplier, revenue0 where s_suppkey = supplier_no and total_revenue = ( select max(total_revenue) from revenue0 ) order by s_suppkey; --Q16 select p_brand, p_type, p_size, count(distinct ps_suppkey) as supplier_cnt from partsupp, part where p_partkey = ps_partkey and p_brand <> 'Brand#45' and p_type not like 'MEDIUM POLISHED%' and p_size in (49, 14, 23, 45, 19, 3, 36, 9) and ps_suppkey not in ( select s_suppkey from supplier where s_comment like '%Customer%Complaints%' ) group by p_brand, p_type, p_size order by supplier_cnt desc, p_brand, p_type, p_size; --Q17 select sum(l_extendedprice) / 7.0 as avg_yearly from lineitem, part where p_partkey = l_partkey and p_brand = 'Brand#23' and p_container = 'MED BOX' and l_quantity < ( select 0.2 *avg(l_quantity) from lineitem where l_partkey = p_partkey ); --Q18 select c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice, sum(l_quantity) from customer, orders, lineitem where o_orderkey in ( select l_orderkey from lineitem group by l_orderkey having sum(l_quantity) > 300 ) and c_custkey = o_custkey and o_orderkey = l_orderkey group by c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice order by o_totalprice desc, o_orderdate limit 100; --Q19 select sum(l_extendedprice* (1 - l_discount)) as revenue from lineitem, part where ( p_partkey = l_partkey and p_brand = 'Brand#12' and p_container in ('SM CASE', 'SM BOX', 'SM PACK', 'SM PKG') and l_quantity >= 1 and l_quantity <= 1 + 10 and p_size between 1 and 5 and l_shipmode in ('AIR', 'AIR REG') and l_shipinstruct = 'DELIVER IN PERSON' ) or ( p_partkey = l_partkey and p_brand = 'Brand#23' and p_container in ('MED BAG', 'MED BOX', 'MED PKG', 'MED PACK') and l_quantity >= 10 and l_quantity <= 10 + 10 and p_size between 1 and 10 and l_shipmode in ('AIR', 'AIR REG') and l_shipinstruct = 'DELIVER IN PERSON' ) or ( p_partkey = l_partkey and p_brand = 'Brand#34' and p_container in ('LG CASE', 'LG BOX', 'LG PACK', 'LG PKG') and l_quantity >= 20 and l_quantity <= 20 + 10 and p_size between 1 and 15 and l_shipmode in ('AIR', 'AIR REG') and l_shipinstruct = 'DELIVER IN PERSON' ); --Q20 select s_name, s_address from supplier, nation where s_suppkey in ( select ps_suppkey from partsupp where ps_partkey in ( select p_partkey from part where p_name like 'forest%' ) and ps_availqty > ( select 0.5 * sum(l_quantity) from lineitem where l_partkey = ps_partkey and l_suppkey = ps_suppkey and l_shipdate >= date '1994-01-01' and l_shipdate < date '1994-01-01' + interval '1' year ) ) and s_nationkey = n_nationkey and n_name = 'CANADA' order by s_name; --Q21 select s_name, count(*) as numwait from supplier, lineitem l1, orders, nation where s_suppkey = l1.l_suppkey and o_orderkey = l1.l_orderkey and o_orderstatus = 'F' and l1.l_receiptdate > l1.l_commitdate and exists ( select * from lineitem l2 where l2.l_orderkey = l1.l_orderkey and l2.l_suppkey <> l1.l_suppkey ) and not exists ( select * from lineitem l3 where l3.l_orderkey = l1.l_orderkey and l3.l_suppkey <> l1.l_suppkey and l3.l_receiptdate > l3.l_commitdate ) and s_nationkey = n_nationkey and n_name = 'SAUDI ARABIA' group by s_name order by numwait DESC, s_name limit 100; --Q22 select cntrycode, count(*) as numcust, sum(c_acctbal) as totacctbal from ( select substring(c_phone, 1, 2) as cntrycode, c_acctbal from customer where substring(c_phone, 1, 2) in ('13', '31', '23', '29', '30', '18', '17') and c_acctbal > ( select avg(c_acctbal) from customer where c_acctbal > 0.00 and substring(c_phone, 1, 2) in ('13', '31', '23', '29', '30', '18', '17') ) and not exists ( select * from orders where o_custkey = c_custkey ) ) as custsale group by cntrycode order by cntrycode; ``` ##### 5.2 StarRocks Native table creation[​](#52-starrocks-native-table-creation "Direct link to 5.2 StarRocks Native table creation") ```sql #Create table customer drop table if exists customer; CREATE TABLE customer ( c_custkey int NOT NULL, c_name VARCHAR(25) NOT NULL, c_address VARCHAR(40) NOT NULL, c_nationkey int NOT NULL, c_phone VARCHAR(15) NOT NULL, c_acctbal decimal(15, 2) NOT NULL, c_mktsegment VARCHAR(10) NOT NULL, c_comment VARCHAR(117) NOT NULL )ENGINE=OLAP DUPLICATE KEY(`c_custkey`) COMMENT "OLAP" DISTRIBUTED BY HASH(`c_custkey`) BUCKETS 24 PROPERTIES ( "replication_num" = "1" ); #Create table lineitem drop table if exists lineitem; CREATE TABLE lineitem ( l_shipdate DATE NOT NULL, l_orderkey int NOT NULL, l_linenumber int not null, l_partkey int NOT NULL, l_suppkey int not null, l_quantity decimal(15, 2) NOT NULL, l_extendedprice decimal(15, 2) NOT NULL, l_discount decimal(15, 2) NOT NULL, l_tax decimal(15, 2) NOT NULL, l_returnflag VARCHAR(1) NOT NULL, l_linestatus VARCHAR(1) NOT NULL, l_commitdate DATE NOT NULL, l_receiptdate DATE NOT NULL, l_shipinstruct VARCHAR(25) NOT NULL, l_shipmode VARCHAR(10) NOT NULL, l_comment VARCHAR(44) NOT NULL )ENGINE=OLAP DUPLICATE KEY(`l_shipdate`, `l_orderkey`) COMMENT "OLAP" DISTRIBUTED BY HASH(`l_orderkey`) BUCKETS 96 PROPERTIES ( "replication_num" = "1", "colocate_with" = "tpch2" ); #Create table nation drop table if exists nation; CREATE TABLE `nation` ( `n_nationkey` int(11) NOT NULL, `n_name` varchar(25) NOT NULL, `n_regionkey` int(11) NOT NULL, `n_comment` varchar(152) NULL ) ENGINE=OLAP DUPLICATE KEY(`N_NATIONKEY`) COMMENT "OLAP" DISTRIBUTED BY HASH(`N_NATIONKEY`) BUCKETS 1 PROPERTIES ( "replication_num" = "3" ); #Create table orders drop table if exists orders; CREATE TABLE orders ( o_orderkey int NOT NULL, o_orderdate DATE NOT NULL, o_custkey int NOT NULL, o_orderstatus VARCHAR(1) NOT NULL, o_totalprice decimal(15, 2) NOT NULL, o_orderpriority VARCHAR(15) NOT NULL, o_clerk VARCHAR(15) NOT NULL, o_shippriority int NOT NULL, o_comment VARCHAR(79) NOT NULL )ENGINE=OLAP DUPLICATE KEY(`o_orderkey`, `o_orderdate`) COMMENT "OLAP" DISTRIBUTED BY HASH(`o_orderkey`) BUCKETS 96 PROPERTIES ( "replication_num" = "1", "colocate_with" = "tpch2" ); #Create table part drop table if exists part; CREATE TABLE part ( p_partkey int NOT NULL, p_name VARCHAR(55) NOT NULL, p_mfgr VARCHAR(25) NOT NULL, p_brand VARCHAR(10) NOT NULL, p_type VARCHAR(25) NOT NULL, p_size int NOT NULL, p_container VARCHAR(10) NOT NULL, p_retailprice decimal(15, 2) NOT NULL, p_comment VARCHAR(23) NOT NULL )ENGINE=OLAP DUPLICATE KEY(`p_partkey`) COMMENT "OLAP" DISTRIBUTED BY HASH(`p_partkey`) BUCKETS 24 PROPERTIES ( "replication_num" = "1", "colocate_with" = "tpch2p" ); #Create table partsupp drop table if exists partsupp; CREATE TABLE partsupp ( ps_partkey int NOT NULL, ps_suppkey int NOT NULL, ps_availqty int NOT NULL, ps_supplycost decimal(15, 2) NOT NULL, ps_comment VARCHAR(199) NOT NULL )ENGINE=OLAP DUPLICATE KEY(`ps_partkey`) COMMENT "OLAP" DISTRIBUTED BY HASH(`ps_partkey`) BUCKETS 24 PROPERTIES ( "replication_num" = "1", "colocate_with" = "tpch2p" ); #Create table region drop table if exists region; CREATE TABLE region ( r_regionkey int NOT NULL, r_name VARCHAR(25) NOT NULL, r_comment VARCHAR(152) )ENGINE=OLAP DUPLICATE KEY(`r_regionkey`) COMMENT "OLAP" DISTRIBUTED BY HASH(`r_regionkey`) BUCKETS 1 PROPERTIES ( "replication_num" = "3" ); #Create table supplier drop table if exists supplier; CREATE TABLE supplier ( s_suppkey int NOT NULL, s_name VARCHAR(25) NOT NULL, s_address VARCHAR(40) NOT NULL, s_nationkey int NOT NULL, s_phone VARCHAR(15) NOT NULL, s_acctbal decimal(15, 2) NOT NULL, s_comment VARCHAR(101) NOT NULL )ENGINE=OLAP DUPLICATE KEY(`s_suppkey`) COMMENT "OLAP" DISTRIBUTED BY HASH(`s_suppkey`) BUCKETS 12 PROPERTIES ( "replication_num" = "1" ); drop view if exists revenue0; create view revenue0 (supplier_no, total_revenue) as select l_suppkey, sum(l_extendedprice * (1 - l_discount)) from lineitem where l_shipdate >= date '1996-01-01' and l_shipdate < date '1996-01-01' + interval '3' month group by l_suppkey; ``` ##### 5.3 Hive external table creation (ORC storage format)[​](#53-hive-external-table-creation-orc-storage-format "Direct link to 5.3 Hive external table creation (ORC storage format)") ```sql create database tpch_hive_orc; use tpch_hive_orc; --Create table customer CREATE TABLE `customer`( `c_custkey` int, `c_name` varchar(25), `c_address` varchar(40), `c_nationkey` int, `c_phone` varchar(15), `c_acctbal` decimal(15,2), `c_mktsegment` varchar(10), `c_comment` varchar(117)) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs://emr-header-1.cluster-49146:9000/user/hive/warehouse/tpch_hive_orc.db/customer'; -- Create table lineitem CREATE TABLE `lineitem`( `l_orderkey` bigint, `l_partkey` int, `l_suppkey` int, `l_linenumber` int, `l_quantity` decimal(15,2), `l_extendedprice` decimal(15,2), `l_discount` decimal(15,2), `l_tax` decimal(15,2), `l_returnflag` varchar(1), `l_linestatus` varchar(1), `l_shipdate` date, `l_commitdate` date, `l_receiptdate` date, `l_shipinstruct` varchar(25), `l_shipmode` varchar(10), `l_comment` varchar(44)) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs://emr-header-1.cluster-49146:9000/user/hive/warehouse/tpch_hive_orc.db/lineitem'; -- Create table nation CREATE TABLE `nation`( `n_nationkey` int, `n_name` varchar(25), `n_regionkey` int, `n_comment` varchar(152)) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs://emr-header-1.cluster-49146:9000/user/hive/warehouse/tpch_hive_orc.db/nation'; -- Create table orders CREATE TABLE `orders`( `o_orderkey` bigint, `o_custkey` int, `o_orderstatus` varchar(1), `o_totalprice` decimal(15,2), `o_orderdate` date, `o_orderpriority` varchar(15), `o_clerk` varchar(15), `o_shippriority` int, `o_comment` varchar(79)) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs://emr-header-1.cluster-49146:9000/user/hive/warehouse/tpch_hive_orc.db/orders'; --Create table part CREATE TABLE `part`( `p_partkey` int, `p_name` varchar(55), `p_mfgr` varchar(25), `p_brand` varchar(10), `p_type` varchar(25), `p_size` int, `p_container` varchar(10), `p_retailprice` decimal(15,2), `p_comment` varchar(23)) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs://emr-header-1.cluster-49146:9000/user/hive/warehouse/tpch_hive_orc.db/part'; --Create table partsupp CREATE TABLE `partsupp`( `ps_partkey` int, `ps_suppkey` int, `ps_availqty` int, `ps_supplycost` decimal(15,2), `ps_comment` varchar(199)) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs://emr-header-1.cluster-49146:9000/user/hive/warehouse/tpch_hive_orc.db/partsupp'; --Create table region CREATE TABLE `region`( `r_regionkey` int, `r_name` varchar(25), `r_comment` varchar(152)) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs://emr-header-1.cluster-49146:9000/user/hive/warehouse/tpch_hive_orc.db/region'; --Create table supplier CREATE TABLE `supplier`( `s_suppkey` int, `s_name` varchar(25), `s_address` varchar(40), `s_nationkey` int, `s_phone` varchar(15), `s_acctbal` decimal(15,2), `s_comment` varchar(101)) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs://emr-header-1.cluster-49146:9000/user/hive/warehouse/tpch_hive_orc.db/supplier'; ``` ##### 5.4 Hive external table creation (CSV storage format)[​](#54-hive-external-table-creation-csv-storage-format "Direct link to 5.4 Hive external table creation (CSV storage format)") ```sql create database tpch_hive_csv; use tpch_hive_csv; --Create the customer external table. CREATE EXTERNAL TABLE `customer`( `c_custkey` int, `c_name` varchar(25), `c_address` varchar(40), `c_nationkey` int, `c_phone` varchar(15), `c_acctbal` double, `c_mktsegment` varchar(10), `c_comment` varchar(117)) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION 'hdfs://emr-header-1.cluster-49146:9000/user/tmp/csv/customer_csv'; --Create the lineitem external table. CREATE EXTERNAL TABLE `lineitem`( `l_orderkey` int, `l_partkey` int, `l_suppkey` int, `l_linenumber` int, `l_quantity` double, `l_extendedprice` double, `l_discount` double, `l_tax` double, `l_returnflag` varchar(1), `l_linestatus` varchar(1), `l_shipdate` date, `l_commitdate` date, `l_receiptdate` date, `l_shipinstruct` varchar(25), `l_shipmode` varchar(10), `l_comment` varchar(44)) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION 'hdfs://emr-header-1.cluster-49146:9000/user/tmp/csv/lineitem_csv'; --Create the nation external table. CREATE EXTERNAL TABLE `nation`( `n_nationkey` int, `n_name` varchar(25), `n_regionkey` int, `n_comment` varchar(152)) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION 'hdfs://emr-header-1.cluster-49146:9000/user/tmp/csv/nation_csv'; --Create the orders external table. CREATE EXTERNAL TABLE `orders`( `o_orderkey` int, `o_custkey` int, `o_orderstatus` varchar(1), `o_totalprice` double, `o_orderdate` date, `o_orderpriority` varchar(15), `o_clerk` varchar(15), `o_shippriority` int, `o_comment` varchar(79)) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION 'hdfs://emr-header-1.cluster-49146:9000/user/tmp/csv/orders_csv'; --Create the part external table. CREATE EXTERNAL TABLE `part`( `p_partkey` int, `p_name` varchar(55), `p_mfgr` varchar(25), `p_brand` varchar(10), `p_type` varchar(25), `p_size` int, `p_container` varchar(10), `p_retailprice` double, `p_comment` varchar(23)) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION 'hdfs://emr-header-1.cluster-49146:9000/user/tmp/csv/part_csv'; --Create the partsupp external table. CREATE EXTERNAL TABLE `partsupp`( `ps_partkey` int, `ps_suppkey` int, `ps_availqty` int, `ps_supplycost` double, `ps_comment` varchar(199)) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION 'hdfs://emr-header-1.cluster-49146:9000/user/tmp/csv/partsupp_csv'; --Create the region external table. CREATE EXTERNAL TABLE `region`( `r_regionkey` int, `r_name` varchar(25), `r_comment` varchar(152)) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION 'hdfs://emr-header-1.cluster-49146:9000/user/tmp/csv/region_csv'; --Create the supplier external table. CREATE EXTERNAL TABLE `supplier`( `s_suppkey` int, `s_name` varchar(25), `s_address` varchar(40), `s_nationkey` int, `s_phone` varchar(15), `s_acctbal` double, `s_comment` varchar(101)) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' WITH SERDEPROPERTIES ( 'field.delim'='|', 'serialization.format'='|') STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' LOCATION 'hdfs://emr-header-1.cluster-49146:9000/user/tmp/csv/supplier_csv'; ``` --- ## Best_practices ### Audit Log-based Resource Group Configuration In StarRocks, **Resource Groups** provide an effective mechanism for resource isolation by allocating CPU, memory, and concurrency limits based on classifiers such as user identity and query type. This feature is essential for achieving efficient resource utilization in multi-tenant environments. Traditional resource group configuration often relies on empirical judgment. By analyzing historical query data from the audit log table `starrocks_audit_db__.starrocks_audit_tbl__`, administrators can instead adopt a **data-driven approach** to tuning resource groups. Key metrics such as CPU time, memory consumption, and query concurrency offer objective insights into actual workload characteristics. This approach helps: * Prevent query latency caused by resource contention * Protect the cluster from resource exhaustion * Improve overall stability and predictability This topic provides step-by-step tutorial on how to derive appropriate resource group parameters based on workload patterns observed from audit logs. note This tutorial is based on the analysis using AuditLoader plugin, which allows you to query audit logs using SQL statements directly within your cluster. For detailed instructions to install the plugin, see [AuditLoader](https://docs.starrocks.io/docs/administration/management/audit_loader.md). #### CPU Resource Allocation[​](#cpu-resource-allocation "Direct link to CPU Resource Allocation") ##### Objective[​](#objective "Direct link to Objective") Determine per-user CPU consumption and allocate CPU resources proportionally using `cpu_weight` or `exclusive_cpu_cores`. ##### Analysis[​](#analysis "Direct link to Analysis") The following SQL aggregates total CPU time per user (`cpuCostNs`) over the last 30 days, converts it to seconds, and calculates the percentage of total CPU usage. ```sql SELECT user, SUM(cpuCostNs) / 1e9 AS total_cpu_seconds, -- Query the total CPU time. ( SUM(cpuCostNs) / ( SELECT SUM(cpuCostNs) FROM starrocks_audit_db__.starrocks_audit_tbl__ WHERE state IN ('EOF','OK') AND timestamp >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY) ) ) * 100 AS cpu_usage_percentage -- Calculate the percentage of total CPU usage per user. FROM starrocks_audit_db__.starrocks_audit_tbl__ WHERE state IN ('EOF','OK') -- Include queries that are finished only. AND timestamp >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY) -- Query the data of the last 30 days. GROUP BY user ORDER BY total_cpu_seconds DESC LIMIT 20; -- List the top 20 users with the most CPU resource consumption. ``` ##### Best Practices[​](#best-practices "Direct link to Best Practices") Assume a fixed number of CPU cores per BE (for example, 64 cores). If a user accounts for 16% (`cpu_usage_percentage`) of total CPU time, allocating approximately `64 × 16% ≈ 11 cores` is reasonable. You can configure the CPU limits for the resource group as follows: * `exclusive_cpu_cores`: * Its value must not exceed the total number of cores on a single BE. * The sum of `exclusive_cpu_cores` of all resource groups must not exceed the total number of cores on a single BE. * `cpu_weight`: * Applies only to **soft-isolation** resource groups. * Determines relative CPU share among competing queries on remaining cores. * Does **not** map directly to a fixed number of CPU cores. #### Memory Management[​](#memory-management "Direct link to Memory Management") ##### Objective[​](#objective-1 "Direct link to Objective") Identify memory-intensive users and define appropriate memory limits and circuit breakers. ##### Analysis[​](#analysis-1 "Direct link to Analysis") The following SQL computes the maximum memory usage per user (`memCostBytes`) for a single query over the last 30 days. ```sql SELECT user, MAX(memCostBytes) / (1024 * 1024) AS max_mem_mb -- Max memory usage (in MB) per query. FROM starrocks_audit_db__.starrocks_audit_tbl__ WHERE state IN ('EOF','OK') -- Include queries that are finished only. AND timestamp >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY) -- Query the data of the last 30 days. GROUP BY user ORDER BY max_mem_mb DESC LIMIT 20; -- List the top 20 users with the most memory resource consumption. ``` ##### Best Practices[​](#best-practices-1 "Direct link to Best Practices") `max_mem_mb` represents **total memory usage across all BEs**. You can calculate the approximate per-BE memory usage as: `max_mem_mb / number_of_BEs`. You can configure the memory limits for the resource group as follows: * `big_query_mem_limit`: * Protects the cluster from anomalously large queries. * You can set it to a relatively high threshold to avoid false-positive query termination. * `mem_limit`: * In most cases, set it to a high value (for example, `0.9`). #### Concurrency Control[​](#concurrency-control "Direct link to Concurrency Control") ##### Objective[​](#objective-2 "Direct link to Objective") Identify peak query concurrency per user and define appropriate `concurrency_limit` values. ##### Analysis[​](#analysis-2 "Direct link to Analysis") The following SQL analyzes per-minute query concurrency over the last 30 days and extracts the maximum observed concurrency per user. ```sql WITH UserConcurrency AS ( SELECT user, DATE_FORMAT(timestamp, '%Y-%m-%d %H:%i') AS minute_bucket, COUNT(*) AS query_concurrency FROM starrocks_audit_db__.starrocks_audit_tbl__ WHERE state IN ('EOF', 'OK') -- Include queries that are finished only. AND timestamp >= DATE_SUB(NOW(), INTERVAL 30 DAY) -- Query the data of the last 30 days. AND LOWER(stmt) LIKE '%select%' -- Include SELECT statements only. GROUP BY user, minute_bucket HAVING query_concurrency > 1 -- Exclude scenarios where concurrency is less than one query per minute. ) SELECT user, minute_bucket, query_concurrency / 60.0 AS query_concurrency_per_second -- Query the per-second concurrency. FROM ( SELECT user, minute_bucket, query_concurrency, ROW_NUMBER() OVER ( PARTITION BY user ORDER BY query_concurrency DESC ) AS rn FROM UserConcurrency ) ranked WHERE rn = 1 -- Keep the highest record for each user. ORDER BY query_concurrency_per_second DESC LIMIT 50; -- List the top 50 users with the highest concurrency. ``` ##### Best Practices[​](#best-practices-2 "Direct link to Best Practices") The above analysis is performed at **minute granularity**. Actual per-second concurrency may be higher. You can configure the concurrency limits for the resource group as follows: * `concurrency_limit` * Set it to approximately **1.5× the observed peak** to provide headroom. * For users with extreme concurrency spikes, you can further enable **Query Queues** to smooth peak load and protect cluster stability. #### Resource Isolation for Asynchronous Materialized Views[​](#resource-isolation-for-asynchronous-materialized-views "Direct link to Resource Isolation for Asynchronous Materialized Views") ##### Objective[​](#objective-3 "Direct link to Objective") Prevent asynchronous materialized view refresh operations from impacting interactive queries. ##### Analysis[​](#analysis-3 "Direct link to Analysis") The following SQL identifies memory-intensive materialized view refresh operations, typically characterized by `INSERT OVERWRITE` statements. ```sql SELECT user, MAX(memCostBytes) / (1024 * 1024) AS max_mem_mb -- Max memory usage (in MB) per query. FROM starrocks_audit_db__.starrocks_audit_tbl__ WHERE state IN ('EOF','OK') -- Include queries that are finished only. AND timestamp >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY) -- Query the data of the last 30 days. AND LOWER(stmt) LIKE '%insert overwrite%' -- Include materialized view refresh operations only. GROUP BY user ORDER BY max_mem_mb DESC LIMIT 20; -- List the top 20 users with the most memory resource consumption. ``` ##### Best Practices[​](#best-practices-3 "Direct link to Best Practices") StarRocks provides a system-defined resource group (`default_mv_wg`) for materialized view refresh tasks by default. However, customizing a dedicated resource group for materialized view refresh tasks is strongly recommended to enforce strict isolation and prevent materialized view refresh operations from degrading foreground query performance. For instructions on configuring resource group limits, see [Best Practice for CPU Resource Allocation](#best-practices) and [Best Practice for Memory Management](#best-practices-1). The following example only provides guidance on creating and assign a dedicated resource group to materialized view refresh tasks. 1. Create a dedicated resource group for materialized view refresh: ```sql CREATE RESOURCE GROUP rg_mv TO ( user = 'mv_user', query_type IN ('insert', 'select') ) WITH ( 'cpu_weight' = '32', 'mem_limit' = '0.9', 'concurrency_limit' = '10', 'spill_mem_limit_threshold' = '0.5' ); ``` 2. Assign the resource group to a materialized view. * Assign while creating a materialized view: ```sql CREATE MATERIALIZED VIEW mv_example REFRESH ASYNC PROPERTIES ( 'resource_group' = 'rg_mv' ) AS SELECT * FROM example_table; ``` * Assign to an existing materialized view: ```sql ALTER MATERIALIZED VIEW mv_example SET ("resource_group" = "rg_mv"); ``` #### See Also[​](#see-also "Direct link to See Also") * [Resource Group](https://docs.starrocks.io/docs/administration/management/resource_management/resource_group.md) * [Query Queues](https://docs.starrocks.io/docs/administration/management/resource_management/query_queues.md) --- ### Authentication and Authorization This topic aims to provide a coherent guide for best practices on developing your own authentication and authorization workflow. For detailed instructions on each operation involved below, see links in [See Also](#see-also). #### Real-World Enterprise Scenario[​](#real-world-enterprise-scenario "Direct link to Real-World Enterprise Scenario") Large enterprises often have complex organizational structures and a vast number of employees using diverse platforms and tools. From an IT governance perspective, having a unified identity, authentication, and authorization system brings significant advantages: * **Simplified User Management**: Admins no longer need to manually create or delete users and assign permissions across multiple systems. User lifecycle management (for example, onboarding/offboarding) becomes seamless and audit-friendly. * **Improved Security**: A single sign-on (SSO) mechanism eliminates the need for users to manage multiple credentials, reducing the attack surface. * **Role-Aligned Access Control**: Access permissions are typically tied to a user’s role or department. A well-structured identity system enables easier and more accurate authorization decisions. ##### Example[​](#example "Direct link to Example") Suppose that three new employees join different departments of a SaaS company: one Marketing Specialist and two Solution Architects. * **Organizationally**, they belong to different teams. * **From an identity standpoint**, their email accounts serve as their login credentials across internal platforms. * **By access rights**, each of the three is granted access to different platforms: * The Marketing Specialist can log in to Hubspot backend to view new leads. * Solution Architects can access the service console, and manage services for assigned customers. Although all three use the same identity provider, their access rights are strictly enforced: * The Marketing Specialist only has access to Hubspot. * Solution Architects can access the service console, but they cannot access the service for users they are not assigned to. They also cannot access Hubspot. #### Three Layers of Access Control[​](#three-layers-of-access-control "Direct link to Three Layers of Access Control") This example highlights the three key components in an enterprise identity and access flow: 1. **Identity Authentication** – “I am Peter, a verified employee of the SaaS company.” 2. **Access Authentication** – “As a Solution Architect, I am authorized to log in to the service console.” (Not all verified employees should have access to all services.) 3. **Action Authorization** – “As the customer of the SaaS company, I can view the information of our own service, but not other customer’s.” ##### In Database Context[​](#in-database-context "Direct link to In Database Context") These layers of access control also apply to the database system: 1. **Identity Verification**: Confirm the user is a valid employee with their own password. 2. **Access Authentication**: Verify the user or their group has permission to log in to a specific cluster. 3. **Operation Authorization**: Check if the user can run a query, load data, etc. As you can see, authentication and authorization are tightly coupled in practice. A user's authentication request often implies a broader access control requirement. Therefore, it is essential to understand the full access flow. #### Key Concepts[​](#key-concepts "Direct link to Key Concepts") ##### LDAP[​](#ldap "Direct link to LDAP") Lightweight Directory Access Protocol (LDAP) is a protocol for accessing and maintaining distributed directory information. You can think of it as your organization’s global address book: * Each user has a unique path (Distinguished Name, DN). * LDAP stores basic user information, including passwords. * LDAP also manages group structures and membership. * `ldapsearch` queries can retrieve users or groups. LDAP can be used: * As an **authentication** source (to validate usernames and passwords). * As a **group information** provider for access control. ##### UNIX Groups[​](#unix-groups "Direct link to UNIX Groups") Sometimes users mirror LDAP groups locally (on the host OS) for security or isolation reasons, avoiding direct communication with external LDAP servers. These local UNIX groups can be used for authentication or access control enforcement. ##### OAuth, OIDC, and JWT[​](#oauth-oidc-and-jwt "Direct link to OAuth, OIDC, and JWT") tip **Quick Explanation of Terms** * **ID Token**: Proof of identity (I am me.) * **Access Token**: Proof of permission to access certain resources (I can do certain things.) * **OAuth 2.0**: Authorization framework that provides access tokens. * **OIDC**: Authentication layer on top of OAuth. Provides ID and Access Tokens. * **JWT**: Token format. Used by both OAuth and OIDC. **Practical Use:** * **OAuth-based login**: Redirects to an external login page (for example, Google), then back to the cluster. Requires browser access and redirect URL setup in advance. * **JWT-based login**: The user passes a token directly to the cluster, which requires a public key or endpoint setup in advance. #### Features[​](#features "Direct link to Features") The system supports all three layers of access control: 1. **User Authentication** – “I am who I say I am.” 2. **Login Authorization** – “I am allowed to access this cluster.” (It depends on individual or group membership.) 3. **Operation Authorization** – “I can run this query or load this dataset.” (Authorization can be based on identity or group affiliation.) From v3.5 onward, StarRocks provides a modular, composable model to support various combinations of identity and access management components. *Feature Mapping* ![Authentication and Authorization](/assets/images/auth_feature-120a5a00c391862013cf840199fdc35b.png) From the feature's perspective: 1. **Authentication Provider** – Supported protocols: Native user, LDAP, OIDC, and OAuth 2.0. 2. **Group Provider** – Supported sources: LDAP, Operating System, and File-based Configuration. 3. **Authorization System** – Supported systems: Native RBAC & IBAC, and Apache Ranger. ##### Authentication[​](#authentication "Direct link to Authentication") Comparison of supported authentication modes: | Method | CREATE USER (Native user) | CREATE SECURITY INTEGRATION (Session-based dummy user) | | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Description | Manually creates users in the cluster. You can associate them with external authentication systems. The user exists explicitly in the cluster. | Defines an external authentication integration. The cluster does not store any user information. You can optionally combine it with a Group Provider to define allowed users. | | Login Process | Users must be pre-created in the cluster. During login, the user is authenticated via StarRocks or via the configured external authentication system (for example, LDAP). Only pre-created users can log in. | Upon login, StarRocks authenticates the user using external identity systems. If succeeds, it creates a temporary, session-scoped "dummy user" internally. This user is discarded after the session ends. | | Authorization Process | Since users exist in the cluster, permissions can be assigned in advance using either the native authorization system or Apache Ranger. | Although users do not persist, you can predefine role-to-group mappings. When a user logs in, the system assigns roles based on their group, enabling RBAC. Apache Ranger can also be used in parallel. | | Pros & Cons, Use Cases | - **Pros**: Full flexibility—supports both native and external authorization systems.
- **Cons**: Requires manual effort for creating users, which can be cumbersome.
- **Recommended for**: Small user bases or cases where the cluster handles access control. | - **Pros**: Easy to set up; only requires external authentication configuration and allowed group definitions.
- **Recommended for**: Ideal for large user bases with role-group mappings. | These authentication modes can coexist. When a user attempts to log in: 1. The cluster first checks if the user exists as a native user and tries to authenticate accordingly. 2. If the user is not found, the cluster proceeds down the `authentication_chain` as defined in the configuration. This hybrid mode provides both flexibility and control, suitable for different organizational requirements. ###### Option 1: Create Native User with External Authentication System[​](#option-1-create-native-user-with-external-authentication-system "Direct link to Option 1: Create Native User with External Authentication System") For example, you can use the following syntax to create a native user with LDAP: ```sql CREATE USER IDENTIFIED WITH authentication_ldap_simple AS 'uid=tom,ou=company,dc=example,dc=com'; ``` Then, you can `GRANT` privileges or roles to the user, or delegate authorization to external systems like Apache Ranger. ###### Option 2: Use Security Integration with External Authentication System[​](#option-2-use-security-integration-with-external-authentication-system "Direct link to Option 2: Use Security Integration with External Authentication System") You can also create a security integration to allow access of your external authentication service to the cluster. ```sql CREATE SECURITY INTEGRATION PROPERTIES ( "type" = "authentication_ldap_simple", "authentication_ldap_simple_server_host" = "", "authentication_ldap_simple_server_port" = "", "authentication_ldap_simple_bind_base_dn" = "", "authentication_ldap_simple_user_search_attr" = "" "authentication_ldap_simple_bind_root_dn" = "", "authentication_ldap_simple_bind_root_pwd" = "", "authentication_ldap_simple_ssl_conn_allow_insecure" = "{true | false}", "authentication_ldap_simple_ssl_conn_trust_store_path" = "", "authentication_ldap_simple_ssl_conn_trust_store_pwd" = "", "comment" = "" ); ``` After that, you need to configure the FE parameter `authentication_chain` and enable the security integration for your cluster. ```sql ADMIN SET FRONTEND CONFIG ( "authentication_chain" = "[... ,]" ); ``` ##### Group Provider (Optional but Recommended)[​](#group-provider-optional-but-recommended "Direct link to Group Provider (Optional but Recommended)") Group information in the cluster is **decoupled** from both the authentication and authorization systems. It serves as a shared layer that can be independently configured and then used across both login control and access control. ###### How Groups Are Used[​](#how-groups-are-used "Direct link to How Groups Are Used") * **Authentication Stage** When used together with a security integration, group membership can define the scope of who is allowed to log in. Only users who pass authentication and belong to a specified group will be allowed to access the cluster. * **Authorization Stage** Group membership is automatically taken into account during authorization. If privileges are granted to a group, all users within that group will inherit the permissions during access checks. ###### Configuration Notes[​](#configuration-notes "Direct link to Configuration Notes") * When configuring a group provider, you should specify: * Groups used to define **who can log in** (login scope) * Groups used to define **who can access specific resources** (authorization) * **Important**: The user identity (for example, username or ID) returned by the group provider **must match** the identity used during authentication and authorization. Inconsistent identifiers will cause permission or login failures. ###### Example[​](#example-1 "Direct link to Example") The following example is based on LDAP. 1. Create a group provider. ```sql -- LDAP Group Provider CREATE GROUP PROVIDER PROPERTIES ( "type" = "ldap", ldap_info, ldap_search_group_arg, ldap_search_attr, [ldap_cache_attr] ) ldap_info ::= "ldap_conn_url" = "", "ldap_bind_root_dn" = "", "ldap_bind_root_pwd" = "", "ldap_bind_base_dn" = "", ["ldap_conn_timeout" = "",] ["ldap_conn_read_timeout" = ""] ldap_search_group_arg ::= { "ldap_group_dn" = "" | "ldap_group_filter" = "" }, "ldap_group_identifier_attr" = "" ldap_search_user_arg ::= "ldap_group_member_attr" = "", "ldap_user_search_attr" = "" ldap_cache_arg ::= "ldap_cache_refresh_interval" = "" ``` 2. Integrate the group provider with a security integration. ```sql ALTER SECURITY INTEGRATION SET ( "group_provider" = "", "permitted_groups" = "" ) ``` 3. Integrate the group provider with the authorization system. You can use either the native authorization or Apache Ranger. * Native authorization: Roles can be assigned to groups. On login, users are automatically assigned roles based on group membership. ```sql GRANT role TO EXTERNAL GROUP ``` * Apache Ranger: Once a user logs in, StarRocks passes group information to Ranger for policy evaluation. ##### Authorization[​](#authorization "Direct link to Authorization") StarRocks supports **both internal and external authorization mechanisms**, which can be used independently or in combination: * **Internal Authorization** StarRocks provides a built-in **RBAC (Role-Based Access Control)** and **IBAC (Identity-Based Access Control)** system. * **RBAC**: Assigns roles to users or groups, and grants privileges to those roles. * **IBAC**: Grants privileges directly to users. * **External Authorization** StarRocks integrates with **Apache Ranger** to support centralized and unified authorization management. Apache Ranger can be used either as an integral solution itself or together with StarRocks' native authorization system. * **Full Ranger Authorization** Both internal tables and external tables (for example, Hive) are authorized via Ranger. * Internal table permissions use the StarRocks plugin for Ranger. * External table permissions can be managed either via the StarRocks plugin or other external service plugins (for example, Hive plugin). * **Hybrid Authorization** * **Internal tables**: Authorized by StarRocks' native system (RBAC/IBAC). * **External tables**: Authorized via Ranger. External table permissions can still be managed either using the StarRocks plugin or through the appropriate external service (for example, Hive, HDFS). This flexibility allows organizations to gradually migrate to centralized authorization or maintain a hybrid model that fits their current infrastructure and security policies. #### Combo Solutions[​](#combo-solutions "Direct link to Combo Solutions") You can choose the solution based on how you want to finish your authentication and authorization workflow. ##### Solution 1: External authentication + External Authorization[​](#solution-1-external-authentication--external-authorization "Direct link to Solution 1: External authentication + External Authorization") You can fully leverage the external authentication and authorization systems to control login and access permissions for the cluster. The overall process is as follows: 1. Use a **security integration** to establish a connection with the external authentication system. 2. Configure the necessary group information for authentication and authorization within the **group provider**. 3. Define the group(s) allowed to log in to the cluster in the **security integration**. Users who belong to these groups will be granted login access. 4. Create a **StarRocks service** in **Apache Ranger** to manage access control for both internal and external tables. For external tables, you may also reuse existing services for authorization. 5. When a user submits a query, the system will send the user's identity along with their group memberships (as configured in the group provider) to Ranger for authorization. 6. If the authorization check passes, the system will proceed to execute the query. note You must ensure that user IDs and group names remain consistent across all integrated systems throughout this process. ![Authentication and Authorization - Solution-1](/assets/images/auth_solution_1-259aba959f1d16cf4683b8436c0020f9.png) ##### Solution 2: External Authentication (Native User) + Internal Authorization[​](#solution-2-external-authentication-native-user--internal-authorization "Direct link to Solution 2: External Authentication (Native User) + Internal Authorization") If you prefer to use the **built-in authorization system** while still relying on **external authentication**, you can follow this approach: 1. **Manually create users** and specify the external authentication method for each user. 2. After the user is created, use standard `GRANT` statements to assign roles or privileges. 3. Once authenticated, the user will be authorized based on the cluster's native permission system. tip While manually created users can still be integrated with a **group provider** and **Ranger**, this approach is more complex and less automated compared to using **security integration**. Therefore, it is **not a recommended best practice**. ##### Solution 3: External Authentication (External Identity) + Internal Authorization[​](#solution-3-external-authentication-external-identity--internal-authorization "Direct link to Solution 3: External Authentication (External Identity) + Internal Authorization") If you prefer to use **StarRocks' built-in authorization system** while still relying on **external authentication**, you can follow this approach: 1. Use a **security integration** to establish a connection with the external authentication system. 2. Configure the necessary group information for authentication and authorization within the **group provider**. 3. Define the group(s) allowed to log in to the StarRocks cluster in the **security integration**. Users who belong to these groups will be granted login access. 4. **Create the necessary roles** within StarRocks and **grant them to external groups**. 5. When a user attempts to log in, they must both pass authentication and belong to an authorized group. Upon successful login, StarRocks will automatically assign the appropriate roles based on group membership. 6. During query execution, StarRocks will enforce **internal RBAC-based authorization** as usual. 7. Additionally, you can combine **Ranger** with this solution. For example, use **StarRocks' native RBAC** for internal table authorization, and use **Ranger** for external table authorization. When performing authorization via Ranger, StarRocks will still pass the **user ID and corresponding group information** to Ranger for access control. ![Authentication and Authorization - Solution-3](/assets/images/auth_solution_3-69183034840fc0cb794b375326e08043.png) #### See also[​](#see-also "Direct link to See also") * **Authentication** * [Native Authentication](https://docs.starrocks.io/docs/administration/user_privs/authentication/native_authentication.md) * [Security Integration](https://docs.starrocks.io/docs/administration/user_privs/authentication/security_integration.md) * [LDAP Authentication](https://docs.starrocks.io/docs/administration/user_privs/authentication/ldap_authentication.md) * [OAuth 2.0 Authentication](https://docs.starrocks.io/docs/administration/user_privs/authentication/oauth2_authentication.md) * [JSON Web Token Authentication](https://docs.starrocks.io/docs/administration/user_privs/authentication/jwt_authentication.md) * [**Group Provider**](https://docs.starrocks.io/docs/administration/user_privs/group_provider.md) * **Authorization** * [Native Authorization](https://docs.starrocks.io/docs/administration/user_privs/authorization/User_privilege.md) * [Apache Ranger Plugin](https://docs.starrocks.io/docs/administration/user_privs/authorization/ranger_plugin.md) --- ### Bucketing A concise field guide to choosing between Hash Bucketing and Random Bucketing in StarRocks, including their mechanics, trade‑offs, and recommended use cases. *** #### Quick‑Look Comparison[​](#quicklook-comparison "Direct link to Quick‑Look Comparison") | Aspect | Hash Bucketing | Random Bucketing | | ---------------------------------------- | ------------------------------------ | --------------------------------------------------- | | Example | `DISTRIBUTED BY HASH(id) BUCKETS 16` | `DISTRIBUTED BY RANDOM` | | Key declaration | Required HASH(col1, …) | None – rows assigned round‑robin | | Initial bucket count when omitted | Auto‑chosen at CREATE, then fixed | Auto‑chosen at CREATE; can grow if bucket\_size set | | Tablet split / shrink | Manual ALTER … BUCKETS | Automatic split ⇢ growth only (≥ v3.2) | | Skew resistance | Depends on key cardinality | High – uniform by design | | Bucket pruning | ✅ (filters, joins) | 🚫 (full tablet scan) | | Colocate joins | ✅ | 🚫 | | Local aggregation / bucket-shuffle joins | ✅ | 🚫 | | Supported table types | All | Duplicate Key tables only | *** #### Hash Bucketing[​](#hash-bucketing "Direct link to Hash Bucketing") ##### How it Works[​](#how-it-works "Direct link to How it Works") Rows are assigned to tablets by hashing one or more columns. Tablet count is fixed after creation unless manually altered. ##### Requirements[​](#requirements "Direct link to Requirements") * Must pick a stable, evenly, high‑cardinality key up front. The cardinality should typically be 1000 times more than the number of BE nodes to prevent data skew among hash buckets. * Choose an appropriate bucket size initially, ideally ranging between 1 to 10 GB. ##### Strengths[​](#strengths "Direct link to Strengths") * Query locality – selective filters and joins touch fewer tablets. * Colocate joins – fact/dim tables can share hash keys for high‑speed joins. * Predictable layout – rows with the same key always land together. * Local aggregation & bucket‑shuffle joins – identical hash layout across partitions enables local aggregation and reduces data shuffle costs for large join ##### Weaknesses[​](#weaknesses "Direct link to Weaknesses") * Vulnerable to hot tablets if data distribution skews. * Tablet count is static; scaling requires maintenance DDL. * Insufficient tablets can adversely affect data ingestion, data compaction, and query execution parallelism. * Excessive use of tablets will expand the metadata footprint. ##### Example: Dimension‑Fact Join and Tablet Pruning[​](#example-dimensionfact-join-and-tablet-pruning "Direct link to Example: Dimension‑Fact Join and Tablet Pruning") ```sql -- Fact table partitioned and hash‑bucketed by (customer_id) CREATE TABLE sales ( sale_id bigint, customer_id int, sale_date date, amount decimal(10,2) ) ENGINE = OLAP DISTRIBUTED BY HASH(customer_id) BUCKETS 48 PARTITION BY date_trunc('DAY', sale_date) PROPERTIES ("colocate_with" = "group1"); -- Dimension table hash‑bucketed on the same key and bucket count colocated with the sales table CREATE TABLE customers ( customer_id int, region varchar(32), status tinyint ) ENGINE = OLAP DISTRIBUTED BY HASH(customer_id) BUCKETS 48 PROPERTIES ("colocate_with" = "group1"); -- StarRocks can do tablet pruning SELECT sum(amount) FROM sales WHERE customer_id = 123 -- StarRocks can do local aggregation SELECT customer_id, sum(amount) AS total_amount FROM sales GROUP BY customer_id ORDER BY total_amount DESC LIMIT 100; -- StarRocks can do colocate join SELECT c.region, sum(s.amount) FROM sales s JOIN customers c USING (customer_id) WHERE s.sale_date BETWEEN '2025-01-01' AND '2025-01-31' GROUP BY c.region; ``` ###### What do you gain from this example?[​](#what-do-you-gain-from-this-example "Direct link to What do you gain from this example?") * **Tablet pruning**: The customer\_id predicate `WHERE customer_id = 123` enables bucket pruning, allowing the query to access only a single tablet, which lowers latency & CPU cycles, especially for point-lookups. * **Local aggregation**: when the hash distribution key is a subset of the aggregation key, StarRocks can bypass the shuffle aggregation phase, reducing the overall cost. * **Colocated join**: because both tables share bucket number and key, each BE can join its local pair of tablets without network shuffle. ##### When to Use[​](#when-to-use "Direct link to When to Use") * Stable schemas with well‑known distribution filter/join keys. * Data warehousing workloads that benefit from bucket pruning. * You need some specific optimization like colocate join/bucket shuffle join/local aggregation * You are using Aggregate/Primary Key tables. *** #### Random Bucketing[​](#random-bucketing "Direct link to Random Bucketing") ##### How it Works[​](#how-it-works-1 "Direct link to How it Works") Rows are assigned round‑robin; no key specified. With `PROPERTIES ("bucket_size"="")`, StarRocks dynamically splits tablets as partitions grow (v3.2+). ##### Strengths[​](#strengths-1 "Direct link to Strengths") * **Zero design debt**–no keys, no bucket math. * **Skew‑proof writes**–uniform pressure across disks & BEs. * **Elastic growth**–tablet splits keep ingest fast as data or cluster grows. ##### Weaknesses[​](#weaknesses-1 "Direct link to Weaknesses") * **No bucket pruning**–every query scans all tablets in a partition. * **No colocated joins**–keyless layout prevents locality. * Limited to **Duplicate Key** tables today. ##### When to Use[​](#when-to-use-1 "Direct link to When to Use") * Log/event or multi‑tenant SaaS tables where keys change or skew. * Write‑heavy pipelines where uniform ingest throughput is critical. *** #### Operational Guidelines[​](#operational-guidelines "Direct link to Operational Guidelines") * Pick a bucket size (e.g., 1 GiB) for random bucketing to enable auto‑split. * For hash bucketing, monitor tablet size; re‑shard before tablets exceed 5–10 GiB --- ### FE / Coordinator Node memory full issue troubleshooting Prevent "FE memory full" issues, quickly recover, and find out the cause as much as possible to prevent recurrence. note * This information is applicable to FE nodes and Coordinator nodes (shared-data deployments). For readability, we will use "FE" to refer to both types of frontend nodes. * This article mainly analyzes the situation and solutions of in-heap memory, without covering off-heap memory. If the OOM is caused by off-heap memory issues, consider reducing the JVM XMX configuration or expanding the machine's memory. #### FE Memory Structure and Allocation Recommendations[​](#fe-memory-structure-and-allocation-recommendations "Direct link to FE Memory Structure and Allocation Recommendations") ##### FE Memory Composition[​](#fe-memory-composition "Direct link to FE Memory Composition") The front-end service (FE) runs on the JVM, and its memory consumption is from the Java heap and from off-heap memory overhead. ###### Overview of JVM Memory Modules[​](#overview-of-jvm-memory-modules "Direct link to Overview of JVM Memory Modules") All memory allocated through the JVM during the FE process can be roughly divided into the following modules: | Module | Type | Description | | -------------------------------- | -------- | ----------------------------------------------------------------------------------------------------------- | | **Java Heap** | Heap | Memory of Java object instances, such as Plan Cache, metadata, temporary objects, etc. | | **Class** | Off-heap | Structures such as metadata generated when loading a Class | | **Thread** | Off-heap | The memory occupied by threads is mainly the stack space of each thread (usually about 1MB per thread). | | **Code** | Off-heap | Memory occupied by JVM when compiling bytecode into native machine code | | **GC** | Off-heap | Internal working data structures of the garbage collector | | **Compiler** | Off-heap | The memory used by the HotSpot compiler may be similar to that of Code. | | **Other** | Off-heap | Including direct memory such as Direct ByteBuffer, Netty buffer, etc. | | **Symbol** | Off-heap | SymbolTable structures such as string constant pool, integer constant pool, etc. | | **Native Memory Tracking (NMT)** | Off-heap | NMT's own recorded memory usage occupies memory | | **Arena Chunk** | Off-heap | Temporary memory blocks (allocated using malloc) used internally by the JVM for temporary storage and reuse | | **Logging** | Off-heap | Memory occupied by logging systems (such as log4j) | | **Arguments** | Off-heap | Memory occupied by JVM startup parameter passing | | **Module** | Off-heap | Structures used in the Java Module System | Considering the operating characteristics of JVM and StarRocks, the memory of FE mainly consists of the following parts: 1. **JVM Heap Memory (Heap)** * Stores Java object instances, such as Plan Cache, metadata cache, etc. * The maximum size is set via the parameter `-Xmx` and is significantly affected by the garbage collection mechanism. 2. **Non-heap Memory (Metaspace)** * Metadata such as storage class definitions, constant pool, etc. * Java 8 and later replaced PermGen. 3. **Direct Memory** * Off-heap memory used by Netty, ByteBuffer, log buffer, RPC framework, etc. * Allocated by `sun.misc.Unsafe`, usually not controlled by GC. 4. **Thread Stack Memory** * Each Java thread will request a thread stack when created, with a default size of approximately 1MB (configurable via `-Xss`). * When the number of threads is large (such as in high-concurrency scenarios), it is easy to cause memory backlog. 5. **JNI / Native Memory Call** * The database may consume memory when calling the native layer through logging components (such as RocksDB, Netty), Cache, etc. ##### JVM Parameter Configuration Recommendations[​](#jvm-parameter-configuration-recommendations "Direct link to JVM Parameter Configuration Recommendations") In the JVM, the heap memory is configured via `Xmx`. Generally, it is reasonable for the memory used by the process to be less than 130% of `Xmx`. That is to say, in addition to the memory configured by JVM `Xmx`, a portion of off-heap memory will also be used. For example, if JVM is configured with 21g, the RSS memory usage of the process seen in `top` will be approximately 27g. Based on experience, the recommended relationship between the number of tablets and FE memory is: | Tablet Count | Recommended FE Memory Configuration | | ------------ | ----------------------------------- | | Less than 1M | 16 GB | | 1M - 2M | 32 GB | | 2M - 5M | 64 GB | | 5M - 10M | 128 GB | For independently deployed environments, it is recommended to reserve sufficient memory for the system. In addition to the memory reserved for the system, off-heap memory also needs to be considered. For hybrid deployment environments, it is necessary to reserve sufficient memory as much as possible based on the usage considerations of other services to prevent OOM. In the case of independent deployment of FE (mixed deployment is not recommended): 1. When the machine memory is less than 32G, `-Xmx` (Max Heap Size) should be configured to a maximum of 70% of the machine memory. 2. When the machine memory is greater than 32G, `-Xmx` (Max Heap Size) should be configured to a maximum of 80% of the machine memory. 3. Since the FE leader node needs to perform checkpoints, in general, the memory of the leader node will be about twice that of the follower node. In version 3.4+, the leader will select a follower node to perform the checkpoint, then download the follower's results and distribute them to other followers. The leader's memory usage will be close to that of the followers. See [PR #52103](https://github.com/StarRocks/starrocks/pull/52103). ##### FE JVM Monitoring and Memory Profile Deployment[​](#fe-jvm-monitoring-and-memory-profile-deployment "Direct link to FE JVM Monitoring and Memory Profile Deployment") For FE state, heap monitoring and GC monitoring can effectively monitor the state of the service and are also key clues for troubleshooting issues afterwards. It is recommended to configure monitoring in a timely manner, and configure corresponding alerts if necessary. The specific monitoring indicators are as follows: | Indicator | Description | | -------------------------- | ---------------------------------------------------------------------------------- | | `jvm_heap_size_bytes` | Heap memory usage. | | `jvm_non_heap_size_bytes` | Mainly includes metaspace, code cache, etc., does not include all off-heap memory. | | `jvm_old_gc{type="count"}` | Full GC count. | | `jvm_old_gc{type="time"}` | Full GC time. | Monitoring information for these metrics and others is at [Monitoring and alerting](https://docs.starrocks.io/docs/administration/management/monitoring/Monitor_and_Alert.md) Memory profile can analyze sudden heap increases. It is necessary to confirm whether it works properly after the service installation is completed. In 3.3.6+, the database will periodically print the memory profile and output it to `log/proc_profile`, in HTML format, compressed via tgz. Control configuration: ```properties proc_profile_collect_interval_s = 600 proc_profile_collect_time_s = 300 proc_profile_cpu_enable = true proc_profile_mem_enable = true ``` For versions prior to 3.3.6, you can periodically print the profile via a script. Create a new script file named `mem_profiler.sh` under the installation directory of each FE and copy the following content into the file. note This script has no impact on the FE process, does not affect the normal use of the service, and will automatically clean up files that have expired for 48 hours. ```bash #!/bin/bash mkdir -p mem_alloc_log cleanup_old_files() { find mem_alloc_log -name "alloc-profile-*.html" -mmin +2880 -exec rm -f {} \; } while true do cleanup_old_files current_time=$(date +'%Y-%m-%d-%H-%M-%S') file_name="mem_alloc_log/alloc-profile-${current_time}.html" ./bin/profiler.sh -e alloc --alloc 2m -d 300 -f "$file_name" `cat bin/fe.pid` done ``` ```bash # Background startup chmod +x mem_profiler.sh nohup ./mem_profiler.sh > mem_profiler.out 2>&1 & # Check if the process exists ps aux | grep mem_profiler.sh # Stop the process pkill -f mem_profiler.sh ``` #### FE Crash[​](#fe-crash "Direct link to FE Crash") The causes of FE crashes are roughly divided into 4 types. ##### 1. The FE process has high memory usage and was killed by the operating system[​](#1-the-fe-process-has-high-memory-usage-and-was-killed-by-the-operating-system "Direct link to 1. The FE process has high memory usage and was killed by the operating system") You can check whether this is the cause by querying the system logs: ```bash # Check dmesg dmesg | grep -iE 'out of memory|oom|kill|killed process' # View message log sudo grep -Ei 'killed process|oom.kill|sending SIG' -C5 /var/log/messages /var/log/syslog 2>/dev/null | tail -n 100 ``` After being OOM killed, the log may print the memory usage of system processes. Fields such as `total_vm` and `rss` displayed by OOM are in units of pages, with the default being 1 page = 4KB: * `total_vm = 8793525` → indicates that it has requested approximately 8793525 × 4KB ≈ 33.5 GB of virtual memory * `rss = 7218120` → indicates that approximately 27.5 GB of physical memory has been actually used, which is 7218120 × 4KB Example OOM log: ```text Jun 10 11:20:24 tp-prod-bigdata-sr-ss-fe-2-b kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name Jun 10 11:21:58 tp-prod-bigdata-sr-ss-fe-2-b kernel: [22654] 1005 22654 8793525 7218120 14943 0 0 java ``` Check the actual RSS usage. If it is very close to the machine's memory, it is generally due to unreasonable JVM allocation. Refer to the [JVM Parameter Configuration Recommendations](#jvm-parameter-configuration-recommendations) section. ##### 2. The FE leader heap memory is high, triggering a Full GC, leading to a leader switch[​](#2-the-fe-leader-heap-memory-is-high-triggering-a-full-gc-leading-to-a-leader-switch "Direct link to 2. The FE leader heap memory is high, triggering a Full GC, leading to a leader switch") When the old leader exits due to a leader switch, the following information is printed in `fe.out`: ```text transfer FE type from LEADER to UNKNOWN. exit ``` or ```text transfer FE type from LEADER to FOLLOWER. exit ``` Most leader switches are caused by Full GC; in a few cases, they are triggered by high leader CPU usage or the execution of `jstack`. **Analyzing GC logs:** Upload the GC log to and click **Pause GC Duration**. If the duration of a single pause GC exceeds 30 seconds, or if there are frequent pause GCs exceeding 10 seconds, it will trigger a leader switch. Click **Heap before GC** to find the time point when heap memory rises. Use this time range to locate the memory profile file for that period, which will identify which module was requesting memory at that time. ##### 3. An internal bug in the FE program caused a crash[​](#3-an-internal-bug-in-the-fe-program-caused-a-crash "Direct link to 3. An internal bug in the FE program caused a crash") This situation is not within the scope of this discussion. Refer to the [Restore Metadata](https://docs.starrocks.io/docs/administration/Meta_recovery.md) documentation. ##### 4. JVM bug causes FE crash[​](#4-jvm-bug-causes-fe-crash "Direct link to 4. JVM bug causes FE crash") This situation is very rare. After encountering it, the cause can be investigated by checking the crash log, located at `log/hs_err_pid%p.log`, where `%p` is the process ID. #### FE Process Is Normal But Memory Is High[​](#fe-process-is-normal-but-memory-is-high "Direct link to FE Process Is Normal But Memory Is High") During the long-term operation of the FE, the in-heap memory usage may abnormally increase, frequently triggering Full GC, which causes queries or imports to slow down. The problem may manifest as heap **sudden increase** or continuous **slow growth**. ##### Process of On-site Investigation[​](#process-of-on-site-investigation "Direct link to Process of On-site Investigation") Open your monitoring tool: [Monitoring and alerting](https://docs.starrocks.io/docs/administration/management/monitoring/Monitor_and_Alert.md) **Step 1: Check the following monitoring** * **FE JVM Panel**: JVM Heap Memory Used metric **Step 2: Identify the type of memory increase** | Type | Feature | Subsequent Troubleshooting Path | | ------------------- | --------------------------------------------------------------------- | ------------------------------------------------------------------ | | **Sudden increase** | Heap memory spikes sharply within a short period (seconds to minutes) | Prioritize looking at QPS, number of connections, and SQL requests | | **Slowly rises** | Memory steadily increases, and post-GC recovery deteriorates. | Prioritize troubleshooting memory leaks or cache accumulation | ##### Memory Spike Investigation[​](#memory-spike-investigation "Direct link to Memory Spike Investigation") Whether the request pressure increases within a short period — high QPS or highly concurrent metadata operations can lead to a surge in memory. **1. Check the following monitoring:** * **QPS Panel**: Has the number of requests received by FE suddenly increased? * **Connections Panel**: Does the number of concurrent connections of FE spike instantaneously? **2. Confirm abnormal SQL through audit tables or logs:** Query high-frequency SQL statements in the last 5-10 minutes: ```sql SELECT * FROM starrocks_audit_db__.starrocks_audit_tbl__ WHERE feip = 'IP of memory abnormal node' AND `timestamp` >= now() - INTERVAL 1000 MINUTE ORDER BY `timestamp` DESC; ``` Pay special attention to: * Statements with deep subqueries, multiple JOINs, and complex aggregations * Unconventional business SQL (such as concurrent SQL for datasets generated by the BI platform) * In version 3.3.13+, you can check the FE audit log for the `queryFeMemory` field, which indicates how much memory a query has applied for in total in the FE. * When the FE process exits due to FE OOM, it will automatically print the query being executed and the corresponding `QueryFEAllocatedMemory`. Search keyword in FE log: ```text QueryFEAllocatedMemory ``` **3. Check through the FE log for a large number of metadata operations (table creation and deletion):** ```bash # Create a table grep "Begin to unprotect create table" fe.log # Drop a table grep "Finished drop table" fe.log ``` ##### Memory Slowly Increasing[​](#memory-slowly-increasing "Direct link to Memory Slowly Increasing") **4. The memory configured for the JVM is too small** Monitor the GC status of the FE process to determine whether Full GC occurs frequently. Check the JVM memory usage through the `jstat` command: ```bash jstat -gcutil $pid 1000 1000 ``` Example output: ```text S0 S1 E O M CCS YGC YGCT FGC FGCT GCT 0.00 100.00 27.78 95.45 97.77 94.45 24 0.226 1 0.065 0.291 0.00 100.00 44.44 95.45 97.77 94.45 24 0.226 1 0.065 0.291 0.00 100.00 55.56 95.45 97.77 94.45 24 0.226 1 0.065 0.291 ``` If the `O` (Old Generation) percentage has been relatively high, it indicates a problem with the JVM memory configuration, and the JVM memory needs to be increased. **5. Observe whether there is a leak through the FE's memory tracker (3.3.7+)** You can locate which module has a memory leak by checking the logs of `MemoryUsageTracker`, which prints the memory consumption of each module once every hour: ```text 2025-06-06 16:37:50.633+08:00 INFO (MemoryUsageTracker|74) [MemoryUsageTracker.trackMemory():161] (6ms) Module Dict - CacheDictManager estimated 0B of memory. Contains ColumnDict with 0 object(s). 2025-06-06 16:37:50.657+08:00 INFO (MemoryUsageTracker|74) [MemoryUsageTracker.trackMemory():161] (21ms) Module LocalMetastore - LocalMetastore estimated 3.8MB of memory. Contains Partition with 17473 object(s). 2025-06-06 16:37:50.667+08:00 INFO (MemoryUsageTracker|74) [MemoryUsageTracker.trackMemory():161] (0ms) Module TabletInvertedIndex - TabletInvertedIndex estimated 37.7MB of memory. Contains TabletMeta with 353459 object(s). 2025-06-06 16:37:50.741+08:00 INFO (MemoryUsageTracker|74) [MemoryUsageTracker.trackMemory():108] total tracked memory: 46.8MB, jvm: Process used: 18.6GB, heap used: 4.4GB, non heap used: 289.1MB, direct buffer used: 395.5MB ``` Compare the memory growth of each module across recent log entries to identify which module is growing. ##### On-site Information Collection[​](#on-site-information-collection "Direct link to On-site Information Collection") **If the process has not been restarted yet, collect `jmap` information immediately.** ###### jmap Memory Troubleshooting Process[​](#jmap-memory-troubleshooting-process "Direct link to jmap Memory Troubleshooting Process") **Step 1: Confirm the process PID of FE** ```bash ps aux | grep FE ``` **Step 2: Use jmap** Precautions * `jmap -dump` will pause the FE process for a short period of time. Use with caution when high online stability is required, and it is recommended to notify in advance. * `jmap -histo` generally has no impact on the process; GC triggering may take dozens of milliseconds. * Frequent use of `-dump` to export heap files is not recommended, as it incurs significant overhead. * `histo` is sufficient for preliminary judgment of large objects and does not necessarily require a dump. **`jmap -histo:live` (Use with caution in production environments)** Captures the current in-heap object distribution by forcing a GC. Using the `live` parameter may resolve high memory usage issues: ```bash jmap -histo:live > jmap_histo_$(date +%s).log ``` It is recommended to take two samples and compare the differences: ```bash jmap -histo:live > histo_1.log sleep 60 jmap -histo:live > histo_2.log ``` **`jmap -histo pid` (lightweight, does not trigger Full GC)** ```bash # Retrieves all objects within the current JVM heap, which may include # objects that have been cleaned up — will affect result analysis. jmap -histo > histo.txt ``` **Step 3: Analyze Top N Object Occupancy** View the top entries of the file: ```bash head -n 30 histo_2.log ``` Field explanation: | Column | Meaning | | ---------- | -------------- | | num | Class number | | #instances | Instance count | | #bytes | Total bytes | | class name | Object name | Compare `histo_1` and `histo_2` to identify which classes have significantly increased instances or memory. Example: Classes starting with `java` are utility classes referenced by business classes. Looking from top to bottom, related classes at the front account for a larger proportion. For example, `com.starrocks.lake.LakeTablet` occupying significant memory indicates excessive tablet usage. **Step 4: Export a complete heap dump (if histo is insufficient)** ```bash # Will trigger Full GC. Use with caution. jmap -dump:live,format=b,file=heap_$(date +%s).hprof ``` This file may be very large (several GB) and needs to be opened and analyzed using **MAT** or **VisualVM**. **Step 5: Configure automatic dump on OOM** Add the following to the JVM configuration in `fe.conf` to automatically generate a dump file when FE runs out of memory: ```bash JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true \ -Xmx8192m \ -XX:+UseG1GC \ -Xlog:gc*:${LOG_DIR}/fe.gc.log.$DATE:time \ -XX:ErrorFile=${LOG_DIR}/hs_err_pid%p.log \ -XX:+HeapDumpOnOutOfMemoryError \ -XX:HeapDumpPath=${LOG_DIR}/heap_dump_oom.hprof \ -Djava.security.policy=${STARROCKS_HOME}/conf/udf_security.policy \ -Djava.security.krb5.conf=/etc/krb5.conf \ -Dsun.security.krb5.debug=true \ -Dsun.security.spnego.debug=true \ -Dcom.sun.management.jmxremote \ -Dcom.sun.management.jmxremote.authenticate=false \ -Dcom.sun.management.jmxremote.ssl=false \ -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:8113" ``` Note: `HeapDumpPath` specifies a directory path, not a file name. ###### Keep Memory Profile Files for Analysis[​](#keep-memory-profile-files-for-analysis "Direct link to Keep Memory Profile Files for Analysis") Starting from version 3.3.6, memory profiles are output to the `log/proc_profile` directory, in HTML format, compressed using tgz. Find the file corresponding to the time of memory increase, decompress it, and open it in a browser to see the flame graph of memory allocation. The wider part of the stack represents more memory allocation. If no memory profile is printed during the period of memory increase, you can turn off the CPU profile and adjust the printing interval to 5 minutes: ```properties proc_profile_cpu_enable = false proc_profile_collect_interval_s = 300 ``` These are dynamic parameters that can be modified directly without restart: ```sql ADMIN SET FRONTEND CONFIG ("proc_profile_cpu_enable" = "false"); ADMIN SET FRONTEND CONFIG ("proc_profile_collect_interval_s" = "300"); ``` #### Emergency Recovery[​](#emergency-recovery "Direct link to Emergency Recovery") **1. Stop the FE service** **2. Adjust the heap memory configuration to a larger value and restart** Modify the `-Xmx` value in `fe.conf`: ```bash JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true -Xmx8192m -XX:+UseG1GC \ -Xlog:gc*:${LOG_DIR}/fe.gc.log.$DATE:time \ -XX:ErrorFile=${LOG_DIR}/hs_err_pid%p.log \ -XX:+HeapDumpOnOutOfMemoryError \ -XX:HeapDumpPath=${LOG_DIR}/heap_dump_oom.hprof \ -Djava.security.policy=${STARROCKS_HOME}/conf/udf_security.policy \ -Dcom.sun.management.jmxremote.authenticate=false \ -Dcom.sun.management.jmxremote.ssl=false \ -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:8113" ``` **3. If the machine itself has limited memory**, restarting may still result in OOM issues. Consider expanding the machine's memory, then increasing the JVM configuration before restarting. **4. If a historically corresponding PR is found in the case**, it is recommended to upgrade to the corresponding fixed version. #### Case Collection[​](#case-collection "Direct link to Case Collection") ##### Case 1: Frequent and large-scale metadata operations cause high FE memory usage[​](#case-1-frequent-and-large-scale-metadata-operations-cause-high-fe-memory-usage "Direct link to Case 1: Frequent and large-scale metadata operations cause high FE memory usage") **Background**: A client's business requires regular table creation and cleanup, with sequential operations creating dozens of tables per second and synchronous operations deleting them, resulting in the creation of hundreds of thousands of tables throughout the process. **Problem manifestation**: Heap memory and machine CPU continue to rise, and query response slows down. After checking the FE logs, a large number of table creation and deletion operations were found, which led to a sharp increase in service pressure. **Recovery Method**: Stop the table creation and deletion jobs, restart the service, and resume normal use. **Solution**: For this business scenario, when using `DROP TABLE`, `FORCE` must be added; otherwise, the metadata will continue to be occupied. At the same time, reduce the concurrency frequency of table creation. *** ##### Case 2: Complex concurrent SQL causes a sudden increase in FE memory[​](#case-2-complex-concurrent-sql-causes-a-sudden-increase-in-fe-memory "Direct link to Case 2: Complex concurrent SQL causes a sudden increase in FE memory") **Problem manifestation**: Heap memory suddenly increased, and scheduled import tasks were significantly delayed. QPS and the number of connections were within the normal range of variation. Review of audit logs and FE logs revealed many highly complex queries in concurrent scheduled SQL, which resulted in a large amount of memory being occupied during query plan parsing. **Recovery Method**: Expand the memory of the FE service — stop the machine to expand its memory, increase the JVM XMX configuration, and restart the service. **Solution**: Split complex SQL, observe the memory usage during peak job periods, and ensure it stays below 80%. *** ##### Case 3: MV refresh is too frequent, causing the leader FE memory to grow slowly and remain high[​](#case-3-mv-refresh-is-too-frequent-causing-the-leader-fe-memory-to-grow-slowly-and-remain-high "Direct link to Case 3: MV refresh is too frequent, causing the leader FE memory to grow slowly and remain high") **Background**: The user has MV refresh tasks, and many tasks are configured for minute or second-level refresh. **Problem manifestation**: FE leader memory continues to rise, and queries are laggy. Checking the FE log reveals many slow locks, causing occupied resources to fail to be released. Example slow lock log: ```text 2025-06-06 15:41:43.813+08:00 WARN (AutoStatistic|41) [LockManager.logSlowLockTrace():423] LockManager detects slow lock : {"owners":[{"id":13479085,"name":"starrocks-taskrun-pool-22075","type":"INTENTION_SHARED","heldFor":6935,"queryId":"ab976f93-42a9-11f0-98e6-fa163e3710f8","waitTime":0,"stack":["java.base@11.0.20.1/java.net.SocketInputStream.socketRead0(Native Method)","java.base@11.0.20.1/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)","java.base@11.0.20.1/java.net.SocketInputStream.read(SocketInputStream.java:168)","java.base@11.0.20.1/java.net.SocketInputStream.read(SocketInputStream.java:140)","app//org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:161)","app//org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:128)","app//org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:113)","app//org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)","app//org.postgresql.core.PGStream.receiveChar(PGStream.java:453)","app//org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2120)","app//org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:356)","app//org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:496)","app//org.postgresql.jdbc.PgStatement.execute(PgStatement.java:413)","app//org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:333)","app//org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:319)","app//org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:295)","app//org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:244)","app//org.postgresql.jdbc.PgDatabaseMetaData.getColumns(PgDatabaseMetaData.java:1584)","app//com.starrocks.connector.jdbc.PostgresSchemaResolver.getColumns(PostgresSchemaResolver.java:49)","app//com.starrocks.connector.jdbc.JDBCMetadata.lambda$getTable$1(JDBCMetadata.java:200)","app//com.starrocks.connector.jdbc.JDBCMetadata$$Lambda$1340/0x000014e17aa20960.apply(Unknown Source)","app//com.starrocks.connector.jdbc.JDBCMetaCache.get(JDBCMetaCache.java:73)","app//com.starrocks.connector.jdbc.JDBCMetadata.getTable(JDBCMetadata.java:197)","app//com.starrocks.connector.CatalogConnectorMetadata.getTable(CatalogConnectorMetadata.java:136)","app//com.starrocks.server.MetadataMgr.lambda$getTable$5(MetadataMgr.java:501)","app//com.starrocks.server.MetadataMgr$$Lambda$513/0x000014e2116f0cb0.apply(Unknown Source)","java.base@11.0.20.1/java.util.Optional.map(Optional.java:265)","app//com.starrocks.server.MetadataMgr.getTable(MetadataMgr.java:501)","app//com.starrocks.sql.analyzer.QueryAnalyzer.resolveTable(QueryAnalyzer.java:1391)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.resolveTableRef(QueryAnalyzer.java:482)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.resolveTableRef(QueryAnalyzer.java:420)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.resolveTableRef(QueryAnalyzer.java:420)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitSelect(QueryAnalyzer.java:363)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitSelect(QueryAnalyzer.java:283)","app//com.starrocks.sql.ast.SelectRelation.accept(SelectRelation.java:232)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.process(QueryAnalyzer.java:288)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitQueryRelation(QueryAnalyzer.java:303)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitQueryStatement(QueryAnalyzer.java:293)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitQueryStatement(QueryAnalyzer.java:283)","app//com.starrocks.sql.ast.QueryStatement.accept(QueryStatement.java:70)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.process(QueryAnalyzer.java:288)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitSubquery(QueryAnalyzer.java:907)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitSubquery(QueryAnalyzer.java:283)","app//com.starrocks.sql.ast.SubqueryRelation.accept(SubqueryRelation.java:66)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.process(QueryAnalyzer.java:288)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitSelect(QueryAnalyzer.java:370)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitSelect(QueryAnalyzer.java:283)","app//com.starrocks.sql.ast.SelectRelation.accept(SelectRelation.java:232)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.process(QueryAnalyzer.java:288)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitQueryRelation(QueryAnalyzer.java:303)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitQueryStatement(QueryAnalyzer.java:293)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitQueryStatement(QueryAnalyzer.java:283)","app//com.starrocks.sql.ast.QueryStatement.accept(QueryStatement.java:70)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.process(QueryAnalyzer.java:288)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitSubquery(QueryAnalyzer.java:907)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitSubquery(QueryAnalyzer.java:283)","app//com.starrocks.sql.ast.SubqueryRelation.accept(SubqueryRelation.java:66)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.process(QueryAnalyzer.java:288)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitJoin(QueryAnalyzer.java:739)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitJoin(QueryAnalyzer.java:283)","app//com.starrocks.sql.ast.JoinRelation.accept(JoinRelation.java:134)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.process(QueryAnalyzer.java:288)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitSelect(QueryAnalyzer.java:370)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitSelect(QueryAnalyzer.java:283)","app//com.starrocks.sql.ast.SelectRelation.accept(SelectRelation.java:232)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.process(QueryAnalyzer.java:288)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitQueryRelation(QueryAnalyzer.java:303)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitQueryStatement(QueryAnalyzer.java:293)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitQueryStatement(QueryAnalyzer.java:283)","app//com.starrocks.sql.ast.QueryStatement.accept(QueryStatement.java:70)","app//com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.process(QueryAnalyzer.java:288)","app//com.starrocks.sql.analyzer.QueryAnalyzer.analyze(QueryAnalyzer.java:121)","app//com.starrocks.sql.analyzer.InsertAnalyzer.analyzeWithDeferredLock(InsertAnalyzer.java:74)","app//com.starrocks.sql.analyzer.InsertAnalyzer.analyze(InsertAnalyzer.java:62)","app//com.starrocks.sql.analyzer.Analyzer$AnalyzerVisitor.visitInsertStatement(Analyzer.java:398)","app//com.starrocks.sql.analyzer.Analyzer$AnalyzerVisitor.visitInsertStatement(Analyzer.java:176)","app//com.starrocks.sql.ast.InsertStmt.accept(InsertStmt.java:304)","app//com.starrocks.sql.ast.AstVisitor.visit(AstVisitor.java:80)","app//com.starrocks.sql.analyzer.Analyzer.analyze(Analyzer.java:173)","app//com.starrocks.sql.StatementPlanner.analyzeStatement(StatementPlanner.java:218)","app//com.starrocks.sql.StatementPlanner.plan(StatementPlanner.java:116)","app//com.starrocks.sql.StatementPlanner.plan(StatementPlanner.java:95)","app//com.starrocks.load.InsertOverwriteJobRunner.executeInsert(InsertOverwriteJobRunner.java:351)","app//com.starrocks.load.InsertOverwriteJobRunner.doLoad(InsertOverwriteJobRunner.java:171)","app//com.starrocks.load.InsertOverwriteJobRunner.handle(InsertOverwriteJobRunner.java:151)","app//com.starrocks.load.InsertOverwriteJobRunner.transferTo(InsertOverwriteJobRunner.java:212)","app//com.starrocks.load.InsertOverwriteJobRunner.prepare(InsertOverwriteJobRunner.java:256)","app//com.starrocks.load.InsertOverwriteJobRunner.handle(InsertOverwriteJobRunner.java:148)","app//com.starrocks.load.InsertOverwriteJobRunner.run(InsertOverwriteJobRunner.java:136)","app//com.starrocks.load.InsertOverwriteJobMgr.executeJob(InsertOverwriteJobMgr.java:91)","app//com.starrocks.qe.StmtExecutor.handleInsertOverwrite(StmtExecutor.java:2152)","app//com.starrocks.qe.StmtExecutor.handleDMLStmt(StmtExecutor.java:2244)","app//com.starrocks.qe.StmtExecutor.handleDMLStmtWithProfile(StmtExecutor.java:2161)","app//com.starrocks.scheduler.PartitionBasedMvRefreshProcessor.refreshMaterializedView(PartitionBasedMvRefreshProcessor.java:1271)","app//com.starrocks.scheduler.PartitionBasedMvRefreshProcessor.doRefreshMaterializedView(PartitionBasedMvRefreshProcessor.java:464)","app//com.starrocks.scheduler.PartitionBasedMvRefreshProcessor.doRefreshMaterializedViewWithRetry(PartitionBasedMvRefreshProcessor.java:373)","app//com.starrocks.scheduler.PartitionBasedMvRefreshProcessor.doMvRefresh(PartitionBasedMvRefreshProcessor.java:332)","app//com.starrocks.scheduler.PartitionBasedMvRefreshProcessor.processTaskRun(PartitionBasedMvRefreshProcessor.java:200)","app//com.starrocks.scheduler.TaskRun.executeTaskRun(TaskRun.java:285)","app//com.starrocks.scheduler.TaskRunExecutor.lambda$executeTaskRun$0(TaskRunExecutor.java:60)","app//com.starrocks.scheduler.TaskRunExecutor$$Lambda$3311/0x000014db15a06cb0.get(Unknown Source)","java.base@11.0.20.1/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)","java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)","java.base@11.0.20.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)","java.base@11.0.20.1/java.lang.Thread.run(Thread.java:829)"]}],"waiter":[{"id":13446845,"name":"thrift-server-pool-367701","type":"WRITE","waitTime":6895},{"id":5268046,"name":"lake-publish-task-8973","type":"INTENTION_EXCLUSIVE","waitTime":6895},{"id":5268447,"name":"lake-publish-task-9181","type":"INTENTION_EXCLUSIVE","waitTime":6874},{"id":5267898,"name":"lake-publish-task-8911","type":"INTENTION_EXCLUSIVE","waitTime":6874},{"id":5267726,"name":"lake-publish-task-8827","type":"INTENTION_EXCLUSIVE","waitTime":6874},{"id":13494208,"name":"thrift-server-pool-369089","type":"INTENTION_EXCLUSIVE","waitTime":6857},{"id":13492395,"name":"starrocks-taskrun-pool-22096","type":"INTENTION_EXCLUSIVE","waitTime":6854},{"id":41,"name":"AutoStatistic","type":"INTENTION_SHARED","waitTime":6854,"queryId":"ad195549-42a9-11f0-98e6-fa163e3710f8"},{"id":486,"name":"nioEventLoopGroup-6-33","type":"INTENTION_SHARED","waitTime":6852},{"id":5267924,"name":"lake-publish-task-8925","type":"INTENTION_SHARED","waitTime":6849},{"id":5271934,"name":"lake-publish-task-9265","type":"INTENTION_SHARED","waitTime":6849},{"id":5344442,"name":"lake-publish-task-9312","type":"INTENTION_EXCLUSIVE","waitTime":6786},{"id":13471238,"name":"thrift-server-pool-368401","type":"INTENTION_SHARED","waitTime":55},{"id":13494164,"name":"thrift-server-pool-369065","type":"INTENTION_SHARED","waitTime":55},{"id":13481141,"name":"thrift-server-pool-368676","type":"INTENTION_SHARED","waitTime":54}]} ``` **Recovery Method**: After leader GC, it switches to the primary and self-recovers. **Solution**: Relax the refresh frequency. Adjust non-real-time strongly dependent scenarios to hourly or even daily levels to avoid a large number of minute-level MVs being triggered simultaneously. Once slow locks disappear, the problem is resolved. *** ##### Case 4: Querying external tables in the Iceberg catalog causes an increase in FE memory[​](#case-4-querying-external-tables-in-the-iceberg-catalog-causes-an-increase-in-fe-memory "Direct link to Case 4: Querying external tables in the Iceberg catalog causes an increase in FE memory") **Background**: User executes a query job on an Iceberg table. **Problem manifestation**: After submitting the SQL, the memory of the FE leader increased significantly, and CPU also continued to rise. Other query and import tasks reported errors. Checking the process status revealed that frequent GC was triggered, causing abnormal service status. **Recovery Method**: Restart the leader node and regularly manage the delete files of the corresponding Iceberg table. *** ##### Case 5: Insert memory leak issue causes abnormal increase in FE memory[​](#case-5-insert-memory-leak-issue-causes-abnormal-increase-in-fe-memory "Direct link to Case 5: Insert memory leak issue causes abnormal increase in FE memory") **Background**: During user operation, it was found that the FE memory abnormally increased, the process restarted, and the leader was not properly switched. The main task of the client cluster was import tasks. **Problem manifestation**: Export `jmap` for the leader node process for troubleshooting: ```bash jmap -histo > jmap.txt # This operation will not trigger Full GC ``` Found in the exported file: ```bash 14: 521358 154321968 com.starrocks.load.loadv2.InsertLoadJob ``` The number of `InsertLoadJob` instances is 520,000. When this number exceeds 10,000, it indicates that a memory leak has occurred. * \[Issue]\( * [Fix PR](https://github.com/StarRocks/starrocks/pull/53809) **Affected versions**: * 3.1.0 - 3.1.16 * 3.2.0 - 3.2.12 * 3.3.0 - 3.3.7 **Fixed versions**: 3.1.17+, 3.2.13+, 3.3.8+ **Recovery Method**: Upgrade the cluster version. *** ##### Case 6: async-profiler causes JVM crash, leading to FE crash[​](#case-6-async-profiler-causes-jvm-crash-leading-to-fe-crash "Direct link to Case 6: async-profiler causes JVM crash, leading to FE crash") **Problem manifestation**: FE process crashed or restarted. **Problem Troubleshooting**: When the process is abnormal and the file `hs_err_pid$pid.log` is generated in the log directory, the FE crash may be caused by a JVM crash. ```text --------------- S U M M A R Y ------------ ... --------------- T H R E A D --------------- Current thread: JavaThread "tablet scheduler" ... Native frames: V [libjvm.so] frame::entry_frame_is_first() const V [libjvm.so] forte_fill_call_trace_given_top(...) V [libjvm.so] AsyncGetCallTrace C [libasyncProfiler.so] Profiler::getJavaTraceAsync(...) C [libasyncProfiler.so] Profiler::recordSample(...) C [libasyncProfiler.so] PerfEvents::signalHandler(...) ``` The crash breakpoint `PerfEvents::signalHandler` is caused by `async-profiler`. The tool triggered a segmentation fault inside the JVM while attempting to obtain the Java call stack. In the enterprise version, this tool is deployed by default to periodically obtain process information. **Processing Method**: Stop `async-profiler` CPU information collection: ```sql ADMIN SET FRONTEND CONFIG ("proc_profile_cpu_enable" = "false"); ``` Also add the following to `fe.conf`: ```properties proc_profile_cpu_enable = false ``` *** ##### Case 7: OOM caused by insufficient JVM heap configuration[​](#case-7-oom-caused-by-insufficient-jvm-heap-configuration "Direct link to Case 7: OOM caused by insufficient JVM heap configuration") Under normal circumstances, with 3 FEs, if the leader node crashes due to insufficient memory, the remaining 2 FEs can elect a new leader and provide services. If the 2 follower nodes cannot communicate properly after the leader crashes, the FE that is about to be promoted to leader may also exit due to insufficient metadata replicas, resulting in 2 FEs crashing. **Problem Troubleshooting**: FE leader log shows: ```text java.lang.OutOfMemoryError: Java heap space 2025-08-09 08:22:30.926-06:00 WARN (thrift-server-pool-9443633|65019425) [TIOStreamTransport.close():153] Error closing output stream. java.net.SocketException: Socket closed at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113) ~[?:?] at java.net.SocketOutputStream.write(SocketOutputStream.java:150) ~[?:?] at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81) ~[?:?] at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142) ~[?:?] at java.io.FilterOutputStream.close(FilterOutputStream.java:182) ~[?:?] at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:151) ~[libthrift-0.20.0.jar:0.20.0] at org.apache.thrift.transport.TSocket.close(TSocket.java:238) ~[libthrift-0.20.0.jar:0.20.0] at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:326) ~[starrocks-fe.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:829) ~[?:?] 2025-08-09 08:22:30.927-06:00 ERROR (thrift-server-accept|144) [SRTThreadPoolServer.execute():221] ExecutorService threw error: java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space 2025-08-09 08:22:30.931-06:00 WARN (starrocks-mysql-nio-pool-42252|65019486) [StmtExecutor.execute():595] New planner error: ... com.starrocks.sql.common.StarRocksPlannerException: StarRocks planner use long time 3000 ms in memo phase, This probably because 1. FE Full GC, 2. Hive external table fetch metadata took a long time, 3. The SQL is very complex. You could 1. adjust FE JVM config, 2. try query again, 3. enlarge new_planner_optimize_timeout session variable at com.starrocks.sql.optimizer.task.SeriallyTaskScheduler.executeTasks(SeriallyTaskScheduler.java:50) ~[starrocks-fe.jar:?] at com.starrocks.sql.optimizer.Optimizer.memoOptimize(Optimizer.java:900) ~[starrocks-fe.jar:?] at com.starrocks.sql.optimizer.Optimizer.optimizeByCost(Optimizer.java:272) ~[starrocks-fe.jar:?] at com.starrocks.sql.optimizer.Optimizer.optimize(Optimizer.java:196) ~[starrocks-fe.jar:?] at com.starrocks.sql.StatementPlanner.createQueryPlanWithReTry(StatementPlanner.java:348) ~[starrocks-fe.jar:?] at com.starrocks.sql.StatementPlanner.plan(StatementPlanner.java:138) ~[starrocks-fe.jar:?] at com.starrocks.sql.StatementPlanner.plan(StatementPlanner.java:95) ~[starrocks-fe.jar:?] at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:580) ~[starrocks-fe.jar:?] at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:389) ~[starrocks-fe.jar:?] at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:598) ~[starrocks-fe.jar:?] at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:936) ~[starrocks-fe.jar:?] at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:69) ~[starrocks-fe.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:829) ~[?:?] 2025-08-09 08:22:30.932-06:00 INFO (TaskCleaner|111) [TaskManager.dropTasks():392] drop tasks:[] 2025-08-09 08:22:35.431-06:00 INFO (main|1) [StarRocksFE.start():123] StarRocks FE starting, version: 3.3.14-ee-5b29ea9 ``` The FE being promoted to leader fails: ```text at com.starrocks.common.util.Daemon.run(Daemon.java:109) ~[starrocks-fe.jar:?] 2025-08-09 08:16:07.844-06:00 ERROR (stateChangeExecutor|86) [GlobalStateMgr.transferToLeader():1333] failed to init journal after transfer to leader! will exit com.starrocks.journal.JournalException: catch exception after retried 3 times at com.starrocks.journal.bdbje.BDBJEJournal.open(BDBJEJournal.java:222) ~[starrocks-fe.jar:?] at com.starrocks.server.GlobalStateMgr.transferToLeader(GlobalStateMgr.java:1323) ~[starrocks-fe.jar:?] at com.starrocks.server.GlobalStateMgr$1.transferToLeader(GlobalStateMgr.java:795) ~[starrocks-fe.jar:?] at com.starrocks.ha.StateChangeExecutor.runOneCycle(StateChangeExecutor.java:125) ~[starrocks-fe.jar:?] at com.starrocks.common.util.Daemon.run(Daemon.java:109) ~[starrocks-fe.jar:?] Caused by: com.sleepycat.je.rep.InsufficientReplicasException: (JE 18.3.20) Commit policy: SIMPLE_MAJORITY required 1 replica. But none were active with this master. at com.sleepycat.je.rep.impl.node.DurabilityQuorum.ensureReplicasForCommit(DurabilityQuorum.java:116) ~[starrocks-bdb-je-18.3.20.jar:?] at com.sleepycat.je.rep.impl.RepImpl.txnBeginHook(RepImpl.java:1171) ~[starrocks-bdb-je-18.3.20.jar:?] at com.sleepycat.je.rep.txn.MasterTxn.txnBeginHook(MasterTxn.java:195) ~[starrocks-bdb-je-18.3.20.jar:?] at com.sleepycat.je.txn.Txn.initTxn(Txn.java:384) ~[starrocks-bdb-je-18.3.20.jar:?] at com.sleepycat.je.txn.Txn.(Txn.java:288) ~[starrocks-bdb-je-18.3.20.jar:?] at com.sleepycat.je.txn.Txn.(Txn.java:267) ~[starrocks-bdb-je-18.3.20.jar:?] at com.sleepycat.je.rep.txn.MasterTxn.(MasterTxn.java:146) ~[starrocks-bdb-je-18.3.20.jar:?] at com.sleepycat.je.rep.txn.MasterTxn$1.create(MasterTxn.java:117) ~[starrocks-bdb-je-18.3.20.jar:?] at com.sleepycat.je.rep.txn.MasterTxn.create(MasterTxn.java:435) ~[starrocks-bdb-je-18.3.20.jar:?] at com.sleepycat.je.rep.impl.RepImpl.createRepUserTxn(RepImpl.java:1145) ~[starrocks-bdb-je-18.3.20.jar:?] at com.sleepycat.je.txn.Txn.createAutoTxn(Txn.java:334) ~[starrocks-bdb-je-18.3.20.jar:?] at com.sleepycat.je.txn.LockerFactory.getWritableLocker(LockerFactory.java:79) ~[starrocks-bdb-je-18.3.20.jar:?] at com.sleepycat.je.Environment.setupDatabase(Environment.java:816) ~[starrocks-bdb-je-18.3.20.jar:?] at com.sleepycat.je.Environment.openDatabase(Environment.java:668) ~[starrocks-bdb-je-18.3.20.jar:?] at com.starrocks.journal.bdbje.BDBEnvironment.openDatabase(BDBEnvironment.java:446) ~[starrocks-fe.jar:?] at com.starrocks.journal.bdbje.BDBJEJournal.open(BDBJEJournal.java:213) ~[starrocks-fe.jar:?] ... 4 more 2025-08-09 08:16:07.845-06:00 INFO (Thread-69|145) [StarRocksFE.lambda$addShutdownHook$1():374] start to execute shutdown hook 2025-08-09 08:16:07.852-06:00 WARN (Thread-70|2103846) [ConnectScheduler.lambda$printAllRunningQuery$4():339] FE ShutDown! Running Query:show frontends;, QueryFEAllocatedMemory: 69224 2025-08-09 08:16:07.853-06:00 WARN (Thread-70|2103846) [ConnectScheduler.lambda$printAllRunningQuery$4():339] FE ShutDown! Running Query:show frontends;, QueryFEAllocatedMemory: 31128 2025-08-09 08:16:07.853-06:00 WARN (Thread-70|2103846) [ConnectScheduler.lambda$printAllRunningQuery$4():339] FE ShutDown! Running Query:SET NAMES utf8, QueryFEAllocatedMemory: 64464 2025-08-09 08:16:07.853-06:00 WARN (Thread-70|2103846) [ConnectScheduler.lambda$printAllRunningQuery$4():339] FE ShutDown! Running Query:SET NAMES utf8, QueryFEAllocatedMemory: 119040 2025-08-09 08:16:07.853-06:00 WARN (Thread-70|2103846) [ConnectScheduler.lambda$printAllRunningQuery$4():339] FE ShutDown! Running Query:show frontends;, QueryFEAllocatedMemory: 83760 2025-08-09 08:16:07.853-06:00 WARN (Thread-70|2103846) [ConnectScheduler.lambda$printAllRunningQuery$4():339] FE ShutDown! Running Query:show frontends;, QueryFEAllocatedMemory: 69080 2025-08-09 08:16:07.853-06:00 WARN (Thread-70|2103846) [ConnectScheduler.lambda$printAllRunningQuery$4():339] FE ShutDown! Running Query:show frontends;, QueryFEAllocatedMemory: 83760 2025-08-09 08:16:07.853-06:00 WARN (Thread-70|2103846) [ConnectScheduler.lambda$printAllRunningQuery$4():339] FE ShutDown! Running Query:show frontends;, QueryFEAllocatedMemory: 84072 2025-08-09 08:16:07.853-06:00 WARN (Thread-70|2103846) [ConnectScheduler.lambda$printAllRunningQuery$4():339] FE ShutDown! Running Query:SET NAMES utf8, QueryFEAllocatedMemory: 10048 ``` **Processing Method**: Increase the FE JVM memory configuration, with a recommended minimum of 16GB in production, and adjust later based on the increment of metadata. For recovery, refer to steps 8 and 9 in the [Metadata Recovery](https://docs.starrocks.io/docs/administration/Meta_recovery.md) documentation. *** ##### Case 8: Slow FE response leads to increased time consumption for import tasks[​](#case-8-slow-fe-response-leads-to-increased-time-consumption-for-import-tasks "Direct link to Case 8: Slow FE response leads to increased time consumption for import tasks") **Problem Phenomenon**: The agent prints `jstack` when communicating with FE and encountering a situation where FE does not respond in a timely manner. The logs of the two follower FEs contain the printed message "notify new FE type". There are a large number of `jstack` collection records in the agent log. `jstack` collection exacerbated the unresponsive state duration of FE. **Processing Method**: Turn off `jstack` collection. Version 3.3+ has already turned it off by default. You can remove the `jstack` command in the cluster configuration. *** ##### Case 9: FE Memory Slowly Increasing (version 3.3.13 - 3.3.18)[​](#case-9-fe-memory-slowly-increasing-version-3313---3318 "Direct link to Case 9: FE Memory Slowly Increasing (version 3.3.13 - 3.3.18)") **Version scope**: This issue occurs between versions 3.3.13 and 3.3.18. **Problem Phenomenon**: If the FE heap memory is found to continuously and slowly increase, it may be caused by this issue. Obtaining `jmap` and checking reveals that `com.starrocks.catalog.Replica` occupies excessive memory. All operations involving large amounts of `DROP`, `SWAP`, and `INSERT OVERWRITE` may trigger this issue. This issue is due to the optimization of the tablet deletion path in version 3.3.13, which introduced a memory leak and was fixed in 3.3.18. * Optimization PR in 3.3.13: \[BugFix] Fix recycle bin missing to delete lake mv's expired partitions after mv refreshed * Fix PR in [3.3.18](https://github.com/StarRocks/starrocks/pull/61582) **Processing Method**: * Temporary workaround: Restart FE * Fundamental solution: Upgrade to a fixed version **Fixed versions**: 3.3.18, 3.4.7, 3.5.4 *** #### Appendix: Tool Usage Commands[​](#appendix-tool-usage-commands "Direct link to Appendix: Tool Usage Commands") ##### jstat — Monitor JVM GC Status[​](#jstat--monitor-jvm-gc-status "Direct link to jstat — Monitor JVM GC Status") ```bash jstat -gcutil 1000 10 ``` Prints the GC utilization of the FE process once every 1 second, continuously printing 10 times. **Common indicators:** | Indicator | Meaning | | --------- | --------------------------------------------- | | S0, S1 | Survivor space utilization (Young Generation) | | E | Eden space utilization (Young Generation) | | O | Old Generation utilization | | YGC, YGCT | Young GC count and time | | FGC, FGCT | Full GC count and time | **Purpose**: Quickly determine whether GC is frequent, whether the old generation is approaching full, and whether there is a Full GC. ##### jmap — View JVM Memory Object Distribution[​](#jmap--view-jvm-memory-object-distribution "Direct link to jmap — View JVM Memory Object Distribution") **Method 1: View objects in the current heap (including garbage objects)** ```bash jmap -histo | head -n 30 ``` Counts the number of instances and occupied size of all object types. Use this to check what the large objects in the heap are, such as `byte[]`, `String`, `HashMap`, etc. **Method 2: Check live objects after forcing GC** ```bash jmap -histo:live | head -n 30 ``` Triggers a Full GC, then only displays objects that still survive after the GC. Use this to troubleshoot issues such as leaked objects or cache not being released. --- ### Best Practices These best practices are written by experienced database engineers. Designing for efficiency does more than improve query speed, it decreases costs by reducing storage, CPU, and object storage (e.g., S3) API costs. #### General table design[​](#general-table-design "Direct link to General table design") Three guides covering: * [Partitioning](https://docs.starrocks.io/docs/best_practices/partitioning.md) * [Clustering](https://docs.starrocks.io/docs/best_practices/table_clustering.md) * [Bucketing](https://docs.starrocks.io/docs/best_practices/bucketing.md) Learn about: * The differences between partitioning and bucketing * When to partition * How to choose an efficient sort key * Choosing between hash and random bucketing #### Primary key tables[​](#primary-key-tables "Direct link to Primary key tables") The [Primary Key](https://docs.starrocks.io/docs/best_practices/primarykey_table.md) table uses a new storage engine designed by StarRocks. Its main advantage lies in supporting real-time data updates while ensuring efficient performance for complex ad-hoc queries. In real-time business analytics, decision-making can benefit from Primary Key tables, which use the newest data to analyze results in real-time, mitigating data latency in data analysis. Learn about: * Choosing the type of primary key index * Choosing the primary key * Monitoring and managing memory use * Tuning #### Query Tuning[​](#query-tuning "Direct link to Query Tuning") [Query tuning](https://docs.starrocks.io/docs/best_practices/query_tuning/query_plan_intro.md) is essential for achieving high performance and reliability in StarRocks. This directory brings together practical guides, reference materials, and actionable recipes to help you analyze, diagnose, and optimize query performance at every stage—from writing SQL to interpreting execution details. Effective query tuning in StarRocks typically follows a top-down process: 1. **Identify the Problem** 2. **Collect and Analyze Execution Information** 3. **Locate the Root Cause** 4. **Apply Tuning Strategies** 5. **Validate and Iterate** Whether you're a DBA, developer, or data engineer, these resources will help you: * Diagnose and resolve slow or resource-intensive queries * Understand optimizer choices and execution details * Apply best practices and advanced tuning strategies Start with the [overview](https://docs.starrocks.io/docs/best_practices/query_tuning/query_plan_intro.md), dive into the references as needed, and use the recipes and tips to solve real-world performance challenges in StarRocks. --- ### Partitioning Fast analytics in StarRocks begin with a table layout that matches your query patterns. This guide distills hands‑on experience into clear rules for **partitioning**, helping you: * **Scan less data** via aggressive partition pruning * **Manage lifecycle tasks** (TTL, GDPR deletes, tiering) with metadata‑only ops; * **Scale smoothly** as tenant counts, data volume, or retention windows grow. * **Controlled write amplification**–New data lands in the “hot” partition; compaction happens in historical partitions Keep this advice close when modeling a new table or refactoring an old one—each section gives purpose‑driven criteria, design heuristics, and operational guard‑rails so you avoid costly re‑partitioning down the road. #### Partitioning vs. Bucketing–different jobs[​](#partitioning-vs-bucketingdifferent-jobs "Direct link to Partitioning vs. Bucketing–different jobs") Understanding the distinction between partitioning and bucketing is fundamental when designing performant StarRocks tables. While both help manage large datasets, they serve different purposes: * **Partitioning** allows StarRocks to skip entire blocks of data at query time using partition pruning, and enables metadata-only lifecycle operations like dropping old or tenant-specific data. * **Bucketing**, on the other hand, helps distribute data across tablets to parallelize query execution and balance load, especially when combined with hash functions. | Aspect | Partitioning | Bucketing (Hash/Random) | | ---------------------- | ------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | | **Primary goal** | Coarse‑grain data pruning and lifecycle control(TTL, archiving). | Fine‑grain parallelism and data locality inside each partition. | | **Planner visibility** | Partitions are catalog objects; FE can skip them via predicates. | Only equality predicates support bucket pruning | | **Lifecycle ops** | DROP PARTITION is metadata‑only—ideal for GDPR deletes, monthly roll‑off. | Buckets can’t be dropped; they change only with `ALTER TABLE … MODIFY DISTRIBUTED BY`. | | **Typical count** | 10^2–10^4 per table (days, weeks, tenants). | 10–120 per partition; StarRocks `BUCKETS xxx` tunes this. | | **Skew handling** | Merge or split partitions; consider composite/hybrid scheme. | Raise bucket count, hash on compound key, isolate “whales”, or use random bucketing | | **Red flags** | >100 k partitions can introduce significant memory footprint for FE | >200 k tablets per BE; tablets exceeding 10 GB may encounter compaction issues. | #### When should I partition?[​](#when-should-i-partition "Direct link to When should I partition?") | Table type | Partition? | Typical key | | ------------------------------ | ---------- | -------------------------------- | | Fact / event stream | Yes | `date_trunc('day', event_time)` | | Huge dimension (billions rows) | Sometimes | Time or business key change date | | Small dimension / lookup | No | Rely on hash distribution | #### Choosing the partition key[​](#choosing-the-partition-key "Direct link to Choosing the partition key") 1. **Time‑first default**–If 80 % of queries include a time filter, lead with `date_trunc('day', dt)`. 2. **Tenant isolation**–Add `tenant_id` into the key when you need to manage the data in tenant basis 3. **Retention alignment**–Put the column you plan to purge on into the key. 4. **Composite keys**: `PARTITION BY tenant_id, date_trunc('day', dt)` prunes perfectly but creates `#tenants × #days` partitions. Keep below ≈ 100 k total or FE memory & BE compaction suffer. #### Picking granularity[​](#picking-granularity "Direct link to Picking granularity") The granularity of `PARTITION BY date_trunc('day', dt)` should be adjusted based on the use case. You can use "hour," "day," or "month," etc. See [`date_trunc`](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/date_trunc.md) | Granularity | Use when | Pros | Cons | | ---------------- | -------------------------------- | ------------------------------------- | ----------------------------------- | | Daily (default) | Most BI & reporting | Few partitions (365/yr); simple TTL | Less precise for "last 3 h" queries | | Hourly | > 2 × tablet per day; IoT bursts | Hot‑spot isolation; 24 partitions/day | 8 700 partitions/yr | | Weekly / Monthly | Historical archive | Tiny metadata; merges easy | Coarser pruning | * **Rule of thumb**: Keep each partition ≤ 100 GB and ≤ 20 k tablets/partition across replicas. * **Mixed granularity**: Starting from version 3.4, StarRocks supports mixed granularity by merging historical partitions into coarser granularity. #### Example recipes[​](#example-recipes "Direct link to Example recipes") ##### Click‑stream fact (single‑tenant)[​](#clickstream-fact-singletenant "Direct link to Click‑stream fact (single‑tenant)") ```sql CREATE TABLE click_stream ( user_id BIGINT, event_time DATETIME, url STRING, ... ) DUPLICATE KEY(user_id, event_time) PARTITION BY date_trunc('day', event_time) DISTRIBUTED BY HASH(user_id) BUCKETS xxx; ``` ##### SaaS metrics (multi‑tenant, pattern A)[​](#saas-metrics-multitenant-pattern-a "Direct link to SaaS metrics (multi‑tenant, pattern A)") Recommended for most SaaS workloads. Prunes on time, keeps tenant rows co‑located. ```sql CREATE TABLE metrics ( tenant_id INT, dt DATETIME, metric_name STRING, v DOUBLE ) PRIMARY KEY(tenant_id, dt, metric_name) PARTITION BY date_trunc('DAY', dt) DISTRIBUTED BY HASH(tenant_id) BUCKETS xxx; ``` ##### Whale tenant composite (pattern B)[​](#whale-tenant-composite-pattern-b "Direct link to Whale tenant composite (pattern B)") When tenant-specific DML/DDL is necessary or large-scale tenants are present, be cautious of potential partition explosion. ```sql CREATE TABLE activity ( tenant_id INT, dt DATETIME, id BIGINT, .... ) DUPLICATE KEY(dt, id) PARTITION BY tenant_id, date_trunc('MONTH', dt) DISTRIBUTED BY HASH(id) BUCKETS xxx; ``` --- ### Primary Key tables The Primary Key table uses a new storage engine designed by StarRocks. Its main advantage lies in supporting real-time data updates while ensuring efficient performance for complex ad-hoc queries. In real-time business analytics, decision-making can benefit from Primary Key tables, which use the newest data to analyze results in real-time, which can mitigate data latency in data analysis. However, the primary key is not a free lunch. If used improperly, it can lead to unnecessary resource waste. Therefore, in this section, we will guide you on how to use the primary key model more efficiently to achieve the desired results. #### Choosing Primary Key Index[​](#choosing-primary-key-index "Direct link to Choosing Primary Key Index") The primary index is the most critical component in a Primary Key table. The primary key index which is used to store the mapping between the primary key values and the locations of the data rows identified by the primary key values. Currently, we support three types of primary key index: 1. Full in-memory primary key index. ```sql PROPERTIES ( "enable_persistent_index" = "false" ); ``` 2. Local disk based persistent primary key index. ```sql PROPERTIES ( "enable_persistent_index" = "true", "persistent_index_type" = "LOCAL" ); ``` 3. Cloud native persistent primary key index. ```sql PROPERTIES ( "enable_persistent_index" = "true", "persistent_index_type" = "CLOUD_NATIVE" ); ``` We DO NOT recommend using in-memory indexing, as it can lead to significant memory resource waste. If you are using a shared-data (elastic) StarRocks cluster, we recommend opting for the cloud-native persistent primary index. Unlike the local disk based persistent primary index, it stores the complete index data on remote object storage, with local disks serving only as a cache. Compared to the local disk-based persistent primary index, its advantages include: 1. No dependency on local disk capacity. 2. No need to rebuild indexes after data shard rebalance. #### Choosing the Primary key[​](#choosing-the-primary-key "Direct link to Choosing the Primary key") The primary key usually does not help accelerate queries. You can specify a column different from the primary key as the sort key using the `ORDER BY` clause to speed up queries. Therefore, when selecting a primary key, you only need to consider the uniqueness during data import and update processes. The larger the primary key, the more memory, I/O, and other resources it consumes. Therefore, it is generally recommended to avoid selecting too many or overly large columns as the primary key. The default maximum size for the primary key is 128 bytes, controlled by the `primary_key_limit_size` parameter in `be.conf`. You can increase `primary_key_limit_size` to select a larger primary key, but be aware that this will result in higher resource consumption. How much storage and memory space will a persistent index occupy? ##### Formula for storage space cost[​](#formula-for-storage-space-cost "Direct link to Formula for storage space cost") `(key size + 8 bytes) * row count * 50%` note 50% is the estimated compression efficiency, the actual compression effect depends on the data itself. ##### Formula for memory cost[​](#formula-for-memory-cost "Direct link to Formula for memory cost") `min(l0_max_mem_usage * tablet cnt, update_memory_limit_percent * BE process memory);` #### Memory usage[​](#memory-usage "Direct link to Memory usage") The memory used by Primary Key table can be monitored by mem\_tracker: ```text //View the overall memory statistics http://be_ip:be_http_port/mem_tracker // View primary key table memory statistics http://be_ip:be_http_port/mem_tracker?type=update // View primary key table memory statistics with more details http://be_ip:be_http_port/mem_tracker?type=update&upper_level=4 ``` `update` item in `mem_tracker` record whole memory used by Primary Key table, such as primary key index, delete vector and so on. You can also monitor this `update` item via metrics monitor service. For example, in Grafana, you can check update item via (item in the red box): ![grafana](/assets/images/primary_key-1-09d6340d0330093eb1406fbb8c887fde.png) > More about Monitor and Alert with Prometheus and Grafana: If you are sensitive to memory usage and want to reduce memory consumption during the import process of a PK table, you can achieve this through the following configuration: ```text be.conf l0_max_mem_usage = (some value which smaller than 104857600, default is 104857600) // Shared-nothing cluster transaction_apply_worker_count = (some value smaller than cpu core number, default is cpu core number) // Shared-data cluster transaction_publish_version_worker_count = (some value smaller than cpu core number, default is cpu core number) ``` `l0_max_mem_usage` controls the max memory usage of persistent primary key index per tablet. `transaction_apply_worker_count` and `transaction_publish_version_worker_count` both control the max thread number which can be used for handling upsert and delete in primary key table. But you need to remember, reducing `l0_max_mem_usage` may increase I/O pressure, while decreasing `transaction_apply_worker_count` or `transaction_publish_version_worker_count` could slow down data ingestion. #### Tradeoff between compaction resource, data freshness and query latency[​](#tradeoff-between-compaction-resource-data-freshness-and-query-latency "Direct link to Tradeoff between compaction resource, data freshness and query latency") Compared to tables in other models, primary key tables require additional operations for primary key index lookups and delete vector generation during data import, updates, and deletions, which introduces extra resource overhead. Therefore, you need to make trade-offs among these three factors: * Compaction resources limitation. * Data freshness * Query latency ###### Data freshness & Query latency[​](#data-freshness--query-latency "Direct link to Data freshness & Query latency") If you want to get better data freshness and also better query latency, that means you will introduce high frequency writes, and also want to make sure they can be compacted as soon as possible. Then you will need more compaction resource to handle these writes: ```text // shared-data be.conf compact_threads = 4 // shared-nothing be.conf update_compaction_num_threads_per_disk = 1 update_compaction_per_tablet_min_interval_seconds = 120 ``` You can increase `compact_threads` and `update_compaction_num_threads_per_disk`, or decease `update_compaction_per_tablet_min_interval_seconds` to introduce more compaction resource to handle high frequency writes. How do you know whether current compaction resource and setting can handle current high frequency writes? You can observe it in the following ways: 1. For shared-data cluster, if compaction cannot keep up with the ingestion rate, it may lead to ingestion slowdown or even write failure errors and ingestion stop. a. Ingestion slowdown. You can use `show proc /transactions/{db_name}/running';` to check current running transactions, and if there is any slowdown message like : ```text Partition's compaction score is larger than 100.0, delay commit for xxxms. You can try to increase compaction concurrency ``` show up in ErrMsg field, that means ingestion slowdown happens. E.g. ```sql mysql> show proc '/transactions/test_pri_load_c/running'; +---------------+----------------------------------------------+------------------+-------------------+--------------------+---------------------+------------+-------------+------------+----------------------------------------------------------------------------------------------------------------------------+--------------------+------------+-----------+--------+ | TransactionId | Label | Coordinator | TransactionStatus | LoadJobSourceType | PrepareTime | CommitTime | PublishTime | FinishTime | Reason | ErrorReplicasCount | ListenerId | TimeoutMs | ErrMsg | +---------------+----------------------------------------------+------------------+-------------------+--------------------+---------------------+------------+-------------+------------+----------------------------------------------------------------------------------------------------------------------------+--------------------+------------+-----------+--------+ | 1034 | stream_load_d2753fbaa0b343acadd5f13de92d44c1 | FE: 172.26.94.39 | PREPARE | FRONTEND_STREAMING | 2024-10-24 13:05:01 | NULL | NULL | NULL | Partition's compaction score is larger than 100.0, delay commit for 6513ms. You can try to increase compaction concurrency, | 0 | 11054 | 86400000 | | +---------------+----------------------------------------------+------------------+-------------------+--------------------+---------------------+------------+-------------+------------+----------------------------------------------------------------------------------------------------------------------------+--------------------+------------+-----------+--------+ ``` b. Ingestion stoppage. If there is an ingestion error like : ```text Failed to load data into partition xxx, because of too large compaction score, current/limit: xxx/xxx. You can reduce the loading job concurrency, or increase compaction concurrency ``` That means Ingestion stop because of compaction can't catch up current high frequency writes. 2. For shared-nothing cluster, there is no ingestion slowdown strategy, if the compaction can't catch up current high frequency writes. Ingestion will fail and return error message: ```text Failed to load data into tablet xxx, because of too many versions, current/limit: xxx/xxx. You can reduce the loading job concurrency, or increase loading data batch size. If you are loading data with Routine Load, you can increase FE configs routine_load_task_consume_second and max_routine_load_batch_size. ``` ###### Data freshness & Compaction resources limitation[​](#data-freshness--compaction-resources-limitation "Direct link to Data freshness & Compaction resources limitation") If you have limited compaction resources but still need to maintain sufficient data freshness, this means you'll need to sacrifice some query latency. You can make these config changes to achieve that: * Shared-data cluster ```text fe.conf lake_ingest_slowdown_threshold = xxx (default is 100, you can increase it) lake_compaction_score_upper_bound = xxx (default is 2000, you can increase it) ``` The `lake_ingest_slowdown_threshold` parameter controls the threshold for triggering ingestion slowdown. When a partition's compaction score exceeds this threshold, the system will begin to slowdown data ingestion. Similarly, `lake_compaction_score_upper_bound` determines the threshold for triggering ingestion stoppage. * Shared-nothing cluster ```text be.conf tablet_max_versions = xxx (default is 1000, you can increase it) ``` `tablet_max_versions` determines the threshold for triggering ingestion stoppage. By increasing these configurations, the system can accommodate more small data files and reduce compaction frequency, but this will also impact query latency. ###### Query latency & Compaction resources limitation[​](#query-latency--compaction-resources-limitation "Direct link to Query latency & Compaction resources limitation") If you want to achieve good query latency with limited compaction resources, you need to reduce write frequency and create larger data batches for ingestion. For specific implementation, please refer to the sections on different ingestion methods, which detail how to reduce ingestion frequency and increase batch size. --- ### Query Hint Query hints are directives or comments that explicitly suggest the query optimizer on how to execute a query. Currently, StarRocks supports three types of hints: system variable hint (`SET_VAR`), user-defined variable hint (`SET_USER_VARIABLE`), and Join hint. Hints only take effect within a single query. #### System variable hint[​](#system-variable-hint "Direct link to System variable hint") You can use a `SET_VAR` hint to set one or more [system variables](https://docs.starrocks.io/docs/sql-reference/System_variable.md) in SELECT and SUBMIT TASK statements, and then execute the statements. You can also use a `SET_VAR` hint in the SELECT clause included in other statements, such as CREATE MATERIALIZED VIEW AS SELECT and CREATE VIEW AS SELECT. Note that if the `SET_VAR` hint is used in the SELECT clause of CTE, the `SET_VAR` hint does not take effect even if the statement is executed successfully. Compared with [the general usage of system variables](https://docs.starrocks.io/docs/sql-reference/System_variable.md) which takes effect at the session level, the `SET_VAR` hint takes effect at the statement level and does not impact the entire session. ##### Syntax[​](#syntax "Direct link to Syntax") ```sql [...] SELECT /*+ SET_VAR(key=value [, key = value]) */ ... SUBMIT [/*+ SET_VAR(key=value [, key = value]) */] TASK ... ``` ##### Examples[​](#examples "Direct link to Examples") To specify the aggregation mode for an aggregate query, use the `SET_VAR` hint to set the system variables `streaming_preaggregation_mode` and `new_planner_agg_stage` in the aggregate query. ```sql SELECT /*+ SET_VAR (streaming_preaggregation_mode = 'force_streaming',new_planner_agg_stage = '2') */ SUM(sales_amount) AS total_sales_amount FROM sales_orders; ``` To specify the execution timeout for a SUBMIT TASK statement, use the `SET_VAR` Hint to set the system variable `insert_timeout` in the SUBMIT TASK statement. ```sql SUBMIT /*+ SET_VAR(insert_timeout=3) */ TASK AS CREATE TABLE temp AS SELECT count(*) AS cnt FROM tbl1; ``` To specify the subquery execution timeout for creating a materialized view, use the `SET_VAR` hint to set the system variable `query_timeout` in the SELECT clause. ```sql CREATE MATERIALIZED VIEW mv PARTITION BY dt DISTRIBUTED BY HASH(`key`) BUCKETS 10 REFRESH ASYNC AS SELECT /*+ SET_VAR(query_timeout=500) */ * from dual; ``` Specify system variables in a nested query: ```sql -- To specify hints in the main query WITH t AS (SELECT region, sales_amount FROM sales_orders) SELECT /*+ SET_VAR (streaming_preaggregation_mode = 'force_streaming', new_planner_agg_stage = '2') */ SUM(sales_amount) AS total_sales_amount FROM t; ``` #### User-defined variable hint[​](#user-defined-variable-hint "Direct link to User-defined variable hint") You can use a `SET_USER_VARIABLE` hint to set one or more [user-defined variables](https://docs.starrocks.io/docs/sql-reference/user_defined_variables.md) in the SELECT statements or INSERT statements. If other statements contain a SELECT clause, you can also use the `SET_USER_VARIABLE` hint in that SELECT clause. Other statements can be SELECT statements and INSERT statements, but cannot be CREATE MATERIALIZED VIEW AS SELECT statements and CREATE VIEW AS SELECT statements. Note that if the `SET_USER_VARIABLE` hint is used in the SELECT clause of CTE, the `SET_USER_VARIABLE` hint does not take effect even if the statement is executed successfully. Since v3.2.4, StarRocks supports the user-defined variable hint. Compared with [the general usage of user-defined variables](https://docs.starrocks.io/docs/sql-reference/user_defined_variables.md) which takes effect at the session level, the `SET_USER_VARIABLE` hint takes effect at the statement level and does not impact the entire session. ##### Syntax[​](#syntax-1 "Direct link to Syntax") ```sql [...] SELECT /*+ SET_USER_VARIABLE(@var_name = expr [, @var_name = expr]) */ ... INSERT /*+ SET_USER_VARIABLE(@var_name = expr [, @var_name = expr]) */ ... ``` ##### Examples[​](#examples-1 "Direct link to Examples") The following SELECT statement references scalar subqueries `select max(age) from users` and `select min(name) from users`, so you can use a `SET_USER_VARIABLE` hint to set these two scalar subqueries as user-defined variables and then run the query. ```sql SELECT /*+ SET_USER_VARIABLE (@a = (select max(age) from users), @b = (select min(name) from users)) */ * FROM sales_orders where sales_orders.age = @a and sales_orders.name = @b; ``` #### Join hint[​](#join-hint "Direct link to Join hint") For multi-table Join queries, the optimizer usually selects the optimal Join execution method. In special cases, you can use a Join hint to explicitly suggest the Join execution method to the optimizer or disable Join Reorder. Currently, a Join hint supports suggesting Shuffle Join, Broadcast Join, Bucket Shuffle Join, or Colocate Join as a Join execution method. When a Join hint is used, the optimizer does not perform Join Reorder. So you need to select the smaller table as the right table. Additionally, when suggesting [Colocate Join](https://docs.starrocks.io/docs/using_starrocks/Colocate_join.md) or Bucket Shuffle Join as the Join execution method, make sure that the data distribution of the joined table meets the requirements of these Join execution methods. Otherwise, the suggested Join execution method cannot take effect. ##### Syntax[​](#syntax-2 "Direct link to Syntax") ```sql ... JOIN { [BROADCAST] | [SHUFFLE] | [BUCKET] | [COLOCATE] | [UNREORDER]} ... ``` note Join Hint is case-insensitive. ##### Examples[​](#examples-2 "Direct link to Examples") * Shuffle Join If you need to shuffle the data rows with the same bucketing key values from tables A and B onto the same machine before a Join operation is performed, you can hint the Join execution method as Shuffle Join. ```sql select k1 from t1 join [SHUFFLE] t2 on t1.k1 = t2.k2 group by t2.k2; ``` * Broadcast Join If table A is a large table and table B is a small table, you can hint the Join execution method as Broadcast Join. The data of the table B is fully broadcasted to the machines on which the data of table A resides, and then the Join operation is performed. Compared to Shuffle Join, Broadcast Join saves the cost of shuffling the data of table A. ```sql select k1 from t1 join [BROADCAST] t2 on t1.k1 = t2.k2 group by t2.k2; ``` * Bucket Shuffle Join If the Join equijoin expression in the Join query contains the bucketing key of table A, especially when both tables A and B are large tables, you can hint the Join execution method as Bucket Shuffle Join. The data of table B is shuffled to the machines on which the data of table A resides, according to the data distribution of table A, and then the Join operation is performed. Compared to Broadcast Join, Bucket Shuffle Join significantly reduces data transferring because the data of table B is shuffled only once globally. Tables participating in Bucket Shuffle Join must be either non-partitioned or colocated. ```sql select k1 from t1 join [BUCKET] t2 on t1.k1 = t2.k2 group by t2.k2; ``` * Colocate Join If tables A and B belong to the same Colocation Group which is specified during table creation, the data rows with the same bucketing key values from tables A and B are distributed on the same BE node. When the Join equijoin expression contains the bucketing key of tables A and B in the Join query, you can hint the Join execution method as Colocate Join. Data with the same key values are directly joined locally, reducing the time spent on data transmission between nodes and improving query performance. ```sql select k1 from t1 join [COLOCATE] t2 on t1.k1 = t2.k2 group by t2.k2; ``` ##### View Join execution method[​](#view-join-execution-method "Direct link to View Join execution method") Use the `EXPLAIN` command to view the actual Join execution method. If the returned result shows that the Join execution method matches the Join hint, it means that the Join hint is effective. ```sql EXPLAIN select k1 from t1 join [COLOCATE] t2 on t1.k1 = t2.k2 group by t2.k2; ``` ![8-9](/assets/images/8-9-9999cb1cdef4159128d20893137e56e9.png) --- ### Introduction to Query Tuning Query tuning is essential for achieving high performance and reliability in StarRocks. This directory brings together practical guides, reference materials, and actionable recipes to help you analyze, diagnose, and optimize query performance at every stage—from writing SQL to interpreting execution details. Effective query tuning in StarRocks typically follows a top-down process: 1. **Identify the Problem** * Detect slow queries, high resource usage, or unexpected results. * In StarRocks, leverage built-in monitoring tools, query history, and audit logs to quickly identify problematic queries or unusual patterns. * See: **[Query Tuning Recipes](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_tuning_recipes.md)** for symptom-driven diagnosis and **[Query Profile Overview](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_overview.md)** for accessing query history and profiles. 2. **Collect and Analyze Execution Information** * Obtain the query plan using `EXPLAIN` or `EXPLAIN ANALYZE`. * Enable and review the Query Profile to gather detailed execution metrics. * See: **[Query Plan Overview](https://docs.starrocks.io/docs/best_practices/query_tuning/query_planning.md)** for understanding query plans, **[Explain Analyze & Text-Based Profile Analysis](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_text_based_analysis.md)** for step-by-step analysis, and **[Query Profile Overview](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_overview.md)** for enabling and interpreting profiles. 3. **Locate the Root Cause** * Pinpoint which stage or operator is consuming the most time or resources. * Check for common issues: suboptimal join order, missing indexes, data distribution problems, or inefficient SQL patterns. * See: **[Query Profile Metrics](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_operator_metrics.md)** for a glossary of metrics and operators, and **[Query Tuning Recipes](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_tuning_recipes.md)** for root cause analysis. 4. **Apply Tuning Strategies** * SQL Rewrite: rewrite or optimize the SQL query (e.g., add filters, avoid SELECT \*). * Schema tuning: add indexes, change table types, partitioning, clustering. * Query plan tuning: use hints or variables to guide the optimizer if necessary. * Execution tuning: tune session variables for specific workloads. * See: **[Schema Tuning Recipes](https://docs.starrocks.io/docs/best_practices/query_tuning/schema_tuning.md)** for schema-level optimizations, **[Query Hint](https://docs.starrocks.io/docs/best_practices/query_tuning/query_hint.md)** for optimizer hints, and **[Query Tuning Recipes](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_tuning_recipes.md)** for plan tuning and execution tuning. 5. **Validate and Iterate** * Rerun the query and compare performance before and after changes. * Review the new query plan and profile to ensure improvements. * Repeat the process as needed for further optimization. Whether you're a DBA, developer, or data engineer, these resources will help you: * Diagnose and resolve slow or resource-intensive queries * Understand optimizer choices and execution details * Apply best practices and advanced tuning strategies Start with the overview, dive into the references as needed, and use the recipes and tips to solve real-world performance challenges in StarRocks. --- ### Query plan Optimizing query performance is a common challenge in analytics systems. Slow queries can impair user experience and overall cluster performance. In StarRocks, understanding and interpreting query plans and query profiles is the foundation for diagnosing and improving slow queries. These tools help you: * Identify bottlenecks and expensive operations * Spot suboptimal join strategies or missing indexes * Understand how data is filtered, aggregated, and moved * Troubleshoot and optimize resource usage A **query plan** is a detailed roadmap generated by the StarRocks FE that describes how your SQL statement will be executed. It breaks down the query into a series of operations—such as scans, joins, aggregations, and sorts—and determines the most efficient way to perform them. StarRocks provides several ways to inspect the query plan: 1. **EXPLAIN statement**:
Use `EXPLAIN` to display the logical or physical execution plan for a query. You can add options to control the output: * `EXPLAIN LOGICAL `: Shows the simplified plan. * `EXPLAIN `: Shows the basic physical plan * `EXPLAIN VERBOSE `: Shows the physical plan with detailed information. * `EXPLAIN COSTS `: Includes estimated costs for each operation, which is used to diagnose the statistics issue 2. **EXPLAIN ANALYZE**:
Use `EXPLAIN ANALYZE ` to execute the query and display the actual execution plan along with real runtime statistics. See the [Explain Analyze](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_text_based_analysis.md) documentation for details. Example: ```sql EXPLAIN ANALYZE SELECT * FROM sales_orders WHERE amount > 1000; ``` 3. **Query Profile**:
After running a query, you can view its detailed execution profile, which includes timing, resource usage, and operator-level statistics. See the [Query Profile](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_overview.md) documentation for how to access and interpret this information. * **SQL commands**: `SHOW PROFILELIST` and `ANALYZE PROFILE FOR `: can be used to retrieve the execution profile for a specific query. * **FE HTTP Service**: Access query profiles via the StarRocks FE web UI by navigating to the **Query** or **Profile** section, where you can search for and inspect query execution details. * **Managed Version**: In cloud or managed deployments, use the provided web console or monitoring dashboard to view query plans and profiles, often with enhanced visualization and filtering options. Typically, the query plan is used to diagnose issues related to how a query is planned and optimized, while the query profile helps identify performance problems during query execution. In the following sections, we'll explore the key concepts of query execution and walk through a concrete example of analyzing a query plan. #### Query execution flow[​](#query-execution-flow "Direct link to Query execution flow") The lifecycle of a query in StarRocks consists of three main phases: 1. **Planning**: The query undergoes parsing, analysis, and optimization, culminating in the generation of a query plan. 2. **Scheduling**: The scheduler and coordinator distribute the plan to all participating backend nodes. 3. **Execution**: The plan is executed using the pipeline execution engine. ![SQL Execution Flow](/assets/images/execution_flow-f4e4a147a33bea8ecfc538573e8f758e.png) **Plan structure** The StarRocks plan is hierarchical: * **Fragment**: Top-level slice of work; each fragment spawns multiple **FragmentInstances** that run on different backend nodes. * **Pipeline**: Within an instance, a pipeline strings operators together; several **PipelineDrivers** run the same pipeline concurrently on separate CPU cores. * **Operator**: The atomic step—scan, join, aggregate—that actually processes data. ![profile-3](/assets/images/profile-3-079a0ab5168d571ec15182752d97142d.png) **Pipeline execution engine** The Pipeline Engine executes the query plan in a parallel and efficient manner, handling complex plans and large data volumes for high performance and scalability. ![pipeline\_opeartors](/assets/images/pipeline_operators-930f5c2a794b4961d3e2a9f4daf9cf25.png) **Metric merging strategy** By default, StarRocks merges the FragmentInstance and PipelineDriver layers to reduce profile volume, resulting in a simplified three-layer structure: * Fragment * Pipeline * Operator You can control this merging behavior through the session variable `pipeline_profile_level`. #### Example[​](#example "Direct link to Example") ##### How to read a query plan and profile[​](#how-to-read-a-query-plan-and-profile "Direct link to How to read a query plan and profile") 1. **Understand the structure**: Query plans are split into fragments, each representing a stage of execution. Read from the bottom up: scan nodes first, then joins, aggregations, and finally the result. 2. **Overall analysis**: * Check total runtime, memory usage, and CPU/wall time ratio. * Find slow operators by sorting by operator time. * Ensure filters are pushed down where possible. * Look for data skew (uneven operator times or row counts). * Monitor for high memory or disk spill; adjust join order or use rollup views if needed. * Use materialized views and query hints (`BROADCAST`, `SHUFFLE`, `COLOCATE`) to optimize as needed. 3. **Scan operations**: Look for `OlapScanNode` or similar. Note which tables are scanned, what filters are applied, and if pre-aggregation or materialized views are used. 4. **Join operations**: Identify join types (`HASH JOIN`, `BROADCAST`, `SHUFFLE`, `COLOCATE`, `BUCKET SHUFFLE`). The join method affects performance: * **Broadcast**: Small table sent to all nodes; good for small tables. * **Shuffle**: Rows are partitioned and shuffled; good for large tables. * **Colocate**: Tables partitioned the same way; enables local joins. * **Bucket Shuffle**: Only one table is shuffled to reduce network cost. 5. **Aggregation and sorting**: Look for `AGGREGATE`, `TOP-N`, or `ORDER BY`. These can be expensive with large or high-cardinality data. 6. **Data movement**: `EXCHANGE` nodes show data transfer between fragments or nodes. Too much data movement can hurt performance. 7. **Predicate pushdown**: Filters applied early (at scan) reduce downstream data. Check `PREDICATES` or `PushdownPredicates` to see which filters are pushed down. ##### Example query plan[​](#example-query-plan "Direct link to Example query plan") tip This is query 96 from the TPC-DS benchmark. ```sql explain logical select count(*) from store_sales ,household_demographics ,time_dim , store where ss_sold_time_sk = time_dim.t_time_sk and ss_hdemo_sk = household_demographics.hd_demo_sk and ss_store_sk = s_store_sk and time_dim.t_hour = 8 and time_dim.t_minute >= 30 and household_demographics.hd_dep_count = 5 and store.s_store_name = 'ese' order by count(*) limit 100; ``` The output is a hierarchical plan showing how StarRocks will execute the query. The plan is structured as a tree of operators, read from bottom to top. The logical plan shows the sequence of operations with cost estimates: ```text - Output => [69:count] - TOP-100(FINAL)[69: count ASC NULLS FIRST] Estimates: {row: 1, cpu: 8.00, memory: 8.00, network: 8.00, cost: 68669801.20} - TOP-100(PARTIAL)[69: count ASC NULLS FIRST] Estimates: {row: 1, cpu: 8.00, memory: 8.00, network: 8.00, cost: 68669769.20} - AGGREGATE(GLOBAL) [] Estimates: {row: 1, cpu: 8.00, memory: 8.00, network: 0.00, cost: 68669737.20} 69:count := count(69:count) - EXCHANGE(GATHER) Estimates: {row: 1, cpu: 8.00, memory: 0.00, network: 8.00, cost: 68669717.20} - AGGREGATE(LOCAL) [] Estimates: {row: 1, cpu: 3141.35, memory: 0.80, network: 0.00, cost: 68669701.20} 69:count := count() - HASH/INNER JOIN [9:ss_store_sk = 40:s_store_sk] => [71:auto_fill_col] Estimates: {row: 3490, cpu: 111184.52, memory: 8.80, network: 0.00, cost: 68668128.93} 71:auto_fill_col := 1 - HASH/INNER JOIN [7:ss_hdemo_sk = 25:hd_demo_sk] => [9:ss_store_sk] Estimates: {row: 19940, cpu: 1841177.20, memory: 2880.00, network: 0.00, cost: 68612474.92} - HASH/INNER JOIN [4:ss_sold_time_sk = 30:t_time_sk] => [7:ss_hdemo_sk, 9:ss_store_sk] Estimates: {row: 199876, cpu: 69221191.15, memory: 7077.97, network: 0.00, cost: 67671726.32} - SCAN [store_sales] => [4:ss_sold_time_sk, 7:ss_hdemo_sk, 9:ss_store_sk] Estimates: {row: 5501341, cpu: 66016092.00, memory: 0.00, network: 0.00, cost: 33008046.00} partitionRatio: 1/1, tabletRatio: 192/192 predicate: 7:ss_hdemo_sk IS NOT NULL - EXCHANGE(BROADCAST) Estimates: {row: 1769, cpu: 7077.97, memory: 7077.97, network: 7077.97, cost: 38928.81} - SCAN [time_dim] => [30:t_time_sk] Estimates: {row: 1769, cpu: 21233.90, memory: 0.00, network: 0.00, cost: 10616.95} partitionRatio: 1/1, tabletRatio: 5/5 predicate: 33:t_hour = 8 AND 34:t_minute >= 30 - EXCHANGE(BROADCAST) Estimates: {row: 720, cpu: 2880.00, memory: 2880.00, network: 2880.00, cost: 14400.00} - SCAN [household_demographics] => [25:hd_demo_sk] Estimates: {row: 720, cpu: 5760.00, memory: 0.00, network: 0.00, cost: 2880.00} partitionRatio: 1/1, tabletRatio: 1/1 predicate: 28:hd_dep_count = 5 - EXCHANGE(BROADCAST) Estimates: {row: 2, cpu: 8.80, memory: 8.80, network: 8.80, cost: 44.15} - SCAN [store] => [40:s_store_sk] Estimates: {row: 2, cpu: 17.90, memory: 0.00, network: 0.00, cost: 8.95} partitionRatio: 1/1, tabletRatio: 1/1 predicate: 45:s_store_name = 'ese' ``` **Reading the plan bottom-up** The query plan should be read from the bottom (leaf nodes) upward to the top (root node), following the data flow: 1. **Scan Operations (Bottom Level)**: The `SCAN` operators at the bottom read data from the base tables: * `SCAN [store_sales]` reads the main fact table with predicate `ss_hdemo_sk IS NOT NULL` * `SCAN [time_dim]` reads the time dimension table with predicates `t_hour = 8 AND t_minute >= 30` * `SCAN [household_demographics]` reads the demographics table with predicate `hd_dep_count = 5` * `SCAN [store]` reads the store table with predicate `s_store_name = 'ese'` Each scan operation shows: * **Estimates**: Row count, CPU, memory, network, and cost estimates * **Partition and tablet ratios**: How many partitions/tablets are scanned (e.g., `partitionRatio: 1/1, tabletRatio: 192/192`) * **Predicates**: Query conditions that are pushed down to the scan level, reducing the amount of data read 2. **Data Exchange (Broadcast)**: The `EXCHANGE(BROADCAST)` operations distribute smaller dimension tables to all nodes processing the larger fact table. This is efficient when dimension tables are small compared to the fact table, as seen with `time_dim`, `household_demographics`, and `store` being broadcast. 3. **Join Operations (Middle Level)**: Data flows upward through `HASH/INNER JOIN` operations: * First, `store_sales` is joined with `time_dim` on `ss_sold_time_sk = t_time_sk` * Then, the result is joined with `household_demographics` on `ss_hdemo_sk = hd_demo_sk` * Finally, the result is joined with `store` on `ss_store_sk = s_store_sk` Each join shows the join condition and estimates for the resulting row count and resource usage. 4. **Aggregation (Upper Level)**: * `AGGREGATE(LOCAL)` performs local aggregation on each node, computing `count()` * `EXCHANGE(GATHER)` collects results from all nodes * `AGGREGATE(GLOBAL)` merges the local results into the final count 5. **Final Operations (Top Level)**: * `TOP-100(PARTIAL)` and `TOP-100(FINAL)` operations handle the `ORDER BY count(*) LIMIT 100` clause, selecting the top 100 results after ordering The logical plan provides cost estimates for each operation, helping you understand where the query spends most of its resources. The actual physical execution plan (from `EXPLAIN` or `EXPLAIN VERBOSE`) includes additional details about how operations are distributed across nodes and executed in parallel. --- ### Query Profile Metrics > Authoritative reference for raw metrics emitted by **StarRocks Query Profile**, grouped by operator.
Use it as a glossary; for troubleshooting guidance jump to **query\_profile\_tuning\_recipes.md**. ##### Summary Metrics[​](#summary-metrics "Direct link to Summary Metrics") Basic information about the query execution: | Metric | Description | | -------------------------- | --------------------------------------------------------------------------------------------------- | | Total | The total time consumed by the query, including Planning, Executing, and Profiling phase durations. | | Query State | Query state, possible states include Finished, Error, and Running. | | Query ID | Unique identifier for the query. | | Start Time | Timestamp when the query started. | | End Time | Timestamp when the query ended. | | Total | Total duration of the query. | | Query Type | Type of the query. | | Query State | Current state of the query. | | StarRocks Version | Version of StarRocks used. | | User | User who executed the query. | | Default Db | Default database used for the query. | | Sql Statement | SQL statement executed. | | Variables | Important variables used for the query. | | NonDefaultSessionVariables | Non-default session variables used for the query. | | Collect Profile Time | Time taken to collect the profile. | | IsProfileAsync | Indicates if the profile collection was asynchronous. | ##### Planner Metrics[​](#planner-metrics "Direct link to Planner Metrics") It provides a comprehensive overview of the planner. Typically, if the total time spent on the planner is less than 10ms, it is not a cause for concern. In certain scenarios, the planner may require more time: 1. Complex queries may necessitate additional time for parsing and optimization to ensure an optimal execution plan. 2. The presence of numerous materialized views can increase the time required for query rewriting. 3. When multiple concurrent queries exhaust system resources and the query queue is utilized, the `Pending` time may be prolonged. 4. Queries involving external tables may incur additional time for communication with the external metadata server. Example: ```text - -- Parser[1] 0 - -- Total[1] 3ms - -- Analyzer[1] 0 - -- Lock[1] 0 - -- AnalyzeDatabase[1] 0 - -- AnalyzeTemporaryTable[1] 0 - -- AnalyzeTable[1] 0 - -- Transformer[1] 0 - -- Optimizer[1] 1ms - -- MVPreprocess[1] 0 - -- MVTextRewrite[1] 0 - -- RuleBaseOptimize[1] 0 - -- CostBaseOptimize[1] 0 - -- PhysicalRewrite[1] 0 - -- DynamicRewrite[1] 0 - -- PlanValidate[1] 0 - -- InputDependenciesChecker[1] 0 - -- TypeChecker[1] 0 - -- CTEUniqueChecker[1] 0 - -- ColumnReuseChecker[1] 0 - -- ExecPlanBuild[1] 0 - -- Pending[1] 0 - -- Prepare[1] 0 - -- Deploy[1] 2ms - -- DeployLockInternalTime[1] 2ms - -- DeploySerializeConcurrencyTime[2] 0 - -- DeployStageByStageTime[6] 0 - -- DeployWaitTime[6] 1ms - -- DeployAsyncSendTime[2] 0 - DeployDataSize: 10916 Reason: ``` ##### Execution Overview Metrics[​](#execution-overview-metrics "Direct link to Execution Overview Metrics") High-level execution statistics: | Metric | Description | Rule of Thumb | | --------------------------- | ------------------------------------- | ----------------------------------------- | | FrontendProfileMergeTime | FE-side profile processing time | < 10ms normal | | QueryAllocatedMemoryUsage | Total allocated memory across nodes | | | QueryDeallocatedMemoryUsage | Total deallocated memory across nodes | | | QueryPeakMemoryUsagePerNode | Maximum peak memory per node | < 80% capacity normal | | QuerySumMemoryUsage | Total peak memory across nodes | | | QueryExecutionWallTime | Wall time of execution | | | QueryCumulativeCpuTime | Total CPU time across nodes | Compare with `walltime * totalCpuCores` | | QueryCumulativeOperatorTime | Total operator execution time | Denominator for operator time percentages | | QueryCumulativeNetworkTime | Total Exchange node network time | | | QueryCumulativeScanTime | Total Scan node IO time | | | QueryPeakScheduleTime | Maximum Pipeline ScheduleTime | < 1s normal for simple queries | | QuerySpillBytes | Data spilled to disk | < 1GB normal | ##### Fragment Metrics[​](#fragment-metrics "Direct link to Fragment Metrics") Fragment-level execution details: | Metric | Description | | ------------------------------ | -------------------------------------- | | InstanceNum | Number of FragmentInstances | | InstanceIds | IDs of all FragmentInstances | | BackendNum | Number of participating BEs | | BackendAddresses | BE addresses | | FragmentInstancePrepareTime | Fragment Prepare phase duration | | InstanceAllocatedMemoryUsage | Total allocated memory for instances | | InstanceDeallocatedMemoryUsage | Total deallocated memory for instances | | InstancePeakMemoryUsage | Peak memory across instances | ##### Pipeline Metrics[​](#pipeline-metrics "Direct link to Pipeline Metrics") Pipeline execution details and relationships: ![profile\_pipeline\_time\_relationship](/assets/images/profile_pipeline_time_relationship-439d46dd1ab122cafdf39053fedc34cc.jpeg) Key relationships: * DriverTotalTime = ActiveTime + PendingTime + ScheduleTime * ActiveTime = ∑ OperatorTotalTime + OverheadTime * PendingTime = InputEmptyTime + OutputFullTime + PreconditionBlockTime + PendingFinishTime * InputEmptyTime = FirstInputEmptyTime + FollowupInputEmptyTime | Metric | Description | | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | DegreeOfParallelism | Degree of pipeline execution parallelism. | | TotalDegreeOfParallelism | Sum of degrees of parallelism. Since the same Pipeline may execute on multiple machines, this item aggregates all values. | | DriverPrepareTime | Time taken by the Prepare phase. This metric is not included in DriverTotalTime. | | DriverTotalTime | Total execution time of the Pipeline, excluding the time spent in the Prepare phase. | | ActiveTime | Execution time of the Pipeline, including the execution time of each operator and the overall framework overhead, such as time spent in invoking methods like has\_output, need\_input, etc. | | PendingTime | Time the Pipeline is blocked from being scheduled for various reasons. | | InputEmptyTime | Time the Pipeline is blocked due to an empty input queue. | | FirstInputEmptyTime | Time the Pipeline is first blocked due to an empty input queue. The first blocking time is separately calculated because the first blocking is mainly caused by Pipeline dependencies. | | FollowupInputEmptyTime | Time the Pipeline is subsequently blocked due to an empty input queue. | | OutputFullTime | Time the Pipeline is blocked due to a full output queue. | | PreconditionBlockTime | Time the Pipeline is blocked due to unmet dependencies. | | PendingFinishTime | Time the Pipeline is blocked waiting for asynchronous tasks to finish. | | ScheduleTime | Scheduling time of the Pipeline, from entering the ready queue to being scheduled for execution. | | BlockByInputEmpty | Number of times the pipeline is blocked due to InputEmpty. | | BlockByOutputFull | Number of times the pipeline is blocked due to OutputFull. | | BlockByPrecondition | Number of times the pipeline is blocked due to unmet preconditions. | ##### Operator Metrics[​](#operator-metrics "Direct link to Operator Metrics") | Metric | Description | | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | PrepareTime | Time spent on preparation. | | OperatorTotalTime | Total time consumed by the Operator. It satisfies the equation: OperatorTotalTime = PullTotalTime + PushTotalTime + SetFinishingTime + SetFinishedTime + CloseTime. It excludes time spent on preparation. | | PullTotalTime | Total time the Operator spends executing push\_chunk. | | PushTotalTime | Total time the Operator spends executing pull\_chunk. | | SetFinishingTime | Total time the Operator spends executing set\_finishing. | | SetFinishedTime | Total time the Operator spends executing set\_finished. | | PushRowNum | Cumulative number of input rows for the Operator. | | PullRowNum | Cumulative number of output rows for the Operator. | | JoinRuntimeFilterEvaluate | Number of times Join Runtime Filter is evaluated. | | JoinRuntimeFilterHashTime | Time spent computing hash for Join Runtime Filter. | | JoinRuntimeFilterInputRows | Number of input rows for Join Runtime Filter. | | JoinRuntimeFilterOutputRows | Number of output rows for Join Runtime Filter. | | JoinRuntimeFilterTime | Time spent on Join Runtime Filter. | ##### Scan Operator[​](#scan-operator "Direct link to Scan Operator") ###### OLAP Scan Operator[​](#olap-scan-operator "Direct link to OLAP Scan Operator") The OLAP\_SCAN Operator is responsible for reading data from StarRocks native tables. | Metric | Description | | -------------------------- | ------------------------------------------------------------------------------------------- | | Table | Table name. | | Rollup | Materialized view name. If no materialized view is hit, it is equivalent to the table name. | | SharedScan | Whether the enable\_shared\_scan session variable is enabled. | | TabletCount | Number of tablets. | | MorselsCount | Number of morsels, which is the basic IO execution unit. | | PushdownPredicates | Number of pushdown predicates. | | Predicates | Predicate expressions. | | BytesRead | Size of data read. | | CompressedBytesRead | Size of compressed data read from disk. | | UncompressedBytesRead | Size of uncompressed data read from disk. | | RowsRead | Number of rows read (after predicate filtering). | | RawRowsRead | Number of raw rows read (before predicate filtering). | | ReadPagesNum | Number of pages read. | | CachedPagesNum | Number of cached pages. | | ChunkBufferCapacity | Capacity of the Chunk Buffer. | | DefaultChunkBufferCapacity | Default capacity of the Chunk Buffer. | | PeakChunkBufferMemoryUsage | Peak memory usage of the Chunk Buffer. | | PeakChunkBufferSize | Peak size of the Chunk Buffer. | | PrepareChunkSourceTime | Time spent preparing the Chunk Source. | | ScanTime | Cumulative scan time. Scan operations are completed in an asynchronous I/O thread pool. | | IOTaskExecTime | Execution time of IO tasks. | | IOTaskWaitTime | Waiting time from successful submission to scheduled execution of IO tasks. | | SubmitTaskCount | Number of times IO tasks are submitted. | | SubmitTaskTime | Time spent on task submission. | | PeakIOTasks | Peak number of IO tasks. | | PeakScanTaskQueueSize | Peak size of the IO task queue. | ###### Connector Scan Operator[​](#connector-scan-operator "Direct link to Connector Scan Operator") It's similar to OLAP\_SCAN operator but used for scan external tables like Iceberg/Hive/Hudi/Detal. | Metric | Description | | -------------------------- | ---------------------------------------------------------------------------------------------- | | DataSourceType | Data source type, can be HiveDataSource, ESDataSource, and so on. | | Table | Table name. | | TabletCount | Number of tablets. | | MorselsCount | Number of morsels. | | Predicates | Predicate expression. | | PredicatesPartition | Predicate expression applied to partitions. | | SharedScan | Whether the `enable_shared_scan` Session variable is enabled. | | ChunkBufferCapacity | Capacity of the Chunk Buffer. | | DefaultChunkBufferCapacity | Default capacity of the Chunk Buffer. | | PeakChunkBufferMemoryUsage | Peak memory usage of the Chunk Buffer. | | PeakChunkBufferSize | Peak size of the Chunk Buffer. | | PrepareChunkSourceTime | Time taken to prepare the Chunk Source. | | ScanTime | Cumulative time for scanning. Scan operation is completed in the asynchronous I/O thread pool. | | IOTaskExecTime | Execution time of I/O tasks. | | IOTaskWaitTime | Waiting time from successful submission to scheduled execution of IO tasks. | | SubmitTaskCount | Number of times IO tasks are submitted. | | SubmitTaskTime | Time taken to submit tasks. | | PeakIOTasks | Peak number of IO tasks. | | PeakScanTaskQueueSize | Peak size of the IO task queue. | ##### Exchange Operator[​](#exchange-operator "Direct link to Exchange Operator") Exchange Operator is responsible for transmitting data between BE nodes. There can be several kinds of exchange operations: GATHER/BROADCAST/SHUFFLE. Typical scenarios that can make Exchange Operator the bottleneck of a query: 1. Broadcast Join: This is a suitable method for a small table. However, in exceptional cases when the optimizer chooses a suboptimal query plan, it can lead to a significant increase in network bandwidth. 2. Shuffle Aggregation/Join: Shuffling a large table can result in a significant increase in network bandwidth. ###### Exchange Sink Operator[​](#exchange-sink-operator "Direct link to Exchange Sink Operator") | Metric | Description | | -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ChannelNum | Number of channels. Generally, the number of channels is equal to the number of receivers. | | DestFragments | List of destination FragmentInstance IDs. | | DestID | Destination node ID. | | PartType | Data distribution mode, including: UNPARTITIONED, RANDOM, HASH\_PARTITIONED, and BUCKET\_SHUFFLE\_HASH\_PARTITIONED. | | SerializeChunkTime | Time taken to serialize chunks. | | SerializedBytes | Size of serialized data. | | ShuffleChunkAppendCounter | Number of Chunk Append operations when PartType is HASH\_PARTITIONED or BUCKET\_SHUFFLE\_HASH\_PARTITIONED. | | ShuffleChunkAppendTime | Time taken for Chunk Append operations when PartType is HASH\_PARTITIONED or BUCKET\_SHUFFLE\_HASH\_PARTITIONED. | | ShuffleHashTime | Time taken to calculate hash when PartType is HASH\_PARTITIONED or BUCKET\_SHUFFLE\_HASH\_PARTITIONED. | | RequestSent | Number of data packets sent. | | RequestUnsent | Number of unsent data packets. This metric is non-zero when there is a short-circuit logic; otherwise, it is zero. | | BytesSent | Size of sent data. | | BytesUnsent | Size of unsent data. This metric is non-zero when there is a short-circuit logic; otherwise, it is zero. | | BytesPassThrough | If the destination node is the current node, data will not be transmitted over the network, which is called passthrough data. This metric indicates the size of such passthrough data. Passthrough is controlled by `enable_exchange_pass_through`. | | PassThroughBufferPeakMemoryUsage | Peak memory usage of the PassThrough Buffer. | | CompressTime | Compression time. | | CompressedInputBytes | Size of the serialized (pre-compression) data that was actually fed to the compressor. Chunks skipped by the adaptive compression strategy are not counted. `CompressedInputBytes / CompressedBytes` gives the compression ratio, and `SerializedBytes - CompressedInputBytes` is the size of data that was not compressed. | | CompressedBytes | Size of compressed data. Only chunks that were actually compressed are counted. | | OverallThroughput | Throughput rate. | | NetworkTime | Time taken for data packet transmission (excluding post-reception processing time). | | NetworkBandwidth | Estimated network bandwidth. | | WaitTime | Waiting time due to a full sender queue. | | OverallTime | Total time for the entire transmission process, i.e., from sending the first data packet to confirming the correct reception of the last data packet. | | RpcAvgTime | Average time for RPC. | | RpcCount | Total number of RPCs. | ###### Exchange Source Operator[​](#exchange-source-operator "Direct link to Exchange Source Operator") | Metric | Description | | ------------------------ | ---------------------------------------------- | | RequestReceived | Size of received data packets. | | BytesReceived | Size of received data. | | DecompressChunkTime | Time taken to decompress chunks. | | DeserializeChunkTime | Time taken to deserialize chunks. | | ClosureBlockCount | Number of blocked Closures. | | ClosureBlockTime | Blocked time for Closures. | | ReceiverProcessTotalTime | Total time taken for receiver-side processing. | | WaitLockTime | Lock waiting time. | ##### Aggregate Operator[​](#aggregate-operator "Direct link to Aggregate Operator") **Metrics List** | Metric | Description | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | | `GroupingKeys` | `GROUP BY` columns. | | `AggregateFunctions` | Time taken for aggregate function calculations. | | `AggComputeTime` | Time for AggregateFunctions + Group By. | | `ChunkBufferPeakMem` | Peak memory usage of the Chunk Buffer. | | `ChunkBufferPeakSize` | Peak size of the Chunk Buffer. | | `ExprComputeTime` | Time for expression computation. | | `ExprReleaseTime` | Time for expression release. | | `GetResultsTime` | Time to extract aggregate results. | | `HashTableSize` | Size of the Hash Table. | | `HashTableMemoryUsage` | Memory size of the Hash Table. | | `InputRowCount` | Number of input rows. | | `PassThroughRowCount` | In Auto mode, the number of data rows processed in streaming mode after low aggregation leads to degradation to streaming mode. | | `ResultAggAppendTime` | Time taken to append aggregate result columns. | | `ResultGroupByAppendTime` | Time taken to append Group By columns. | | `ResultIteratorTime` | Time to iterate over the Hash Table. | | `StreamingTime` | Processing time in streaming mode. | ##### Join Operator[​](#join-operator "Direct link to Join Operator") **Metrics List** | Metric | Description | | --------------------------- | -------------------------------------------------------------------- | | `DistributionMode` | Distribution type, including: BROADCAST, PARTITIONED, COLOCATE, etc. | | `JoinPredicates` | Join predicates. | | `JoinType` | Join type. | | `BuildBuckets` | Number of buckets in the Hash Table. | | `BuildKeysPerBucket` | Number of keys per bucket in the Hash Table. | | `BuildConjunctEvaluateTime` | Time taken for conjunct evaluation during build phase. | | `BuildHashTableTime` | Time taken to build the Hash Table. | | `ProbeConjunctEvaluateTime` | Time taken for conjunct evaluation during probe phase. | | `SearchHashTableTimer` | Time taken to search the Hash Table. | | `CopyRightTableChunkTime` | Time taken to copy chunks from the right table. | | `OutputBuildColumnTime` | Time taken to output the column of build side. | | `OutputProbeColumnTime` | Time taken to output the column of probe side. | | `HashTableMemoryUsage` | Memory usage of the Hash Table. | | `RuntimeFilterBuildTime` | Time taken to build runtime filters. | | `RuntimeFilterNum` | Number of runtime filters. | ##### Window Function Operator[​](#window-function-operator "Direct link to Window Function Operator") | Metric | Description | | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ProcessMode` | Execution mode, including two parts: the first part includes Materializing and Streaming; the second part includes Cumulative, RemovableCumulative, ByDefinition. | | `ComputeTime` | Time taken for window function calculations. | | `PartitionKeys` | Partition columns. | | `AggregateFunctions` | Aggregate functions. | | `ColumnResizeTime` | Time taken for column resizing. | | `PartitionSearchTime` | Time taken to search partition boundaries. | | `PeerGroupSearchTime` | Time taken to search Peer Group boundaries. Meaningful only when the window type is `RANGE`. | | `PeakBufferedRows` | Peak number of rows in the buffer. | | `RemoveUnusedRowsCount` | Number of times unused buffers are removed. | | `RemoveUnusedTotalRows` | Total number of rows removed from unused buffers. | ##### Sort Operator[​](#sort-operator "Direct link to Sort Operator") | Metric | Description | | ------------------ | ----------------------------------------------------------------------- | | `SortKeys` | Sorting keys. | | `SortType` | Query result sorting method: full sorting or sorting the top N results. | | `MaxBufferedBytes` | Peak size of buffered data. | | `MaxBufferedRows` | Peak number of buffered rows. | | `NumSortedRuns` | Number of sorted runs. | | `BuildingTime` | Time taken to maintain internal data structures during sorting. | | `MergingTime` | Time taken to merge sorted runs during sorting. | | `SortingTime` | Time taken for sorting. | | `OutputTime` | Time taken to build the output sorted sequence. | ##### Merge Operator[​](#merge-operator "Direct link to Merge Operator") | Metric | Description | Level | | ---------------------------------------- | ------------------------------------------------------------------------------------ | --------- | | `Limit` | Limit. | Primary | | `Offset` | Offset. | Primary | | `StreamingBatchSize` | Size of data processed per Merge operation when Merge is performed in Streaming mode | Primary | | `LateMaterializationMaxBufferChunkNum` | Maximum number of chunks in the buffer when late materialization is enabled. | Primary | | `OverallStageCount` | Total execution count of all stages. | Primary | | `OverallStageTime` | Total execution time for each stage. | Primary | | `1-InitStageCount` | Execution count of the Init stage. | Secondary | | `2-PrepareStageCount` | Execution count of the Prepare stage. | Secondary | | `3-ProcessStageCount` | Execution count of the Process stage. | Secondary | | `4-SplitChunkStageCount` | Execution count of the SplitChunk stage. | Secondary | | `5-FetchChunkStageCount` | Execution count of the FetchChunk stage. | Secondary | | `6-PendingStageCount` | Execution count of the Pending stage. | Secondary | | `7-FinishedStageCount` | Execution count of the Finished stage. | Secondary | | `1-InitStageTime` | Execution time for the Init stage. | Secondary | | `2-PrepareStageTime` | Execution time for the Prepare stage. | Secondary | | `3-ProcessStageTime` | Execution time for the Process stage. | Secondary | | `4-SplitChunkStageTime` | Time taken for the Split stage. | Secondary | | `5-FetchChunkStageTime` | Time taken for the Fetch stage. | Secondary | | `6-PendingStageTime` | Time taken for the Pending stage. | Secondary | | `7-FinishedStageTime` | Time taken for the Finished stage. | Secondary | | `LateMaterializationGenerateOrdinalTime` | Time taken for generating ordinal columns during late materialization. | Tertiary | | `SortedRunProviderTime` | Time taken to retrieve data from the provider during the Process stage. | Tertiary | ##### TableFunction Operator[​](#tablefunction-operator "Direct link to TableFunction Operator") | Metric | Description | | ------------------------ | -------------------------------------------- | | `TableFunctionExecTime` | Computation time for the Table Function. | | `TableFunctionExecCount` | Number of executions for the Table Function. | ##### Project Operator[​](#project-operator "Direct link to Project Operator") Project Operator is responsible for performing `SELECT `. If there're some expensive expressions in the query, this operator can take significant time. | Metric | Description | | -------------------------- | -------------------------------------------- | | `ExprComputeTime` | Computation time for expressions. | | `CommonSubExprComputeTime` | Computation time for common sub-expressions. | ##### LocalExchange Operator[​](#localexchange-operator "Direct link to LocalExchange Operator") | Metric | Description | | ------------------------------------ | ------------------------------------------------------------------------------- | | Type | Type of Local Exchange, including: `Passthrough`, `Partition`, and `Broadcast`. | | `ShuffleNum` | Number of shuffles. This metric is only valid when `Type` is `Partition`. | | `LocalExchangePeakMemoryUsage` | Peak memory usage. | | `LocalExchangePeakBufferSize` | Peak size of the buffer. | | `LocalExchangePeakBufferMemoryUsage` | Peak memory usage of the buffer. | | `LocalExchangePeakBufferChunkNum` | Peak number of chunks in the buffer. | | `LocalExchangePeakBufferRowNum` | Peak number of rows in the buffer. | | `LocalExchangePeakBufferBytes` | Peak size of data in the buffer. | | `LocalExchangePeakBufferChunkSize` | Peak size of chunks in the buffer. | | `LocalExchangePeakBufferChunkRowNum` | Peak number of rows per chunk in the buffer. | | `LocalExchangePeakBufferChunkBytes` | Peak size of data per chunk in the buffer. | ##### OlapTableSink Operator[​](#olaptablesink-operator "Direct link to OlapTableSink Operator") OlapTableSink Operator is responsible for performing the `INSERT INTO
` operation. tip * An excessive difference between the Max and Min values of the `PushChunkNum` metric of `OlapTableSink` indicates data skew in the upstream operators, which may lead to a bottleneck in loading performance. * `RpcClientSideTime` equals `RpcServerSideTime` plus network transmission time plus RPC framework processing time. If there is a significant difference between `RpcClientSideTime` and `RpcServerSideTime`, consider enabling compression to reduce transmission time. | Metric | Description | | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | `IndexNum` | Number of the synchronous materialized views created for the destination table. | | `ReplicatedStorage` | Whether Single Leader Replication is enabled. | | `TxnID` | ID of the loading transaction. | | `RowsRead` | Number of rows read from upstream operators. | | `RowsFiltered` | Number of rows filtered out due to inadequate data quality. | | `RowsReturned` | Number of rows written to the destination table. | | `RpcClientSideTime` | Total RPC time consumption for loading recorded by the client side. | | `RpcServerSideTime` | Total RPC time consumption for loading recorded by the server side. | | `PrepareDataTime` | Total time consumption for the data preparation phase, including data format conversion and data quality check. | | `SendDataTime` | Local time consumption for sending the data, including time for serializing and compressing data, and for submitting tasks to the sender queue. | --- ### Query Profile Overview #### Introduction[​](#introduction "Direct link to Introduction") Query Profile records execution information for all working nodes involved in a query, helping you quickly identify bottlenecks affecting query performance. It is a powerful tool for diagnosing and tuning query performance in StarRocks. > From v3.3.0 onwards, StarRocks supports providing Query Profile for data loading with INSERT INTO FILES() and Broker Load. For details of the metrics involved, see [OlapTableSink Operator](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_operator_metrics.md#olaptablesink-operator). #### How to Enable Query Profile[​](#how-to-enable-query-profile "Direct link to How to Enable Query Profile") ##### Enable Query Profile[​](#enable-query-profile "Direct link to Enable Query Profile") You can enable Query Profile by setting the variable `enable_profile` to `true`: ```sql SET enable_profile = true; SET GLOBAL enable_profile = true; ``` ##### Query Profile for Slow Queries[​](#query-profile-for-slow-queries "Direct link to Query Profile for Slow Queries") It is not recommended to enable Query Profile globally in production for long periods, as it may impose additional system overhead. To capture and analyze only slow queries, set the variable `big_query_profile_threshold` to a time duration greater than `0s`. For example, setting it to `30s` means only queries exceeding 30 seconds will trigger Query Profile. ```sql -- 30 seconds SET global big_query_profile_threshold = '30s'; -- 500 milliseconds SET global big_query_profile_threshold = '500ms'; -- 60 minutes SET global big_query_profile_threshold = '60m'; ``` ##### Runtime Query Profile[​](#runtime-query-profile "Direct link to Runtime Query Profile") For long-running queries, it can be difficult to determine progress or detect issues before completion. The Runtime Query Profile feature (v3.1+) collects and reports Query Profile data at fixed intervals during execution, providing real-time insight into query progress and bottlenecks. When Query Profile is enabled, Runtime Query Profile is automatically activated with a default reporting interval of 10 seconds. Adjust the interval with `runtime_profile_report_interval`: ```sql SET runtime_profile_report_interval = 30; ``` ##### Configurations[​](#configurations "Direct link to Configurations") | Configuration Item | Type | Valid Values | Default | Description | | ------------------------------------ | ----------- | ---------------- | ------- | -------------------------------------------------------------------------------------- | | enable\_profile | Session Var | true/false | false | Enable Query Profile | | pipeline\_profile\_level | Session Var | 1/2 | 1 | 1: merge metrics; 2: retain original structure (disables visualization tools) | | runtime\_profile\_report\_interval | Session Var | Positive integer | 10 | Runtime Query Profile report interval (seconds) | | big\_query\_profile\_threshold | Session Var | String | 0s | Enable Query Profile for queries exceeding this duration (e.g., '30s', '500ms', '60m') | | enable\_statistics\_collect\_profile | FE Dynamic | true/false | false | Enable Query Profile for statistics collection-related queries | #### How to Obtain Query Profile[​](#how-to-obtain-query-profile "Direct link to How to Obtain Query Profile") ##### Via Web UI[​](#via-web-ui "Direct link to Via Web UI") 1. Access `http://:` in your browser. 2. Click **queries** in the top navigation. 3. In the **Finished Queries** list, select the query you want to analyze and click the link in the **Profile** column. ![img](/assets/images/profile-1-e14d11ad744041b7af92ed2ee2b98758.png) You will be redirected to the detailed page of the selected Query Profile. ![img](/assets/images/profile-2-2801dbb3484feef652b8a1287838b0e0.png) ##### Via SQL Function (`get_query_profile`)[​](#via-sql-function-get_query_profile "Direct link to via-sql-function-get_query_profile") Example workflow: * `last_query_id()`: Returns the ID of the most recently executed query in your session. Useful for quickly retrieving the profile of your last query. * `show profilelist;`: Lists recent queries along with their IDs and status. Use this to find the `query_id` needed for profile analysis. * `get_query_profile('')`: Returns the detailed execution profile for the specified query. Use this to analyze how a query was executed and where time or resources were spent. ```sql -- Enable the profiling feature. SET enable_profile = true; -- Run a query that performs a scan and aggregation to generate a meaningful profile. -- (Using a system table ensures this works on any cluster) SELECT count(*) FROM information_schema.columns; -- Get the query_id of the query. SELECT last_query_id(); +--------------------------------------+ | last_query_id() | +--------------------------------------+ | 019b364f-10c4-704c-b79a-af2cc3a77b89 | +--------------------------------------+ -- Get the list of profiles SHOW PROFILELIST; -- Obtain the query profile. SELECT get_query_profile('019b364f-10c4-704c-b79a-af2cc3a77b89')\G ``` ##### In Managed Version[​](#in-managed-version "Direct link to In Managed Version") In StarRocks Managed (Enterprise) environments, you can conveniently access query profiles directly from the query history in the web console. The managed UI provides an intuitive, visual representation of each query's execution profile, making it easy to analyze performance and identify bottlenecks without manual SQL commands. #### Interpret Query Profile[​](#interpret-query-profile "Direct link to Interpret Query Profile") ##### Explain Analyze[​](#explain-analyze "Direct link to Explain Analyze") Most users may find it challenging to analyze the raw text directly. StarRocks provides a [Text-based Query Profile Visualized Analysis](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_text_based_analysis.md) method for a more intuitive understanding. ##### Managed Version[​](#managed-version "Direct link to Managed Version") In the StarRocks Enterprise Edition (EE), the Managed Version provides a built-in visualization tool for query profiles. This tool offers an interactive, graphical interface that makes it much easier to interpret complex query execution details compared to raw text output. **Key features of the visualization tool include:** * **Operator-level breakdown:** View the execution plan as a tree or graph, with each operator's metrics (time, rows, memory) clearly displayed. * **Bottleneck highlighting:** Quickly identify slow or resource-intensive operators through color-coded indicators. * **Drill-down capability:** Click on any operator to see detailed statistics, including input/output rows, CPU time, memory usage, and more. **How to use:** 1. Open the StarRocks Managed web console. 2. Navigate to the **Query** or **Query History** section. 3. Select a query and click the **Profile** or **Visualize** button. 4. Explore the visualized profile to analyze performance and identify optimization opportunities. This visualization tool is exclusive to the Managed/Enterprise Edition and is designed to accelerate troubleshooting and performance tuning for complex workloads. --- ### Explain Analyze This document explains how to obtain and analyze text-based Query Profiles in StarRocks. It will help you understand query performance and find ways to optimize your SQL queries. #### Analyze Profiles of Existing Queries Using ANALYZE PROFILE[​](#analyze-profiles-of-existing-queries-using-analyze-profile "Direct link to Analyze Profiles of Existing Queries Using ANALYZE PROFILE") To analyze the text-based Profile of an existing (historical or running) query in you cluster, you first need to use the [SHOW PROFILELIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plan_profile/SHOW_PROFILELIST.md) statement to obtain a summary of the query. This command lists all queries that have finished successfully, failed with error, and those are still running (for more than 10 seconds and not yet finished). Through this statement, you can get the corresponding Query ID for subsequent analysis. The syntax is as follows: ```sql SHOW PROFILELIST [LIMIT ]; ``` Examples: ```sql SHOW PROFILELIST; SHOW PROFILELIST LIMIT 5; ``` Output: ```plaintext +--------------------------------------+---------------------+-------+----------+-----------------------------------------------------------------------------------------------------------------------------------+ | QueryId | StartTime | Time | State | Statement | +--------------------------------------+---------------------+-------+----------+-----------------------------------------------------------------------------------------------------------------------------------+ | a40456b2-8428-11ee-8d02-6a32f8c68848 | 2023-11-16 10:34:18 | 21ms | Finished | SELECT ROUTINE_NAME FROM INFORMATION_SCHEMA.ROUTINES\n WHERE ROUTINE_TYPE="FUNCTION" AND ROUTINE_SCHEMA = "None" | | a3fc4060-8428-11ee-8d02-6a32f8c68848 | 2023-11-16 10:34:17 | 39ms | Finished | select TABLE_NAME, COLUMN_NAME from information_schema.columns\n where table_schema = 'Non ... | | a3f7d38d-8428-11ee-8d02-6a32f8c68848 | 2023-11-16 10:34:17 | 15ms | Finished | select connection_id() | | a3efbd3b-8428-11ee-8d02-6a32f8c68848 | 2023-11-16 10:34:17 | 16ms | Finished | select connection_id() | | a26ec286-8428-11ee-8d02-6a32f8c68848 | 2023-11-16 10:34:15 | 269ms | Error | EXPLAIN ANALYZE SELECT c_nation, s_nation, year(lo_orderdate) AS year , SUM(lo_revenue) AS revenue FROM lineorder_flat WHERE ... | +--------------------------------------+---------------------+-------+----------+-----------------------------------------------------------------------------------------------------------------------------------+ ``` Once you have the Query ID, you can proceed with Query Profile analysis using the [ANALYZE PROFILE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plan_profile/ANALYZE_PROFILE.md) statement. The syntax is as follows: ```sql ANALYZE PROFILE FROM '' [, [, ...] ] ``` * `Query_ID`: The ID corresponding to the query, obtained from the `SHOW PROFILELIST` statement. * `Node_ID`: Profile node ID. For nodes whose IDs are specified, StarRocks returns detailed metric information for those node. For nodes whose IDs are not specified, StarRocks only returns the summary information. The Profile includes the following sections: * Summary: Summary information of the Profile. * QueryID * Version information * Query status, including `Finished`, `Error`, and `Running`. * Total query time. * Memory usage * Top 10 nodes with the highest CPU usage. * Top 10 nodes with the highest memory usage. * Session variables who has a different value from the default value. * Fragments: Displays metrics for each node in each Fragment. * Time, memory usage, cost estimate information, and output rows for each node. * Nodes with a time usage percentage exceeding 30% are highlighted in red. * Nodes with a time usage percentage exceeding 15% and less than 30% are highlighted in pink. Example 1: Querying the Query Profile without specifying node ID. ![img](/assets/images/text_based_profile_without_node_id-07ed49a240ec43e9f3a498dcefba5c70.jpeg) Example 2: Querying the Query Profile and specifying node ID as `0`. StarRocks returns all detailed metrics for Node ID `0` and highlights metrics with high usage for easier problem identification. ![img](/assets/images/text_based_profile_with_node_id-ed71ee1e7f25bdbb97752e03e7be34d1.jpeg) In addition, the above methods also support the display and analysis of Runtime Query Profile, that is, Profile generated for running queries. When the Query Profile feature is enabled, you can use this method to obtain the Profile of queries that are currently running for more than 10 seconds. Compared to those of the finished queries, the text-based Query Profile for running queries contains the following information: * Operator status: * ⏳: Operators not started. These operators may not have started execution due to dependency relationships. * 🚀: Running operators. * ✅: Operators that have finished execution. * Overall progress: Calculated based on `number of operators that have finished execution / total number of operators`. Due to the lack of detailed information on data rows, this value may be slightly distorted. * Operator progress: Calculated based on `number of rows processed / total number of rows`. If the total number of rows cannot be calculated, the progress is displayed as `?`. Example: ![img](/assets/images/text_based_runtime_profile-7957caf58242cda61abe63939311a74c.jpeg) #### Simulate a query for Profile Analysis Using EXPLAIN ANALYZE[​](#simulate-a-query-for-profile-analysis-using-explain-analyze "Direct link to Simulate a query for Profile Analysis Using EXPLAIN ANALYZE") StarRocks provides the [EXPLAIN ANALYZE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plan_profile/EXPLAIN_ANALYZE.md) statement, allowing you to simulate and analyze the profile of a query directly. The syntax is as follows: ```sql EXPLAIN ANALYZE ``` When executing `EXPLAIN ANALYZE`, StarRocks will, by default, enable the Query Profile feature for the current session. Currently, `EXPLAIN ANALYZE` supports two types of SQL statements: SELECT statements and INSERT INTO statements. You can only simulate and analyze the Query Profile of INSERT INTO statements on internal tables within the default catalog of StarRocks. Please note that when simulating and analyzing the Query Profile of INSERT INTO statements, no actual data will be loaded. By default, the import transaction will be aborted to ensure that no unintended changes are made to the data during the analysis. Example 1: Simulate and analyze a SELECT statement. The query result are discarded. ![img](/assets/images/text_based_explain_analyze_select-54f611db544479a1b4df4aadab264388.jpeg) Example 2: Simulate and analyze an INSERT INTO statement. The loading transaction will be aborted. ![img](/assets/images/text_based_explain_analyze_insert-178eb5df5ee41c8a33f648ece8a9bff8.jpeg) #### Limitations[​](#limitations "Direct link to Limitations") * `EXPLAIN ANALYZE INSERT INTO` statements are only supported for tables in the default catalog. * To achieve better visual effects, the output text contains ANSI characters to provide color, highlighting, and other features. It is recommended to use the MyCLI client. For clients that do not support ANSI features, such as the MySQL client, there may be some slight display disorders. Usually, they will not affect the usage. For example: ![img](/assets/images/text_based_profile_not_aligned-ce88feb8e672741e051b69db71806121.jpeg) --- ### Query Tuning Recipes > A pragmatic playbook: **symptom → root cause → proven fixes**.
Use it when you’ve opened a profile and spotted a red-flag metric but still need to answer “*now what?*”. *** #### 1 · Fast Diagnosis Workflow[​](#1--fast-diagnosis-workflow "Direct link to 1 · Fast Diagnosis Workflow") 1. **Skim the Execution Overview**
If `QueryPeakMemoryUsagePerNode > 80 %` or `QuerySpillBytes > 1 GB`, jump straight to the memory & spill recipes. 2. **Find the slowest Pipeline / Operator**
⟶ In *Query Profile UI* click **Sort by OperatorTotalTime %**.
The hottest operator tells you which recipe block to read next (Scan, Join, Aggregate, …). 3. **Confirm the bottleneck subtype**
Each recipe begins with its *signature* metric pattern. Match those before trying the fixes. *** #### 2 · Recipes by Operator[​](#2--recipes-by-operator "Direct link to 2 · Recipes by Operator") ##### 2.1 OLAP / Connector Scan [\[metrics\]](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_operator_metrics.md#scan-operator)[​](#21-olap--connector-scan--metrics "Direct link to 21-olap--connector-scan--metrics") To facilitate a better understanding of the various metrics within the Scan Operator, the following diagram demonstrates the associations between these metrics and storage structures. ![profile\_scan\_relationship](/assets/images/profile_scan_relationship-416858c52e5b2db2f66c2f3da1f948ac.jpeg) To retrieve data from disk and apply the predicates, the storage engine utilize several techniques: 1. **Data Storage**: Encoded and compressed data is stored on disk in segments, accompanied by various indices. 2. **Index Filtering**: The engine leverages indices such as BitmapIndex, BloomfilterIndex, ZonemapIndex, ShortKeyIndex, and NGramIndex to skip unnecessary data. 3. **Pushdown Predicates**: Simple predicates, like `a > 1`, are pushed down to evaluate on specific columns. 4. **Late Materialization**: Only the required columns and filtered rows are retrieved from disk. 5. **Non-Pushdown Predicates**: Predicates that cannot be pushed down are evaluated. 6. **Projection Expression**: Expressions, such as `SELECT a + 1`, are computed. The Scan Operator utilizes an additional thread pool for executing IO tasks. Therefore, the relationship between time metrics for this node is illustrated below: ![profile\_scan\_time\_relationship](/assets/images/profile_scan_time_relationship-ea4e7d277c9e7a17db3efe120866370a.jpeg) ###### Common performance bottlenecks[​](#common-performance-bottlenecks "Direct link to Common performance bottlenecks") **Cold or slow storage** – When `BytesRead`, `ScanTime`, or `IOTaskExecTime` dominate and disk I/O hovers around 80‑100 %, the scan is hitting cold or under‑provisioned storage. Move hot data to NVMe/SSD and enable the Data Cache. Size it via BE `datacache_*` settings (or legacy `block_cache_*`), and enable scan‑time usage via session `enable_scan_datacache`. **Filter push‑down missing** – If `PushdownPredicates` stays near 0 while `ExprFilterRows` is high, predicates aren’t reaching the storage layer. Rewrite them as simple comparisons (avoid `%LIKE%` and wide `OR` chains) or add zonemap/Bloom indexes or materialized views so they can be pushed down. **Thread‑pool starvation** – A high `IOTaskWaitTime` together with a low `PeakIOTasks` signals saturated I/O concurrency. Enable and size the Data Cache (BE `datacache_*` and session `enable_scan_datacache`), move hot data to faster storage (NVMe/SSD) **Data skew across tablets** – A wide gap between the maximum and minimum `OperatorTotalTime` means some tablets do much more work than others. Re‑bucket on a higher‑cardinality key or increase the bucket count to spread the load. **Rowset/segment fragmentation** – Exploding `RowsetsReadCount`/`SegmentsReadCount` plus a long `SegmentInitTime` indicate many tiny rowsets. Trigger a manual compaction and batch small loads so segments merge up‑front. **Accumulated soft deletes** – A large `DeleteFilterRows` implies heavy soft‑delete usage. Run BE compaction to purge soft deletes. ##### 2.2 Aggregate [\[metrics\]](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_operator_metrics.md#aggregate-operator)[​](#22-aggregate--metrics "Direct link to 22-aggregate--metrics") ![aggregation\_operator](/assets/images/aggregation_operator-a61f7fc9ca608ba1be4a9767eea86b74.png) Aggregate Operator is responsible for executing aggregation functions, `GROUP BY`, and `DISTINCT`. **Multi forms of aggregation algorithm** | Form | When the planner chooses it | Internal data structure | Highlights / caveats | | ---------------------------- | --------------------------------------------- | -------------------------------------------- | -------------------------------------------------------------- | | Hash aggregation | keys fit into memory; cardinality not extreme | Compact hash table with SIMD probing | default path, excellent for modest key counts | | Sorted aggregation | input already ordered on the GROUP BY keys | Simple row comparison + running state | zero hash table cost, often 2-3× faster on probing heavy skews | | Spillable aggregation (3.2+) | hash table outsizes memory limit | Hybrid hash/merge with disk spill partitions | prevents OOM, preserves pipeline parallelism | **Multi-Stage Distributed Aggregation** In StarRocks the aggregation is implemented in distributed manner, which can be multi-stage depends on the query pattern and optimizer decision. ```text ┌─────────┐ ┌──────────┐ ┌────────────┐ ┌────────────┐ │ Stage 0 │ local │ Stage 1 │ shard/ │ Stage 2 │ gather/│ Stage 3 │ final │ Partial │───► │ Update │ hash │ Merge │ shard │ Finalize │ output └─────────┘ └──────────┘ └────────────┘ └────────────┘ ``` | Stages | When Used | What Happens | | --------------------------------------------------- | ---------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | One-stage | The `DISTRIBUTED BY` is a subset of `GROUP BY`, the partitions are colocated | Partial aggregates immediately become the final result. | | Two-stage (local + global) | Typical distributed `GROUP BY` | Stage 0 inside each BE collapses duplicates adaptively; Stage 1 shuffles data based on `GROUP BY` then perform global aggregation | | Three-stage (local + shuffle + final) | Heavy `DISTINCT` and high-cardinality `GROUP BY` | Stage 0 as above; Stage 1 shuffles by `GROUP BY`, then aggregate by `GROUP BY` and `DISTINCT`; Stage 2 merges partial state as `GROUP BY` | | Four-stage (local + partial + intermediate + final) | Heavy `DISTINCT` and low-cardinality `GROUP BY` | Introduce an additional stage to shuffle by `GROUP BY` and `DISTINCT` to avoid single-point bottleneck | ###### Common performance bottlenecks[​](#common-performance-bottlenecks-1 "Direct link to Common performance bottlenecks") **High‑cardinality GROUP BY** – When `HashTableSize` or `HashTableMemoryUsage` balloons toward the memory limit, the grouping key is too wide or too distinct. Enable sorted streaming aggregation (`enable_streaming_preaggregation = true`), create a roll‑up materialized view, or cast wide string keys to `INT`. **Shuffle skew** – Large differences in `HashTableSize` or `InputRowCount` across fragments reveal an unbalanced shuffle. Add a salt column to the key or use the `DISTINCT [skew]` hint so rows distribute evenly. **State‑heavy aggregate functions** – If `AggregateFunctions` dominates runtime and the functions include `HLL_`, `BITMAP_`, or `COUNT(DISTINCT)`, enormous state objects are being moved around. Pre‑compute HLL/bitmap sketches during ingestion or switch to approximate variants. **Partial aggregation degraded** – A huge `InputRowCount` with modest `AggComputeTime`, plus massive `BytesSent` in the upstream EXCHANGE, means pre‑aggregation was bypassed. Force it back on with `SET streaming_preaggregation_mode = "force_preaggregation"`. **Expensive key expressions** – When `ExprComputeTime` rivals `AggComputeTime`, the GROUP BY keys are computed row by row. Materialize those expressions in a sub‑query or promote them to generated columns. ##### 2.3 Join [\[metrics\]](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_operator_metrics.md#join-operator)[​](#23-join--metrics "Direct link to 23-join--metrics") ![join\_operator](/assets/images/join_operator-260010cf2d3e1088eee85c6fab0f7c1e.png) Join Operator is responsible for implementing explicit join or implicit joins. During execution the join operator is split into Build (hash-table construction) and Probe phases that run in parallel inside the pipeline engine. Vector chunks (up to 4096 rows) are batch-hashed with SIMD; consumed keys generate runtime filters—Bloom or IN filters—pushed back to upstream scans to cut probe input early. **Join Strategies** StarRocks relies on a vectorized, pipeline-friendly hash-join core that can be wired into four physical strategies the cost-based optimizer weighs at plan time: | Strategy | When the optimizer picks it | What makes it fast | | ------------------- | ------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------- | | Colocate Join | Both tables belong to the same colocation group (identical bucket keys, bucket count, and replica layout).  | No network shuffle: each BE joins only its local buckets. | | Bucket-Shuffle Join | One of join tables has the same bucket key with join key | Only need to shuffle one join table, which can reduce the network cost | | Broadcast Join | Build side is very small (row/byte thresholds or explicit hint).  | Small table is replicated to every probe node; avoids shuffling large table. | | Shuffle (Hash) Join | General case, keys don’t align. | Hash-partition each row on the join key so probes are balanced across BEs. | ###### Common performance bottlenecks[​](#common-performance-bottlenecks-2 "Direct link to Common performance bottlenecks") **Oversized build side** – Spikes in `BuildHashTableTime` and `HashTableMemoryUsage` show the build side has outgrown memory. Swap probe/build tables, pre‑filter the build table, or enable hash spilling. **Cache‑unfriendly probe** – When `SearchHashTableTime` dominates, the probe side is not cache‑efficient. Sort the probe rows on the join keys and enable runtime filters. **Shuffle skew** – If a single fragment’s `ProbeRows` dwarfs all others, the data is skewed. Switch to a higher‑cardinality key or append a salt such as `key || mod(id, 16)`. **Accidental broadcast** – Join type **BROADCAST** with huge `BytesSent` means a table you thought was small isn’t. Lower `broadcast_row_limit` or enforce shuffle with the `SHUFFLE` hint. **Missing runtime filters** – A tiny `JoinRuntimeFilterEvaluate` together with full‑table scans suggests runtime filters never propagated. Rewrite the join as pure equality and make sure the column types line up. **Non‑equi fallback** – When the operator type is `CROSS` or `NESTLOOP`, an inequality or function prevents a hash join. Add a true equality predicate or pre‑filter the larger table. ##### 2.4 Exchange (Network) [\[metrics\]](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_operator_metrics.md#exchange-operator)[​](#24-exchange-network--metrics "Direct link to 24-exchange-network--metrics") **Oversized shuffle or broadcast** – If `NetworkTime` exceeds 30 % and `BytesSent` is large, the query is shipping too much data. Re‑evaluate the join strategy and reduce the shuffle/broadcast volume (e.g., enforce shuffle instead of broadcast, or pre‑filter upstream). **Receiver backlog** – High `WaitTime` in the sink with sender queues that stay full indicates the receiver cannot keep up. Increase the receiver thread pool (`brpc_num_threads`) and confirm NIC bandwidth and QoS settings. **Enable exchange compression** – When network bandwidth is the bottleneck, compress exchange payloads. Set `SET transmission_compression_type = 'zstd';` and optionally increase `SET transmission_encode_level = 7;` to enable adaptive column encoding. Expect higher CPU usage in exchange for reduced bytes on the wire. ##### 2.5 Sort / Merge / Window[​](#25-sort--merge--window "Direct link to 2.5 Sort / Merge / Window") For ease of understanding various metrics, Merge can be represented as the following state mechanism: ```plaintext ┌────────── PENDING ◄──────────┐ │ │ │ │ ├──────────────◄───────────────┤ │ │ ▼ │ INIT ──► PREPARE ──► SPLIT_CHUNK ──► FETCH_CHUNK ──► FINISHED ▲ | | one traverse from leaf to root | ▼ PROCESS ``` **Sort spilling** – When `MaxBufferedBytes` rises above roughly 2 GB or `SpillBytes` is non‑zero, the sort phase is spilling to disk. Add a `LIMIT`, pre‑aggregate upstream, or raise `sort_spill_threshold` if the machine has enough memory. **Merge starvation** – A high `PendingStageTime` tells you the merge is waiting for upstream chunks. Optimize the producer operator first or enlarge pipeline buffers. **Wide window partitions** – Huge `PeakBufferedRows` inside a window operator point to very broad partitions or an ORDER BY lacking frame limits. Partition more granularly, add `RANGE BETWEEN` bounds, or materialize intermediate aggregates. *** #### 3 · Memory & Spill Cheatsheet[​](#3--memory--spill-cheatsheet "Direct link to 3 · Memory & Spill Cheatsheet") | Threshold | What to watch | Practical action | | --------------------------------- | --------------------------------------------- | --------------------------------------------------------------------- | | **80 %** of BE memory | `QueryPeakMemoryUsagePerNode` | Lower session `exec_mem_limit` or add BE RAM | | Spill detected (`SpillBytes` > 0) | `QuerySpillBytes`, per-operator `SpillBlocks` | Increase memory limit; upgrade to SR 3.2+ for hybrid hash/merge spill | *** #### 4 · Template for Your Post-mortem[​](#4--template-for-your-post-mortem "Direct link to 4 · Template for Your Post-mortem") ````text 1. Symptom – Slow stage: Aggregate (OperatorTotalTime 68 %) – Red-flag: HashTableMemoryUsage 9 GB (> exec_mem_limit) 2. Root cause – GROUP BY high-cardinality UUID 3. Fix applied – Added sorted streaming agg + roll-up MV 4. Outcome – Query runtime ↓ from 95 s ➜ 8 s; memory peak 0.7 GB``` ```` --- ### Schema Tuning Recipes This document provides practical tips and best practices for optimizing query performance in StarRocks through effective schema design and foundational table choices. By understanding how different table types, keys, and distribution strategies impact query execution, you can significantly improve both speed and resource efficiency. Use these guidelines to make informed decisions when designing schemas, selecting table types, and tuning your StarRocks environment for high-performance analytics. #### Table Type Selection[​](#table-type-selection "Direct link to Table Type Selection") StarRocks supports four table types: Duplicate Key table, Aggregate table, Unique Key table, and Primary Key table. All of them are sorted by KEY. * `AGGREGATE KEY`: When records with the same AGGREGATE KEY is loaded into StarRocks, the old and new records are aggregated. Currently, Aggregate tables supports the following aggregate functions: SUM, MIN, MAX, and REPLACE. Aggregate tables support aggregating data in advance, facilitating business statements and multi-dimensional analyses. * `DUPLICATE KEY`: You only need to specify the sort key for a DUPLICATE KEY table. Records with the same DUPLICATE KEY exist at the same time. It is suitable for analyses that do not involve aggregating data in advance. * `UNIQUE KEY`: When records with the same UNIQUE KEY is loaded into StarRocks, the new record overwrites the old one. A UNIQUE KEY tables is similar to an Aggregate table with REPLACE function. Both are suitable for analyses involving constant updates. * `PRIMARY KEY`: Primary Key tables guarantee the uniqueness of records, and allow you to perform realtime updating. ```sql CREATE TABLE site_visit ( siteid INT, city SMALLINT, username VARCHAR(32), pv BIGINT SUM DEFAULT '0' ) AGGREGATE KEY(siteid, city, username) DISTRIBUTED BY HASH(siteid); CREATE TABLE session_data ( visitorid SMALLINT, sessionid BIGINT, visittime DATETIME, city CHAR(20), province CHAR(20), ip varchar(32), browser CHAR(20), url VARCHAR(1024) ) DUPLICATE KEY(visitorid, sessionid) DISTRIBUTED BY HASH(sessionid, visitorid); CREATE TABLE sales_order ( orderid BIGINT, status TINYINT, username VARCHAR(32), amount BIGINT DEFAULT '0' ) UNIQUE KEY(orderid) DISTRIBUTED BY HASH(orderid); CREATE TABLE sales_order ( orderid BIGINT, status TINYINT, username VARCHAR(32), amount BIGINT DEFAULT '0' ) PRIMARY KEY(orderid) DISTRIBUTED BY HASH(orderid); ``` #### Colocate Table[​](#colocate-table "Direct link to Colocate Table") To speed up queries, tables with the same distribution can use a common bucketing column. In that case, data can be joined locally without being transferred across the cluster during the `join` operation. ```sql CREATE TABLE colocate_table ( visitorid SMALLINT, sessionid BIGINT, visittime DATETIME, city CHAR(20), province CHAR(20), ip varchar(32), browser CHAR(20), url VARCHAR(1024) ) DUPLICATE KEY(visitorid, sessionid) DISTRIBUTED BY HASH(sessionid, visitorid) PROPERTIES( "colocate_with" = "group1" ); ``` For more information about colocate join and replica management, see [Colocate join](https://docs.starrocks.io/docs/using_starrocks/Colocate_join.md) #### Flat table and star schema[​](#flat-table-and-star-schema "Direct link to Flat table and star schema") StarRocks supports star schema, which is more flexible in modelling than flat tables. You can create a view to replace flat tables during modelling and then query data from multiple tables to accelerate queries. Flat tables have the following drawbacks: * Costly dimension updates because a flat table usually contains a massive number of dimensions. Each time a dimension is updated, the entire table must be updated. The situation exacerbates as the update frequency increases. * High maintenance cost because flat tables require additional development workloads, storage space, and data backfilling operations. * High data ingestion cost because a flat table has many fields and an Aggregate table may contain even more key fields. During data loading, more fields need to be sorted, which prolongs data loading. If you have high requirements on query concurrency or low latency, you can still use flat tables. #### Partition and bucket[​](#partition-and-bucket "Direct link to Partition and bucket") StarRocks supports two levels of partitioning: the first level is RANGE partition and the second level is HASH bucket. * RANGE partition: RANGE partition is used to divide data into different intervals (can be understood as dividing the original table into multiple sub-tables). Most users choose to set partitions by time, which has the following advantages: * Easier to distinguish between hot and cold data * Be able to leverage StarRocks tiered storage (SSD + SATA) * Faster to delete data by partition * HASH bucket: Divides data into different buckets according to the hash value. * It is recommended to use a column with a high degree of discrimination for bucketing to avoid data skew. * To facilitate data recovery, it is recommended to keep the size of compressed data in each bucket between 100 MB to 1 GB. We recommend you configure an appropriate number of buckets when you create a table or add a partition. * Random bucketing is not recommended. You must explicitly specify the HASH bucketing column when you create a table. #### Sparse index and bloomfilter index[​](#sparse-index-and-bloomfilter-index "Direct link to Sparse index and bloomfilter index") StarRocks stores data in an ordered manner and builds sparse indexes at a granularity of 1024 rows. StarRocks selects a fixed-length prefix (currently 36 bytes) in the schema as the sparse index. When creating a table, it is recommended to place common filter fields at the beginning of the schema declaration. Fields with the highest differentiation and query frequency must be placed first. A VARCHAR field must placed at the end of a sparse index because the index gets truncated from the VARCHAR field. If the VARCHAR field appears first, the index may be less than 36 bytes. Use the above `site_visit` table as an example. The table has four columns: `siteid, city, username, pv`. The sort key contains three columns `siteid, city, username`, which occupy 4, 2, and 32 bytes respectively. So the prefix index (sparse index) can be the first 30 bytes of `siteid + city + username`. In addition to sparse indexes, StarRocks also provides bloomfilter indexes, which are effective for filtering columns with high discrimination. If you want to place VARCHAR fields before other fields, you can create bloomfilter indexes. #### Inverted Index[​](#inverted-index "Direct link to Inverted Index") StarRocks adopts Bitmap Indexing technology to support inverted indexes that can be applied to all columns of the Duplicate Key table and the key column of the Aggregate table and Unique Key table. Bitmap Index is suitable for columns with a small value range, such as gender, city, and province. As the range expands, the bitmap index expands in parallel. #### Materialized view (rollup)[​](#materialized-view-rollup "Direct link to Materialized view (rollup)") A rollup is essentially a materialized index of the original table (base table). When creating a rollup, only some columns of the base table can be selected as the schema, and the order of the fields in the schema can be different from that of the base table. Below are some use cases of using a rollup: * Data aggregation in the base table is not high, because the base table has fields with high differentiation. In this case, you may consider selecting some columns to create rollups. Use the above `site_visit` table as an example: ```sql site_visit(siteid, city, username, pv) ``` `siteid` may lead to poor data aggregation. If you need to frequently calculate PVs by city, you can create a rollup with only `city` and `pv`. ```sql ALTER TABLE site_visit ADD ROLLUP rollup_city(city, pv); ``` * The prefix index in the base table cannot be hit, because the way the base table is built cannot cover all the query patterns. In this case, you may consider creating a rollup to adjust the column order. Use the above `session_data` table as an example: ```sql session_data(visitorid, sessionid, visittime, city, province, ip, browser, url) ``` If there are cases where you need to analyze visits by `browser` and `province` in addition to `visitorid`, you can create a separate rollup: ```sql ALTER TABLE session_data ADD ROLLUP rollup_browser(browser,province,ip,url) DUPLICATE KEY(browser,province); ``` #### Schema change[​](#schema-change "Direct link to Schema change") There are three ways to change schemas in StarRocks: sorted schema change, direct schema change, and linked schema change. * Sorted schema change: Change the sorting of a column and reorder the data. For example, deleting a column in a sorted schema leads to data reorder. `ALTER TABLE site_visit DROP COLUMN city;` * Direct schema change: Transform the data instead of reordering it, for example, changing the column type or adding a column to a sparse index. `ALTER TABLE site_visit MODIFY COLUMN username varchar(64);` * Linked schema change: Complete changes without transforming data, for example, adding columns. `ALTER TABLE site_visit ADD COLUMN click bigint SUM default '0';` It is recommended to choose an appropriate schema when you create tables to accelerate schema changes. --- ### Table clustering A thoughtful sort‑key is the highest‑leverage physical‑design knob in StarRocks. This guide explains how the sort key works under the hood, the systemic benefits it unlocks, and a concrete playbook for picking an effective key for your own workload. #### Example[​](#example "Direct link to Example") Suppose you run a telemetry system that receives billions of rows per day, each tagged with a `device_id` and `ts` (timestamp). Defining `ORDER BY (device_id, ts)` on your fact table ensures: * Point queries on `device_id` return in milliseconds. * Dashboards filter recent time windows for each device prune most data. * Aggregations like `GROUP BY device_id` benefit from streaming aggregation. * Compression improves due to runs of nearby timestamps per device. This simple two-column sort key `ORDER BY (device_id, ts)` delivers I/O reduction, CPU savings, and more stable query performance across billions of rows. ```sql CREATE TABLE telemetry ( device_id VARCHAR, ts DATETIME, value DOUBLE ) ENGINE=OLAP PRIMARY KEY(device_id, ts) PARTITION BY date_trunc('day', ts) DISTRIBUTED BY HASH(device_id) BUCKETS 16 ORDER BY (device_id, ts); ``` *** #### Benefits in Depth[​](#benefits-in-depth "Direct link to Benefits in Depth") 1. Massive I/O Elimination—Segment & Page Pruning How it works: Each segment and 64 KB page stores min/max values for all columns. If a predicate falls outside that range, StarRocks skips the entire chunk and never touches the disk. Example: ```sql SELECT count(*) FROM events WHERE tenant_id = 42 AND ts BETWEEN '2025-05-01' AND '2025-05-07'; ``` With `ORDER BY (tenant_id, ts)` only the segments whose first key equals 42 are considered, and within them only the pages whose ts window overlaps those seven days. A 100 B‑row table may scan less than 1 B rows, turning minutes into seconds. *** 2. Millisecond Point  Look‑Ups—Sparse Prefix Index How it works: A sparse prefix index stores every ~1 Kth sort‑key value. A binary search lands on the right page, then a single disk read (often already cached) returns the row. Example: ```sql SELECT * FROM orders WHERE order_id = 982347234; ``` With `ORDER BY (order_id)` the probe needs ≈ 50 key comparisons across a 50 B‑row table—sub‑10 ms latency even on cold data cache. *** 3. Faster Sorted Aggregation How it works: When the sort key aligns with the GROUP BY clause, StarRocks performs streaming aggregation as it scans—no sorting or hash-table needed. This sorted-aggregation plan scans rows in sort-key order and emits groups on the fly, exploiting CPU cache locality and skipping intermediate materialisation. Example: ```sql SELECT device_id, COUNT(*) FROM telemetry WHERE ts BETWEEN '2025-01-01' AND '2025-01-31' GROUP BY device_id; ``` If the table is `ORDER BY (device_id, ts)`, the engine groups rows as they stream in—without building a hash table or re-sorting. For high-cardinality keys like device\_id, this can reduce both CPU and memory usage dramatically. Streaming aggregation with sorted input typically improves throughput by 2–3× over hash-aggregation for large group cardinalities. *** 4. Higher  Compression &  Hotter  Caches How it works: Sorted data shows small deltas or long runs, accelerating dictionary, RLE, and frame‑of‑reference encodings. Compact pages stream sequentially through CPU caches. Example: A telemetry table sorted by (device\_id, ts) achieved 1.8 × better compression (LZ4) and 25 % lower CPU/scan than the same data ingested unsorted. *** #### How the Sort Key Works[​](#how-the-sort-key-works "Direct link to How the Sort Key Works") The impact of a sort key starts the moment a row is written and persists through every read‑time optimisation. This section walks through that lifecycle—write path ➜ storage hierarchy ➜ segment internals ➜ read path—to show how each layer compounds the value of ordering. 1. Write Path 1. Ingest: rows land in a MemTable, are sorted by the declared sort key, and then flushed as a new Rowset containing one or more ordered Segments. 2. Compaction: Background cumulative/base jobs merge many small Rowsets into larger ones, reclaiming deletes and lowering segment count without re‑sorting, because every source Rowset already shares the same order. 3. Replication: Each Tablet (the shard that owns the Rowset) is synchronously replicated to peer Back‑End nodes, guaranteeing that sorted order is consistent across replicas. ![write path steps](/assets/images/table_clustering-1-33d2c6e3d94f21ba5f3cd79263a2f3e1.png) 2. Storage Hierarchy | Object | What it is | Why is matters for the sort key | | --------- | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ | | Partition | A coarse‑grained logical slice of a table (e.g., date or tenant\_id). | Enables planner‑time partition pruning and isolates lifecycle ops (TTL, bulk load). | | Tablet | A hash/random bucket within a partition, independently replicated across Back‑End nodes. | Unit whose rows are physically ordered by the sort key; all intra‑partition pruning starts here. | | MemTable | In‑memory write buffer (~96 MB) that sorts by the declared key before flushing to disk. | Guarantees that every on‑disk Segment is already ordered—No external sort needed later. | | Rowset | Immutable bundle of one or more Segments produced by a flush, streaming load, or compaction cycle. | Append‑only design lets StarRocks ingest concurrently while readers stay lock‑free. | | Segment | Self‑contained columnar file (~512 MB) inside a Rowset carrying data pages plus pruning indexes. | Segment‑level zone‑maps and prefix indexes rely on the order established at the MemTable stage. | 3. Inside a Segment File ![write path steps](/assets/images/table_clustering-2-63760bef534000a53042e303a88d795d.png) Each Segment is self‑describing. From top to bottom you’ll find: * Column data pages 64 KB blocks encoded (Dictionary, RLE, Delta) and compressed (LZ4 default). * Ordinal index Maps a row ordinal → page offset so the engine can jump directly to page n. * Zone‑map index min, max, and has\_null per page and for the whole Segment—first line of defence for pruning. * Short‑key (prefix) index Sparse binary‑search table of the first 36 bytes of the sort key every ~1 K rows—enables millisecond point/range seeks. * Footer & magic number Offsets to every index and a checksum for integrity; lets StarRocks memory‑map just the tail to discover the rest. Because the pages are already sorted by the key, those indexes are tiny yet brutally effective. 4. Read Path 1. Partition pruning (planner‑time) If the WHERE clause constrains the partition key (e.g. `dt BETWEEN '2025‑05‑01' AND '2025‑05‑07'`), the optimizer opens only matching partition directories. 2. Tablet pruning(planner-time) When the equality filter includes the hash distribution column, StarRocks computes the target tablet IDs and schedules just those Tablets. 3. Prefix‑index seek A sparse short‑key index on the leading sort columns homes in on the exact segment or page. 4. Zone‑map pruning min/max metadata per Segment and 64 KB page discards blocks that miss the predicate window. 5. Vectorized scan & late materialization Surviving column pages stream sequentially through CPU caches; only referenced rows\&columns are materialized, keeping memory tight. Because data is committed in key order on every flush, each read‑time pruning layer compounds on the one before it, delivering sub‑second scans on multi‑billion‑row tables. *** #### How to Choose an Effective Sort Key[​](#how-to-choose-an-effective-sort-key "Direct link to How to Choose an Effective Sort Key") 1. Start with Workload Intelligence Analyse the top‑N query patterns first: * Equality predicates (`=` / `IN`). Columns almost always filtered by equality make ideal leading candidates. * Range predicates. Timestamps and numeric ranges typically follow equality columns in the sort key. * Aggregation keys. If a range column also appears in `GROUP BY` clauses, placing it earlier in the key (after selective filters) can enable sorted aggregation. * Join/group-by keys. Consider placing join or grouping keys early if they are common Measure column cardinality: high‑cardinality columns (millions of distinct values) prune best. 2. Heuristics & Rules of Thumb 1. Order rule: (high‑selectivity equality columns) → (primary range column) → (cluster helpers). 2. Cardinality Ordering: Placing low-cardinality columns before high-cardinality columns can enhance data compression. 3. Width: Keep to 3‑5 columns. Very wide keys slow ingest and overflow the 36‑byte prefix‑index limit. 4. String columns: a long leading string column may occupy most or all of the 36-byte limit in the prefix index, preventing subsequent columns in the sort key from being indexed effectively. This reduces the pruning power of the prefix index and degrades point query performance. 3. Coordinate with Other Design Knobs * Partitioning: Choose a partition key that is coarser than the leading sort column (for example, `PARTITION BY date`, `ORDER BY (tenant_id, ts)`). That way partition pruning removes whole date ranges first, and sort pruning cleans up inside. * Bucketing: Using the same columns for both bucketing and clustering serves different purposes. Bucketing ensures even data distribution across the cluster, while sorting enables efficient I/O elimination. * Table type: Primary-Key tables default to using the primary key as the sort key, but they can also specify additional columns to refine physical order and enhance pruning. Aggregate and duplicate tables should follow the analytic predicate-driven sort key strategies discussed above. *** 4. Reference Templates | Scenario | Partition | Sort Key | Rationale | | ----------------- | ----------------------------- | --------------------- | -------------------------------------------------------------------- | | B2C Orders | date\_trunc('day', order\_ts) | (user\_id, order\_ts) | Most queries filter by user first, then recent time ranges. | | IoT Telemetry | date\_trunc('day', ts) | (device\_id, ts) | Device‑scoped time‑series reads dominate. | | SaaS Multi‑Tenant | tenant\_id | (dt, event\_id) | Tenant isolation via partition; sort clusters by day for dashboards. | | Dimension Lookup | none | (dim\_id) | Small table, pure point look‑ups—single‑column suffices. | *** #### Conclusion[​](#conclusion "Direct link to Conclusion") A well‑designed sort key trades a small, predictable ingest overhead for dramatic improvements in scan latency, storage efficiency, and CPU utilisation. By grounding your choice in workload realities, respecting cardinality, and validating with `EXPLAIN`, you can keep StarRocks humming even as data and user counts grow 10× and beyond. --- ## Data_source ### Block cache warmup Some data lake analytics and shared-data cluster scenarios have high performance requirements for queries, such as BI reports and proof of concept (PoC) performance testing. Loading remote data into local block cache can avoid the need to fetch the same data multiple times, significantly speeding up query execution and minimizing resource usage. StarRocks v3.3 introduces the Block Cache Warmup feature, which is an enhancement to [Block Cache](https://docs.starrocks.io/docs/data_source/data_cache.md#principles-of-block-cache). Block Cache is a process of passively populating the cache, in which data is written to the cache during data querying. Block Cache Warmup, however, is an active process of populating the cache. It proactively fetches the desired data from remote storage in advance. #### Scenarios[​](#scenarios "Direct link to Scenarios") * The disk used for block cache has a storage capacity much larger than the amount of data to warm up. If the disk capacity is less than the data to warm up, the expected warmup effect cannot be achieved. For example, if 100 GB data needs to be warmed up but the disk has only 50 GB of space, then only 50 GB data can be loaded to the cache and the previously loaded 50 GB data will be replaced by the 50 GB data that is loaded later. * Data access on the disk used for block cache is relatively stable. If there is a surge in the access volume, the expected warmup effect cannot be achieved. For example, if 100 GB data needs to be warmed up and the disk has 200 GB of space, then the first condition is met. However, if a large amount of new data (150 GB) is written to the cache during the warmup process, or if an unexpected large cold query needs to load 150 GB data to the cache, it may result in the eviction of the warmed data. #### How it works[​](#how-it-works "Direct link to How it works") StarRocks provides the CACHE SELECT syntax to implement Block Cache Warmup. Before using CACHE SELECT, make sure that the Block Cache feature has been enabled. Syntax of CACHE SELECT: ```sql CACHE SELECT [, ...] FROM [.][.] [WHERE ] [PROPERTIES("verbose"="true")] ``` Parameters: * `column_name`: The columns to fetch. You can use `*` to fetch all columns in the external table. * `catalog_name`: The name of the catalog, default is DEFAULT\_CATALOG. If you have switched to the catalog using SET CATALOG, it can be left unspecified. * `db_name`: The name of the database. If you have switched to that database, it can be left unspecified. * `table_name`: The name of the table from which to fetch data. * `boolean_expression`: The filter condition. * `PROPERTIES`: Currently, only the `verbose` property is supported. It is used to return detailed warmup metrics. CACHE SELECT is a synchronous process and it can warm up only one table at a time. Upon successful execution, it will return warmup-related metrics. ##### Warm up all data in an external table[​](#warm-up-all-data-in-an-external-table "Direct link to Warm up all data in an external table") The following example loads all data from external table `lineitem`: ```plaintext mysql> cache select * from hive_catalog.test_db.lineitem; +-----------------+------------------+----------------------+-------------------+ | READ_CACHE_SIZE | WRITE_CACHE_SIZE | AVG_WRITE_CACHE_TIME | TOTAL_CACHE_USAGE | +-----------------+------------------+----------------------+-------------------+ | 48.2MB | 3.7GB | 59ms | 96.83% | +-----------------+------------------+----------------------+-------------------+ 1 row in set (19.56 sec) ``` Return fields: * `READ_CACHE_SIZE`: The total size of data read from the block cache by all nodes. * `WRITE_CACHE_SIZE`: The total size of data written to the block cache by all nodes. * `AVG_WRITE_CACHE_TIME`: The average time taken by each node to write data to the block cache. * `TOTAL_CACHE_USAGE`: The disk space usage of the block cache of the entire cluster after this warmup task is complete. This metric can be used to assess whether the block cache has sufficient space. ##### Warm up specified columns with filter conditions[​](#warm-up-specified-columns-with-filter-conditions "Direct link to Warm up specified columns with filter conditions") You can specify columns and predicates to achieve fine-grained warmup, which helps reduce the amount to data to warm up, reducing disk I/O and CPU consumption. ```plaintext mysql> cache select l_orderkey from hive_catalog.test_db.lineitem where l_shipdate='1994-10-28'; +-----------------+------------------+----------------------+-------------------+ | READ_CACHE_SIZE | WRITE_CACHE_SIZE | AVG_WRITE_CACHE_TIME | TOTAL_CACHE_USAGE | +-----------------+------------------+----------------------+-------------------+ | 957MB | 713.5MB | 3.6ms | 97.33% | +-----------------+------------------+----------------------+-------------------+ 1 row in set (9.07 sec) ``` The following example prefetches a specific column from a cloud-native table `lineorder` in a shared-data cluster: ```plaintext mysql> cache select lo_orderkey from ssb.lineorder; +-----------------+------------------+----------------------+-------------------+ | READ_CACHE_SIZE | WRITE_CACHE_SIZE | AVG_WRITE_CACHE_TIME | TOTAL_CACHE_USAGE | +-----------------+------------------+----------------------+-------------------+ | 118MB | 558.9MB | 200.6ms | 4.66% | +-----------------+------------------+----------------------+-------------------+ 1 row in set (29.88 sec) ``` ##### Warm up in verbose mode[​](#warm-up-in-verbose-mode "Direct link to Warm up in verbose mode") By default, the metrics returned by `CACHE SELECT` are metrics combined on multiple BEs. You can append `PROPERTIES("verbose"="true")` at the end of CACHE SELECT to obtain detailed metrics of each BE. ```plaintext mysql> cache select * from hive_catalog.test_db.lineitem properties("verbose"="true"); +---------------+-----------------+---------------------+------------------+----------------------+-------------------+ | IP | READ_CACHE_SIZE | AVG_READ_CACHE_TIME | WRITE_CACHE_SIZE | AVG_WRITE_CACHE_TIME | TOTAL_CACHE_USAGE | +---------------+-----------------+---------------------+------------------+----------------------+-------------------+ | 172.26.80.233 | 376MB | 127.8micros | 0B | 0s | 3.85% | | 172.26.80.231 | 272.5MB | 121.8micros | 20.7MB | 146.5micros | 3.91% | | 172.26.80.232 | 355.5MB | 147.7micros | 0B | 0s | 3.91% | +---------------+-----------------+---------------------+------------------+----------------------+-------------------+ 3 rows in set (0.54 sec) ``` In verbose mode, an extra metric will be returned: * `AVG_READ_CACHE_TIME`: the average time for each node to read data when block cache is hit. #### Periodic scheduling of CACHE SELECT tasks[​](#periodic-scheduling-of-cache-select-tasks "Direct link to Periodic scheduling of CACHE SELECT tasks") You can use CACHE SELECT with [SUBMIT TASK](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/ETL/SUBMIT_TASK.md) to achieve periodic warmup. For example, the following case warms up the `lineitem` table every 5 minutes: ```plaintext mysql> submit task always_cache schedule every(interval 5 minute) as cache select l_orderkey from hive_catalog.test_db.lineitem where l_shipdate='1994-10-28'; +--------------+-----------+ | TaskName | Status | +--------------+-----------+ | always_cache | SUBMITTED | +--------------+-----------+ 1 row in set (0.03 sec) ``` ##### Manage CACHE SELECT tasks[​](#manage-cache-select-tasks "Direct link to Manage CACHE SELECT tasks") ###### View created tasks[​](#view-created-tasks "Direct link to View created tasks") ```plaintext mysql> select * from default_catalog.information_schema.tasks; +--------------+---------------------+-----------------------------------------------------+---------------+------------------------------+---------------------------------------------------------------------+---------------------+------------+ | TASK_NAME | CREATE_TIME | SCHEDULE | CATALOG | DATABASE | DEFINITION | EXPIRE_TIME | PROPERTIES | +--------------+---------------------+-----------------------------------------------------+---------------+------------------------------+---------------------------------------------------------------------+---------------------+------------+ | always_cache | 2024-04-11 16:01:00 | PERIODICAL START(2024-04-11T16:01) EVERY(5 MINUTES) | emr_hive_test | zz_tpch_sf1000_hive_orc_zlib | cache select l_orderkey from lineitem where l_shipdate='1994-10-28' | NULL | | +--------------+---------------------+-----------------------------------------------------+---------------+------------------------------+---------------------------------------------------------------------+---------------------+------------+ 1 row in set (0.21 sec) ``` ###### View task execution history[​](#view-task-execution-history "Direct link to View task execution history") ```plaintext mysql> select * from default_catalog.information_schema.task_runs; +--------------------------------------+--------------+---------------------+---------------------+---------+---------------+------------------------------+---------------------------------------------------------------------+---------------------+------------+---------------+----------+------------------------------------------------------------------------------------------------------------------------+------------+ | QUERY_ID | TASK_NAME | CREATE_TIME | FINISH_TIME | STATE | CATALOG | DATABASE | DEFINITION | EXPIRE_TIME | ERROR_CODE | ERROR_MESSAGE | PROGRESS | EXTRA_MESSAGE | PROPERTIES | +--------------------------------------+--------------+---------------------+---------------------+---------+---------------+------------------------------+---------------------------------------------------------------------+---------------------+------------+---------------+----------+------------------------------------------------------------------------------------------------------------------------+------------+ | 55b30204-f7da-11ee-b03e-7ea526d0b618 | always_cache | 2024-04-11 16:06:00 | 2024-04-11 16:07:22 | SUCCESS | emr_hive_test | zz_tpch_sf1000_hive_orc_zlib | cache select l_orderkey from lineitem where l_shipdate='1994-10-28' | 2024-04-12 16:06:00 | 0 | NULL | 100% | AlreadyCachedSize: 15.7GB, AvgReadCacheTime: 1ms, WriteCacheSize: 0B, AvgWriteCacheTime: 0s, TotalCacheUsage: 75.94% | | | a2e3dc7e-f7d9-11ee-b03e-7ea526d0b618 | always_cache | 2024-04-11 16:01:00 | 2024-04-11 16:02:39 | SUCCESS | emr_hive_test | zz_tpch_sf1000_hive_orc_zlib | cache select l_orderkey from lineitem where l_shipdate='1994-10-28' | 2024-04-12 16:01:00 | 0 | NULL | 100% | AlreadyCachedSize: 15.7GB, AvgReadCacheTime: 1.2ms, WriteCacheSize: 0B, AvgWriteCacheTime: 0s, TotalCacheUsage: 75.87% | | +--------------------------------------+--------------+---------------------+---------------------+---------+---------------+------------------------------+---------------------------------------------------------------------+---------------------+------------+---------------+----------+------------------------------------------------------------------------------------------------------------------------+------------+ 2 rows in set (0.04 sec) ``` The `EXTRA_MESSAGE` field records metrics of CACHE SELECT. ###### Drop tasks[​](#drop-tasks "Direct link to Drop tasks") ```sql DROP TASK ``` #### Use cases[​](#use-cases "Direct link to Use cases") 1. During PoC performance testing, if you want to assess StarRocks' performance without interference from external storage systems, you can use the CACHE SELECT statement to load the data of the table to test into the block cache in advance. 2. The business team need to view BI reports at 8 a.m. every morning. To ensure a relatively stable query performance, you can schedule a CACHE SELECT task to start running at 7 a.m. each day. ```sql mysql> submit task BI schedule START('2024-02-03 07:00:00') EVERY(interval 1 day) AS cache select * from hive_catalog.test_db.lineitem where l_shipdate='1994-10-28'; +--------------+-----------+ | TaskName | Status | +--------------+-----------+ | BI | SUBMITTED | +--------------+-----------+ 1 row in set (0.03 sec) ``` 3. To minimize system resource consumption during warmup, you can specify session variables in the SUBMIT TASK statement. For example, you can designate a resource group for the CACHE SELECT task, adjust the Degree of Parallelism (DOP), and specify the filter condition in WHERE to reduce the impact of warmup on regular queries. ```sql mysql> submit task cache_select properties("pipeline_dop"="1", "resource_group"="warmup") schedule EVERY(interval 1 day) AS cache select * from hive_catalog.test_db.lineitem where l_shipdate>='1994-10-28'; +--------------+-----------+ | TaskName | Status | +--------------+-----------+ | cache_select | SUBMITTED | +--------------+-----------+ 1 row in set (0.03 sec) ``` #### Limits and usage notes[​](#limits-and-usage-notes "Direct link to Limits and usage notes") * To use CACHE SELECT, you must first enable the Block Cache feature and have the SELECT privilege on the destination table. * CACHE SELECT supports warming up only a single table and does not support operators like ORDER BY, LIMIT, or GROUP BY. * CACHE SELECT can be used in both shared-nothing and shared-data clusters. * CACHE SELECT can warm up remote TEXT, ORC, Parquet files. * The data warmed up by CACHE SELECT may not be retained in cache forever. The cached data may still be evicted based on the SLRU rule of the Block Cache feature. * If you are a data lake user, you can check the remaining capacity of the block cache by using `SHOW BACKENDS\G` or `SHOW COMPUTE NODES\G` to assess whether SLRU eviction may occur. * If you are a shared-data cluster user, you can check the block cache usage by viewing the metrics of the shared-data cluster. * Currently, the implementation of CACHE SELECT uses the INSERT INTO BLACKHOLE() approach, which warms up the table following the normal query process. Therefore, the performance overhead of CACHE SELECT is similar to that of regular queries. Improvements will be made in the future to enhance the performance. #### What to expect in later versions[​](#what-to-expect-in-later-versions "Direct link to What to expect in later versions") In the future, StarRocks will introduce adaptive Block Cache Warmup to ensure a higher cache hit rate. --- ### Overview This topic describes what a catalog is, and how to manage and query internal data and external data by using a catalog. StarRocks supports the catalog feature from v2.3 onwards. Catalogs enable you to manage internal and external data in one system and offer a flexible way for you to easily query and analyze data that is stored in various external systems. #### Basic concepts[​](#basic-concepts "Direct link to Basic concepts") * **Internal data**: refers to the data stored in StarRocks. * **External data**: refers to the data stored in an external data source, such as Apache Hive™, Apache Iceberg, Apache Hudi, Delta Lake, and JDBC. #### Catalog[​](#catalog "Direct link to Catalog") Currently, StarRocks provides two types of catalogs: internal catalog and external catalog. ![figure1](/assets/images/3.8.1-e149861830788363c2a5fc3b7ddd4291.png) * **Internal catalog** manages internal data of StarRocks. For example, if you execute the CREATE DATABASE or CREATE TABLE statements to create a database or a table, the database or table is stored in the internal catalog. Each StarRocks cluster has only one internal catalog named [default\_catalog](https://docs.starrocks.io/docs/data_source/catalog/default_catalog.md). * **External catalog** acts like a link to externally managed metastores, which grants StarRocks direct access to external data sources. You can query external data directly with zero data loading or migration. Currently, StarRocks supports the following types of external catalogs: * [Hive catalog](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md): used to query data from Hive. * [Iceberg catalog](https://docs.starrocks.io/docs/data_source/catalog/iceberg/iceberg_catalog.md): used to query data from Iceberg. * [Hudi catalog](https://docs.starrocks.io/docs/data_source/catalog/hudi_catalog.md): used to query data from Hudi. * [Delta Lake catalog](https://docs.starrocks.io/docs/data_source/catalog/deltalake_catalog.md): used to query data from Delta Lake. * [JDBC catalog](https://docs.starrocks.io/docs/data_source/catalog/jdbc_catalog.md): used to query data from JDBC-compatible data sources. * [Elasticsearch catalog](https://docs.starrocks.io/docs/data_source/catalog/elasticsearch_catalog.md): used to query data from Elasticsearch. Elasticsearch catalogs are supported from v3.1 onwards. * [Paimon catalog](https://docs.starrocks.io/docs/data_source/catalog/paimon_catalog.md): used to query data from Paimon. Paimon catalogs are supported from v3.1 onwards. * [Unified catalog](https://docs.starrocks.io/docs/data_source/catalog/unified_catalog.md): used to query data from from Hive, Iceberg, Hudi, and Delta Lake data sources as a unified data source. Unified catalogs are supported from v3.2 onwards. StarRocks interacts with the following two components of external data sources when you query external data: * **Metastore service**: used by the FEs to access the metadata of external data sources. The FEs generate a query execution plan based on the metadata. * **Data storage system**: used to store external data. Both distributed file systems and object storage systems can be used as data storage systems to store data files in various formats. After the FEs distribute the query execution plan to all BEs or CNs, all BEs or CNs scan the target external data in parallel, perform calculations, and then return the query result. #### Access catalog[​](#access-catalog "Direct link to Access catalog") You can use the [SET CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SET_CATALOG.md) statement to switch to a specified catalog in the current session. Then, you can query data by using that catalog. #### Query data[​](#query-data "Direct link to Query data") ##### Query internal data[​](#query-internal-data "Direct link to Query internal data") To query data in StarRocks, see [Default catalog](https://docs.starrocks.io/docs/data_source/catalog/default_catalog.md). ##### Query external data[​](#query-external-data "Direct link to Query external data") To query data from external data sources, see [Query external data](https://docs.starrocks.io/docs/data_source/catalog/query_external_data.md). ##### Cross-catalog query[​](#cross-catalog-query "Direct link to Cross-catalog query") To perform a cross-catalog federated query from your current catalog, specify the data you want to query in the `catalog_name.database_name` or `catalog_name.database_name.table_name` format. * Query `hive_table` in `hive_db` when the current session is `default_catalog.olap_db`. ```sql SELECT * FROM hive_catalog.hive_db.hive_table; ``` * Query `olap_table` in `default_catalog` when the current session is `hive_catalog.hive_db`. ```sql SELECT * FROM default_catalog.olap_db.olap_table; ``` * Perform a JOIN query on `hive_table` in `hive_catalog` and `olap_table` in `default_catalog` when the current session is `hive_catalog.hive_db`. ```sql SELECT * FROM hive_table h JOIN default_catalog.olap_db.olap_table o WHERE h.id = o.id; ``` * Perform a JOIN query on `hive_table` in `hive_catalog` and `olap_table` in `default_catalog` by using a JOIN clause when the current session is another catalog. ```sql SELECT * FROM hive_catalog.hive_db.hive_table h JOIN default_catalog.olap_db.olap_table o WHERE h.id = o.id; ``` --- ### Default catalog This topic describes what the default catalog is, and how to query the internal data of StarRocks by using the default catalog. StarRocks 2.3 and later provide an internal catalog to manage the internal data of StarRocks. Each StarRocks cluster has only one internal catalog named `default_catalog`. Currently, you cannot modify the name of the internal catalog or create a new internal catalog. #### Query internal data[​](#query-internal-data "Direct link to Query internal data") 1. Connect your StarRocks cluster. * If you use the MySQL client to connect the StarRocks cluster, you go to `default_catalog` by default after connecting. * If you use JDBC to connect the StarRocks cluster, you can go directly to the destination database in the default catalog by specifying `default_catalog.db_name` when connecting. 2. (Optional) Use [SHOW DATABASES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATABASES.md) to view databases: ```sql SHOW DATABASES; ``` Or ```sql SHOW DATABASES FROM ; ``` 3. (Optional) Use [SET CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SET_CATALOG.md) to switch to the destination catalog in the current session: ```sql SET CATALOG ; ``` Then, use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to specify the active database in the current session: ```sql USE ; ``` Or, you can use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to directly go to the active database in the destination catalog: ```sql USE .; ``` 4. Use [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md) to query internal data: ```sql SELECT * FROM ; ``` If you do not specify the active database in the preceding steps, you can directly specify it in a select query: ```sql SELECT * FROM .; ``` Or ```sql SELECT * FROM default_catalog..; ``` #### Examples[​](#examples "Direct link to Examples") To query data in `olap_db.olap_table`, you can perform one of the following operations: ```sql USE olap_db; SELECT * FROM olap_table limit 1; ``` Or ```sql SELECT * FROM olap_db.olap_table limit 1; ``` Or ```sql SELECT * FROM default_catalog.olap_db.olap_table limit 1; ``` #### References[​](#references "Direct link to References") To query data from external data sources, see [Query external data](https://docs.starrocks.io/docs/data_source/catalog/query_external_data.md). --- ### Delta Lake catalog A Delta Lake catalog is a kind of external catalog that enables you to query data from Delta Lake without ingestion. Also, you can directly transform and load data from Delta Lake by using [INSERT INTO](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) based on Delta Lake catalogs. StarRocks supports Delta Lake catalogs from v2.5 onwards. To ensure successful SQL workloads on your Delta Lake cluster, your StarRocks cluster must be able to access the storage system and metastore of your Delta Lake cluster. StarRocks supports the following storage systems and metastores: * Distributed file system (HDFS) or object storage like AWS S3, Microsoft Azure Storage, Google GCS, or other S3-compatible storage system (for example, MinIO) * Metastore like Hive metastore or AWS Glue note If you choose AWS S3 as storage, you can use HMS or AWS Glue as metastore. If you choose any other storage system, you can only use HMS as metastore. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * The file format of Delta Lake that StarRocks supports is Parquet. Parquet files support the following compression formats: SNAPPY, LZ4, ZSTD, GZIP, and NO\_COMPRESSION. * The data types of Delta Lake that StarRocks does not support are MAP and STRUCT. #### Integration preparations[​](#integration-preparations "Direct link to Integration preparations") Before you create a Delta Lake catalog, make sure your StarRocks cluster can integrate with the storage system and metastore of your Delta Lake cluster. ##### AWS IAM[​](#aws-iam "Direct link to AWS IAM") If your Delta Lake cluster uses AWS S3 as storage or AWS Glue as metastore, choose your suitable authentication method and make the required preparations to ensure that your StarRocks cluster can access the related AWS cloud resources. The following authentication methods are recommended: * Instance profile * Assumed role * IAM user Of the above-mentioned three authentication methods, instance profile is the most widely used. For more information, see [Preparation for authentication in AWS IAM](https://docs.starrocks.io/docs/integrations/authenticate_to_aws_resources.md#preparation-for-iam-user-based-authentication). ##### HDFS[​](#hdfs "Direct link to HDFS") If you choose HDFS as storage, configure your StarRocks cluster as follows: * (Optional) Set the username that is used to access your HDFS cluster and Hive metastore. By default, StarRocks uses the username of the FE and BE or CN processes to access your HDFS cluster and Hive metastore. You can also set the username by adding `export HADOOP_USER_NAME=""` at the beginning of the **fe/conf/hadoop\_env.sh** file of each FE and at the beginning of the **be/conf/hadoop\_env.sh** file of each BE or the **cn/conf/hadoop\_env.sh** file of each CN. After you set the username in these files, restart each FE and each BE or CN to make the parameter settings take effect. You can set only one username for each StarRocks cluster. * When you query Delta Lake data, the FEs and BEs or CNs of your StarRocks cluster use the HDFS client to access your HDFS cluster. In most cases, you do not need to configure your StarRocks cluster to achieve that purpose, and StarRocks starts the HDFS client using the default configurations. You need to configure your StarRocks cluster only in the following situations: * High availability (HA) is enabled for your HDFS cluster: Add the **hdfs-site.xml** file of your HDFS cluster to the **$FE\_HOME/conf** path of each FE and to the **$BE\_HOME/conf** path of each BE or the **$CN\_HOME/conf** path of each CN. * View File System (ViewFs) is enabled for your HDFS cluster: Add the **core-site.xml** file of your HDFS cluster to the **$FE\_HOME/conf** path of each FE and to the **$BE\_HOME/conf** path of each BE or the **$CN\_HOME/conf** path of each CN. note If an error indicating an unknown host is returned when you send a query, you must add the mapping between the host names and IP addresses of your HDFS cluster nodes to the **/etc/hosts** path. ##### Kerberos authentication[​](#kerberos-authentication "Direct link to Kerberos authentication") If Kerberos authentication is enabled for your HDFS cluster or Hive metastore, configure your StarRocks cluster as follows: * Run the `kinit -kt keytab_path principal` command on each FE and each BE or CN to obtain Ticket Granting Ticket (TGT) from Key Distribution Center (KDC). To run this command, you must have the permissions to access your HDFS cluster and Hive metastore. Note that accessing KDC with this command is time-sensitive. Therefore, you need to use cron to run this command periodically. * Add `JAVA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf"` to the **$FE\_HOME/conf/fe.conf** file of each FE and to the **$BE\_HOME/conf/be.conf** file of each BE or the **$CN\_HOME/conf/cn.conf** file of each CN. In this example, `/etc/krb5.conf` is the save path of the **krb5.conf** file. You can modify the path based on your needs. #### Create a Delta Lake catalog[​](#create-a-delta-lake-catalog "Direct link to Create a Delta Lake catalog") ##### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE EXTERNAL CATALOG [COMMENT ] PROPERTIES ( "type" = "deltalake", MetastoreParams, StorageCredentialParams, MetadataUpdateParams ) ``` ##### Parameters[​](#parameters "Direct link to Parameters") ###### catalog\_name[​](#catalog_name "Direct link to catalog_name") The name of the Delta Lake catalog. The naming conventions are as follows: * The name can contain letters, digits (0-9), and underscores (\_). It must start with a letter. * The name is case-sensitive and cannot exceed 1023 characters in length. ###### comment[​](#comment "Direct link to comment") The description of the Delta Lake catalog. This parameter is optional. ###### type[​](#type "Direct link to type") The type of your data source. Set the value to `deltalake`. ###### MetastoreParams[​](#metastoreparams "Direct link to MetastoreParams") A set of parameters about how StarRocks integrates with the metastore of your data source. ###### Hive metastore[​](#hive-metastore "Direct link to Hive metastore") If you choose Hive metastore as the metastore of your data source, configure `MetastoreParams` as follows: ```sql "hive.metastore.type" = "hive", "hive.metastore.uris" = "" ``` note Before querying Delta Lake data, you must add the mapping between the host names and IP addresses of your Hive metastore nodes to the `/etc/hosts` path. Otherwise, StarRocks may fail to access your Hive metastore when you start a query. The following table describes the parameter you need to configure in `MetastoreParams`. | Parameter | Required | Description | | ------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | hive.metastore.type | Yes | The type of metastore that you use for your Delta Lake cluster. Set the value to `hive`. | | hive.metastore.uris | Yes | The URI of your Hive metastore. Format: `thrift://:`.
If high availability (HA) is enabled for your Hive metastore, you can specify multiple metastore URIs and separate them with commas (`,`), for example, `"thrift://:,thrift://:,thrift://:"`. | ###### AWS Glue[​](#aws-glue "Direct link to AWS Glue") If you choose AWS Glue as the metastore of your data source, which is supported only when you choose AWS S3 as storage, take one of the following actions: * To choose the instance profile-based authentication method, configure `MetastoreParams` as follows: ```sql "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.region" = "" ``` * To choose the assumed role-based authentication method, configure `MetastoreParams` as follows: ```sql "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.iam_role_arn" = "", "aws.glue.region" = "" ``` * To choose the IAM user-based authentication method, configure `MetastoreParams` as follows: ```sql "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "false", "aws.glue.access_key" = "", "aws.glue.secret_key" = "", "aws.glue.region" = "" ``` The following table describes the parameters you need to configure in `MetastoreParams`. | Parameter | Required | Description | | ------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | hive.metastore.type | Yes | The type of metastore that you use for your Delta Lake cluster. Set the value to `glue`. | | aws.glue.use\_instance\_profile | Yes | Specifies whether to enable the instance profile-based authentication method and the assumed role-based authentication method. Valid values: `true` and `false`. Default value: `false`. | | aws.glue.iam\_role\_arn | No | The ARN of the IAM role that has privileges on your AWS Glue Data Catalog. If you use the assumed role-based authentication method to access AWS Glue, you must specify this parameter. | | aws.glue.region | Yes | The region in which your AWS Glue Data Catalog resides. Example: `us-west-1`. | | aws.glue.access\_key | No | The access key of your AWS IAM user. If you use the IAM user-based authentication method to access AWS Glue, you must specify this parameter. | | aws.glue.secret\_key | No | The secret key of your AWS IAM user. If you use the IAM user-based authentication method to access AWS Glue, you must specify this parameter. | For information about how to choose an authentication method for accessing AWS Glue and how to configure an access control policy in the AWS IAM Console, see [Authentication parameters for accessing AWS Glue](https://docs.starrocks.io/docs/integrations/authenticate_to_aws_resources.md#authentication-parameters-for-accessing-aws-glue). ###### StorageCredentialParams[​](#storagecredentialparams "Direct link to StorageCredentialParams") A set of parameters about how StarRocks integrates with your storage system. This parameter set is optional. If you use HDFS as storage, you do not need to configure `StorageCredentialParams`. If you use AWS S3, other S3-compatible storage system, Microsoft Azure Storage, or Google GCS as storage, you must configure `StorageCredentialParams`. ###### AWS S3[​](#aws-s3 "Direct link to AWS S3") If you choose AWS S3 as storage for your Delta Lake cluster, take one of the following actions: * To choose the instance profile-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "" ``` * To choose the assumed role-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "", "aws.s3.region" = "" ``` * To choose the IAM user-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ----------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | aws.s3.use\_instance\_profile | Yes | Specifies whether to enable the instance profile-based authentication method and the assumed role-based authentication method. Valid values: `true` and `false`. Default value: `false`. | | aws.s3.iam\_role\_arn | No | The ARN of the IAM role that has privileges on your AWS S3 bucket. If you use the assumed role-based authentication method to access AWS S3, you must specify this parameter. | | aws.s3.region | Yes | The region in which your AWS S3 bucket resides. Example: `us-west-1`. | | aws.s3.access\_key | No | The access key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter. | | aws.s3.secret\_key | No | The secret key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter. | For information about how to choose an authentication method for accessing AWS S3 and how to configure an access control policy in AWS IAM Console, see [Authentication parameters for accessing AWS S3](https://docs.starrocks.io/docs/integrations/authenticate_to_aws_resources.md#authentication-parameters-for-accessing-aws-s3). ###### S3-compatible storage system[​](#s3-compatible-storage-system "Direct link to S3-compatible storage system") Delta Lake catalogs support S3-compatible storage systems from v2.5 onwards. If you choose an S3-compatible storage system, such as MinIO, as storage for your Delta Lake cluster, configure `StorageCredentialParams` as follows to ensure a successful integration: ```sql "aws.s3.enable_ssl" = "false", "aws.s3.enable_path_style_access" = "true", "aws.s3.endpoint" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ---------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | aws.s3.enable\_ssl | Yes | Specifies whether to enable SSL connection.
Valid values: `true` and `false`. Default value: `true`. | | aws.s3.enable\_path\_style\_access | Yes | Specifies whether to enable path-style access.
Valid values: `true` and `false`. Default value: `false`. For MinIO, you must set the value to `true`.
Path-style URLs use the following format: `https://s3..amazonaws.com//`. For example, if you create a bucket named `DOC-EXAMPLE-BUCKET1` in the US West (Oregon) Region, and you want to access the `alice.jpg` object in that bucket, you can use the following path-style URL: `https://s3.us-west-2.amazonaws.com/DOC-EXAMPLE-BUCKET1/alice.jpg`. | | aws.s3.endpoint | Yes | The endpoint that is used to connect to your S3-compatible storage system instead of AWS S3. | | aws.s3.access\_key | Yes | The access key of your IAM user. | | aws.s3.secret\_key | Yes | The secret key of your IAM user. | ###### Microsoft Azure Storage[​](#microsoft-azure-storage "Direct link to Microsoft Azure Storage") Delta Lake catalogs support Microsoft Azure Storage from v3.0 onwards. ###### Azure Blob Storage[​](#azure-blob-storage "Direct link to Azure Blob Storage") If you choose Blob Storage as storage for your Delta Lake cluster, take one of the following actions: * To choose the Shared Key authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.blob.storage_account" = "", "azure.blob.shared_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | --------------------------- | ------------ | -------------------------------------------- | | azure.blob.storage\_account | Yes | The username of your Blob Storage account. | | azure.blob.shared\_key | Yes | The shared key of your Blob Storage account. | * To choose the SAS Token authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.blob.storage_account" = "", "azure.blob.container" = "", "azure.blob.sas_token" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | --------------------------- | ------------ | --------------------------------------------------------------- | | azure.blob.storage\_account | Yes | The username of your Blob Storage account. | | azure.blob.container | Yes | The name of the blob container that stores your data. | | azure.blob.sas\_token | Yes | The SAS token that is used to access your Blob Storage account. | ###### Azure Data Lake Storage Gen2[​](#azure-data-lake-storage-gen2 "Direct link to Azure Data Lake Storage Gen2") If you choose Data Lake Storage Gen2 as storage for your Delta Lake cluster, take one of the following actions: * To choose the Managed Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------------------ | ------------ | ------------------------------------------------------------------------------------------------ | | azure.adls2.oauth2\_use\_managed\_identity | Yes | Specifies whether to enable the Managed Identity authentication method. Set the value to `true`. | | azure.adls2.oauth2\_tenant\_id | Yes | The ID of the tenant whose data you want to access. | | azure.adls2.oauth2\_client\_id | Yes | The client (application) ID of the managed identity. | * To choose the Shared Key authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.storage_account" = "", "azure.adls2.shared_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ---------------------------- | ------------ | -------------------------------------------------------------- | | azure.adls2.storage\_account | Yes | The username of your Data Lake Storage Gen2 storage account. | | azure.adls2.shared\_key | Yes | The shared key of your Data Lake Storage Gen2 storage account. | * To choose the Service Principal authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_client_secret" = "", "azure.adls2.oauth2_client_endpoint" = "" ``` The following table describes the parameters you need to configure `in StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------------ | ------------ | -------------------------------------------------------------------------- | | azure.adls2.oauth2\_client\_id | Yes | The client (application) ID of the service principal. | | azure.adls2.oauth2\_client\_secret | Yes | The value of the new client (application) secret created. | | azure.adls2.oauth2\_client\_endpoint | Yes | The OAuth 2.0 token endpoint (v1) of the service principal or application. | * To choose the Workload Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_token_file" = "", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | azure.adls2.oauth2\_token\_file | Yes | The absolute file path to the OAuth2 token file projected into the pod by the Azure Workload Identity webhook. | | azure.adls2.oauth2\_tenant\_id | Yes | The ID of the tenant whose data you want to access. | | azure.adls2.oauth2\_client\_id | Yes | The client ID (application ID) of the Azure AD application (user-assigned managed identity or app registration) associated with the workload identity. | ###### Azure Data Lake Storage Gen1[​](#azure-data-lake-storage-gen1 "Direct link to Azure Data Lake Storage Gen1") If you choose Data Lake Storage Gen1 as storage for your Delta Lake cluster, take one of the following actions: * To choose the Managed Service Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls1.use_managed_service_identity" = "true" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------------------- | ------------ | -------------------------------------------------------------------------------------------------------- | | azure.adls1.use\_managed\_service\_identity | Yes | Specifies whether to enable the Managed Service Identity authentication method. Set the value to `true`. | * To choose the Service Principal authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls1.oauth2_client_id" = "", "azure.adls1.oauth2_credential" = "", "azure.adls1.oauth2_endpoint" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------ | ------------ | -------------------------------------------------------------------------- | | azure.adls1.oauth2\_client\_id | Yes | The client (application) ID of the service principal. | | azure.adls1.oauth2\_credential | Yes | The value of the new client (application) secret created. | | azure.adls1.oauth2\_endpoint | Yes | The OAuth 2.0 token endpoint (v1) of the service principal or application. | ###### Google GCS[​](#google-gcs "Direct link to Google GCS") Delta Lake catalogs support Google GCS from v3.0 onwards. If you choose Google GCS as storage for your Delta Lake cluster, take one of the following actions: * To choose the VM-based authentication method, configure `StorageCredentialParams` as follows: ```sql "gcp.gcs.use_compute_engine_service_account" = "true" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value example** | **Description** | | ---------------------------------------------- | ----------------- | ----------------- | ------------------------------------------------------------------------------------------- | | gcp.gcs.use\_compute\_engine\_service\_account | false | true | Specifies whether to directly use the service account that is bound to your Compute Engine. | * To choose the service account-based authentication method, configure `StorageCredentialParams` as follows: ```sql "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "", ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value example** | **Description** | | ------------------------------------------ | ----------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------- | | gcp.gcs.service\_account\_email | "" | `"user@hello.iam.gserviceaccount.com"` | The email address in the JSON file generated at the creation of the service account. | | gcp.gcs.service\_account\_private\_key\_id | "" | "61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea" | The private key ID in the JSON file generated at the creation of the service account. | | gcp.gcs.service\_account\_private\_key | "" | "-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n" | The private key in the JSON file generated at the creation of the service account. | * To choose the impersonation-based authentication method, configure `StorageCredentialParams` as follows: * Make a VM instance impersonate a service account: ```sql "gcp.gcs.use_compute_engine_service_account" = "true", "gcp.gcs.impersonation_service_account" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value example** | **Description** | | ---------------------------------------------- | ----------------- | ----------------- | ------------------------------------------------------------------------------------------- | | gcp.gcs.use\_compute\_engine\_service\_account | false | true | Specifies whether to directly use the service account that is bound to your Compute Engine. | | gcp.gcs.impersonation\_service\_account | "" | "hello" | The service account that you want to impersonate. | * Make a service account (temporarily named as meta service account) impersonate another service account (temporarily named as data service account): ```sql "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "", "gcp.gcs.impersonation_service_account" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value example** | **Description** | | ------------------------------------------ | ----------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------ | | gcp.gcs.service\_account\_email | "" | `"user@hello.iam.gserviceaccount.com"` | The email address in the JSON file generated at the creation of the meta service account. | | gcp.gcs.service\_account\_private\_key\_id | "" | "61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea" | The private key ID in the JSON file generated at the creation of the meta service account. | | gcp.gcs.service\_account\_private\_key | "" | "-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n" | The private key in the JSON file generated at the creation of the meta service account. | | gcp.gcs.impersonation\_service\_account | "" | "hello" | The data service account that you want to impersonate. | ###### MetadataUpdateParams[​](#metadataupdateparams "Direct link to MetadataUpdateParams") A set of parameters about how StarRocks updates the cached metadata of Delta Lake. This parameter set is optional. From v3.3.3 onwards, Delta Lake Catalog supports [Metadata Local Cache and Retrieval](#appendix-metadata-local-cache-and-retrieval). In most cases, you can ignore `MetadataUpdateParams` and do not need to tune the policy parameters in it, because the default values of these parameters already provide you with an out-of-the-box performance. However, if the frequency of data updates in Delta Lake is high, you can tune these parameters to further optimize the performance of automatic asynchronous updates. note In most cases, if your Delta Lake data is updated at a granularity of 1 hour or less, the data update frequency is considered high. | **Parameter** | **Unit** | **Default** | **Description** | | -------------------------------------------------------- | -------- | -------------- | ----------------------------------------------------------------------------------- | | enable\_deltalake\_table\_cache | - | true | Whether to enable Table Cache in the metadata cache for Delta Lake. | | enable\_deltalake\_json\_meta\_cache | - | true | Whether to enable cache for Delta Log JSON files. | | deltalake\_json\_meta\_cache\_ttl\_sec | Second | 48 \* 60 \* 60 | Time-To-Live (TTL) for the Delta Log JSON file cache. | | deltalake\_json\_meta\_cache\_memory\_usage\_ratio | - | 0.1 | The maximum ratio of JVM Heap size occupied by the Delta Log JSON file cache. | | enable\_deltalake\_checkpoint\_meta\_cache | - | true | Whether to enable cache for Delta Log Checkpoint files. | | deltalake\_checkpoint\_meta\_cache\_ttl\_sec | Second | 48 \* 60 \* 60 | Time-To-Live (TTL) for the Delta Log Checkpoint file cache. | | deltalake\_checkpoint\_meta\_cache\_memory\_usage\_ratio | - | 0.1 | The maximum ratio of JVM Heap size occupied by the Delta Log Checkpoint file cache. | ##### Examples[​](#examples "Direct link to Examples") The following examples create a Delta Lake catalog named `deltalake_catalog_hms` or `deltalake_catalog_glue`, depending on the type of metastore you use, to query data from your Delta Lake cluster. ###### HDFS[​](#hdfs-1 "Direct link to HDFS") If you use HDFS as storage, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083" ); ``` ###### AWS S3[​](#aws-s3-1 "Direct link to AWS S3") ###### If you choose instance profile-based credential[​](#if-you-choose-instance-profile-based-credential "Direct link to If you choose instance profile-based credential") * If you use Hive metastore in your Delta Lake cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "us-west-2" ); ``` * If you use AWS Glue in your Amazon EMR Delta Lake cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_glue PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.region" = "us-west-2", "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "us-west-2" ); ``` ###### If you choose assumed role-based credential[​](#if-you-choose-assumed-role-based-credential "Direct link to If you choose assumed role-based credential") * If you use Hive metastore in your Delta Lake cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::081976408565:role/test_s3_role", "aws.s3.region" = "us-west-2" ); ``` * If you use AWS Glue in your Amazon EMR Delta Lake cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_glue PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.iam_role_arn" = "arn:aws:iam::081976408565:role/test_glue_role", "aws.glue.region" = "us-west-2", "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::081976408565:role/test_s3_role", "aws.s3.region" = "us-west-2" ); ``` ###### If you choose IAM user-based credential[​](#if-you-choose-iam-user-based-credential "Direct link to If you choose IAM user-based credential") * If you use Hive metastore in your Delta Lake cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "us-west-2" ); ``` * If you use AWS Glue in your Amazon EMR Delta Lake cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_glue PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "false", "aws.glue.access_key" = "", "aws.glue.secret_key" = "", "aws.glue.region" = "us-west-2", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "us-west-2" ); ``` ###### S3-compatible storage system[​](#s3-compatible-storage-system-1 "Direct link to S3-compatible storage system") Use MinIO as an example. Run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.enable_ssl" = "true", "aws.s3.enable_path_style_access" = "true", "aws.s3.endpoint" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ); ``` ###### Microsoft Azure Storage[​](#microsoft-azure-storage-1 "Direct link to Microsoft Azure Storage") ###### Azure Blob Storage[​](#azure-blob-storage-1 "Direct link to Azure Blob Storage") * If you choose the Shared Key authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.blob.storage_account" = "", "azure.blob.shared_key" = "" ); ``` * If you choose the SAS Token authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.blob.storage_account" = "", "azure.blob.container" = "", "azure.blob.sas_token" = "" ); ``` ###### Azure Data Lake Storage Gen1[​](#azure-data-lake-storage-gen1-1 "Direct link to Azure Data Lake Storage Gen1") * If you choose the Managed Service Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls1.use_managed_service_identity" = "true" ); ``` * If you choose the Service Principal authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls1.oauth2_client_id" = "", "azure.adls1.oauth2_credential" = "", "azure.adls1.oauth2_endpoint" = "" ); ``` ###### Azure Data Lake Storage Gen2[​](#azure-data-lake-storage-gen2-1 "Direct link to Azure Data Lake Storage Gen2") * If you choose the Managed Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` * If you choose the Shared Key authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.storage_account" = "", "azure.adls2.shared_key" = "" ); ``` * If you choose the Service Principal authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_client_secret" = "", "azure.adls2.oauth2_client_endpoint" = "" ); ``` * If you choose the Workload Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_token_file" = "/var/run/secrets/azure/tokens/azure-identity-token", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` ###### Google GCS[​](#google-gcs-1 "Direct link to Google GCS") * If you choose the VM-based authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.use_compute_engine_service_account" = "true" ); ``` * If you choose the service account-based authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "" ); ``` * If you choose the impersonation-based authentication method: * If you make a VM instance impersonate a service account, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.use_compute_engine_service_account" = "true", "gcp.gcs.impersonation_service_account" = "" ); ``` * If you make a service account impersonate another service account, run a command like below: ```sql CREATE EXTERNAL CATALOG deltalake_catalog_hms PROPERTIES ( "type" = "deltalake", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "", "gcp.gcs.impersonation_service_account" = "" ); ``` #### View Delta Lake catalogs[​](#view-delta-lake-catalogs "Direct link to View Delta Lake catalogs") You can use [SHOW CATALOGS](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CATALOGS.md) to query all catalogs in the current StarRocks cluster: ```sql SHOW CATALOGS; ``` You can also use [SHOW CREATE CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CREATE_CATALOG.md) to query the creation statement of an external catalog. The following example queries the creation statement of a Delta Lake catalog named `deltalake_catalog_glue`: ```sql SHOW CREATE CATALOG deltalake_catalog_glue; ``` #### Switch to a Delta Lake Catalog and a database in it[​](#switch-to-a-delta-lake-catalog-and-a-database-in-it "Direct link to Switch to a Delta Lake Catalog and a database in it") You can use one of the following methods to switch to a Delta Lake catalog and a database in it: * Use [SET CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SET_CATALOG.md) to specify a Delta Lake catalog in the current session, and then use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to specify an active database: ```sql -- Switch to a specified catalog in the current session: SET CATALOG -- Specify the active database in the current session: USE ``` * Directly use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to switch to a Delta Lake catalog and a database in it: ```sql USE . ``` #### Drop a Delta Lake catalog[​](#drop-a-delta-lake-catalog "Direct link to Drop a Delta Lake catalog") You can use [DROP CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/DROP_CATALOG.md) to drop an external catalog. The following example drops a Delta Lake catalog named `deltalake_catalog_glue`: ```sql DROP Catalog deltalake_catalog_glue; ``` #### View the schema of a Delta Lake table[​](#view-the-schema-of-a-delta-lake-table "Direct link to View the schema of a Delta Lake table") You can use one of the following syntaxes to view the schema of a Delta Lake table: * View schema ```sql DESC[RIBE] .. ``` * View schema and location from the CREATE statement ```sql SHOW CREATE TABLE .. ``` #### Query a Delta Lake table[​](#query-a-delta-lake-table "Direct link to Query a Delta Lake table") 1. Use [SHOW DATABASES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATABASES.md) to view the databases in your Delta Lake cluster: ```sql SHOW DATABASES FROM ``` 2. [Switch to a Delta Lake Catalog and a database in it](#switch-to-a-delta-lake-catalog-and-a-database-in-it). 3. Use [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md) to query the destination table in the specified database: ```sql SELECT count(*) FROM LIMIT 10 ``` #### Load data from Delta Lake[​](#load-data-from-delta-lake "Direct link to Load data from Delta Lake") Suppose you have an OLAP table named `olap_tbl`, you can transform and load data like below: ```sql INSERT INTO default_catalog.olap_db.olap_tbl SELECT * FROM deltalake_table ``` #### Configure metadata cache and update strategy[​](#configure-metadata-cache-and-update-strategy "Direct link to Configure metadata cache and update strategy") From v3.3.3 onwards, Delta Lake Catalog supports [Metadata Local Cache and Retrieval](#appendix-metadata-local-cache-and-retrieval). You can configure the Delta Lake metadata cache refresh through the following FE parameters: | **Configuration item** | **Default** | **Description** | | -------------------------------------------------------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | enable\_background\_refresh\_connector\_metadata | `true` | Whether to enable the periodic Delta Lake metadata cache refresh. After it is enabled, StarRocks polls the metastore (Hive Metastore or AWS Glue) of your Delta Lake cluster, and refreshes the cached metadata of the frequently accessed Delta Lake catalogs to perceive data changes. `true` indicates to enable the Delta Lake metadata cache refresh, and `false` indicates to disable it. | | background\_refresh\_metadata\_interval\_millis | `600000` | The interval between two consecutive Delta Lake metadata cache refreshes. Unit: millisecond. | | background\_refresh\_metadata\_time\_secs\_since\_last\_access\_secs | `86400` | The expiration time of a Delta Lake metadata cache refresh task. For the Delta Lake catalog that has been accessed, if it has not been accessed for more than the specified time, StarRocks stops refreshing its cached metadata. For the Delta Lake catalog that has not been accessed, StarRocks will not refresh its cached metadata. Unit: second. | #### Appendix: Metadata Local Cache and Retrieval[​](#appendix-metadata-local-cache-and-retrieval "Direct link to Appendix: Metadata Local Cache and Retrieval") Because the repeated decompression and parsing of metadata files can introduce unnecessary delays, StarRocks employs a new metadata cache strategy of caching deserialized memory objects. By storing these deserialized files in memory, the system can bypass the decompression and parsing stages for subsequent queries. This caching mechanism allows direct access to the required metadata, significantly reducing retrieval times. As a result, the system becomes more responsive and better suited to meet high query demands and materialized view rewriting needs. You can configure this behavior through the Catalog property [MetadataUpdateParams](#metadataupdateparams) and [relevant configuration items](#configure-metadata-cache-and-update-strategy). #### Feature support[​](#feature-support "Direct link to Feature support") Currently, Delta Lake catalogs support the following table features: * V2 Checkpoint (From v3.3.0 onwards) * Timestamp without Timezone (From v3.3.1 onwards) * Column mapping (From v3.3.6 onwards) * Deletion Vector (From v3.4.1 onwards) --- ### Elasticsearch catalog StarRocks supports Elasticsearch catalogs from v3.1 onwards. StarRocks and Elasticsearch are both popular analytical systems with distinct strengths. StarRocks excels in large-scale distributed computing and supports querying data from Elasticsearch through external tables. Elasticsearch is known for its full-text search capabilities. The combination of StarRocks and Elasticsearch provides a more comprehensive OLAP solution. With Elasticsearch catalogs, you can directly analyze all indexed data in your Elasticsearch cluster by using SQL statements on StarRocks without the need for data migration. Unlike catalogs for other data sources, an Elasticsearch catalog has only one database named `default_db` in it upon creation. Each Elasticsearch index is automatically mapped to a data table and mounted to the `default_db` database. #### Create an Elasticsearch catalog[​](#create-an-elasticsearch-catalog "Direct link to Create an Elasticsearch catalog") ##### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE EXTERNAL CATALOG [COMMENT ] PROPERTIES ("key"="value", ...) ``` ##### Parameters[​](#parameters "Direct link to Parameters") ###### `catalog_name`[​](#catalog_name "Direct link to catalog_name") The name of the Elasticsearch catalog. The naming conventions are as follows: * The name can contain letters, digits (0-9), and underscores (\_). It must start with a letter. * The name is case-sensitive and cannot exceed 1023 characters in length. ###### `comment`[​](#comment "Direct link to comment") The description of the Elasticsearch catalog. This parameter is optional. ###### PROPERTIES[​](#properties "Direct link to PROPERTIES") The properties of the Elasticsearch catalog. The following table describes the properties supported for Elasticsearch catalogs. | Parameter | Required | Default value | Description | | ---------------------- | -------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | hosts | Yes | None | The connection address of the Elasticsearch cluster. You can specify one or more addresses. StarRocks can parse the Elasticsearch version and index shard allocation from this address. StarRocks communicates with your Elasticsearch cluster based on the address returned by the `GET /_nodes/http` API operation. Therefore, the value of the `hosts` parameter must be the same as the address returned by the `GET /_nodes/http` API operation. Otherwise, BEs or CNs may not be able to communicate with your Elasticsearch cluster. | | type | Yes | None | The type of the data source. Set this parameter to `es` when you create an Elasticsearch catalog. | | user | No | Empty | The username that is used to log in to the Elasticsearch cluster with HTTP basic authentication enabled. Make sure that you have permissions to access paths such as `/cluster/state/ nodes/http` and have permissions to read the index. | | password | No | Empty | The password that is used to log in to the Elasticsearch cluster. | | es.type | No | \_doc | The type of the index. If you want to query data in Elasticsearch 8 and later versions, you do not need to configure this parameter because the mapping types have been removed in Elasticsearch 8 and later versions. | | es.nodes.wan.only | No | FALSE | Specifies whether StarRocks only uses the addresses specified by `hosts` to access the Elasticsearch cluster and fetch data.- `true`: StarRocks only uses the addresses specified by `hosts` to access the Elasticsearch cluster and fetch data and does not sniff data nodes on which the shards of the Elasticsearch index reside. If StarRocks cannot access the addresses of the data nodes inside the Elasticsearch cluster, you need to set this parameter to `true`.
- `false`: StarRocks uses the addresses specified by `hosts` to sniff data nodes on which the shards of the Elasticsearch cluster indexes reside. After StarRocks generates a query execution plan, BEs or CNs directly access the data nodes inside the Elasticsearch cluster to fetch data from the shards of indexes. If StarRocks can access the addresses of the data nodes inside the Elasticsearch cluster, we recommend that you retain the default value `false`. | | es.net.ssl | No | FALSE | Specifies whether the HTTPS protocol can be used to access the Elasticsearch cluster. Only StarRocks v2.4 and later support configuring this parameter.- `true`: Both the HTTPS and HTTP protocols can be used to access your Elasticsearch cluster.
- `false`: Only the HTTP protocol can be used to access your Elasticsearch cluster. | | enable\_docvalue\_scan | No | TRUE | Specifies whether to obtain the values of the target fields from Elasticsearch columnar storage. In most cases, reading data from columnar storage outperforms reading data from row storage. | | enable\_keyword\_sniff | No | TRUE | Specifies whether to sniff TEXT-type fields in Elasticsearch based on KEYWORD-type fields. If this parameter is set to `false`, StarRocks performs matching after tokenization. | ##### Examples[​](#examples "Direct link to Examples") The following example creates an Elasticsearch catalog named `es_test`: ```sql CREATE EXTERNAL CATALOG es_test COMMENT 'test123' PROPERTIES ( "type" = "es", "es.type" = "_doc", "hosts" = "https://xxx:9200", "es.net.ssl" = "true", "user" = "admin", "password" = "xxx", "es.nodes.wan.only" = "true" ); ``` #### Predicate pushdown[​](#predicate-pushdown "Direct link to Predicate pushdown") StarRocks supports pushing the predicates specified in queries against Elasticsearch tables down to Elasticsearch for execution. This minimizes the distance between the query engine and the storage source and improves query performance. The following table lists the operators that can be pushed down to Elasticsearch. | SQL syntax | Elasticsearch syntax | | -------------- | ---------------------- | | `=` | term query | | `in` | terms query | | `>=, <=, >, <` | range | | `and` | bool.filter | | `or` | bool.should | | `not` | bool.must\_not | | `not in` | bool.must\_not + terms | | `esquery` | ES Query DSL | #### Query examples[​](#query-examples "Direct link to Query examples") The `esquery()` function can be used to push Elasticsearch queries, such as match and geoshape queries, that cannot be expressed in SQL down to Elasticsearch for filtering and processing. In the `esquery()` function, the first parameter that specifies a column name is used to associate with the index, and the second parameter is an Elasticsearch query's Elasticsearch Query DSL-based JSON representation enclosed in curly braces (`{}`). The JSON representation can and must have only one root key, such as `match`, `geo_shape`, or `bool`. * Match query ```sql SELECT * FROM es_table WHERE esquery(k4, '{ "match": { "k4": "StarRocks on elasticsearch" } }'); ``` * Geoshape query ```sql SELECT * FROM es_table WHERE esquery(k4, '{ "geo_shape": { "location": { "shape": { "type": "envelope", "coordinates": [ [ 13, 53 ], [ 14, 52 ] ] }, "relation": "within" } } }'); ``` * Boolean query ```sql SELECT * FROM es_table WHERE esquery(k4, ' { "bool": { "must": [ { "terms": { "k1": [ 11, 12 ] } }, { "terms": { "k2": [ 100 ] } } ] } }'); ``` #### Usage notes[​](#usage-notes "Direct link to Usage notes") * From v5.x onwards, Elasticsearch adopts a different data scanning method. StarRocks only supports querying data from Elasticsearch v5.x and later. * StarRocks only supports querying data from Elasticsearch clusters that have HTTP basic authentication enabled. * Some queries, such as `count()`-involved queries, run much slower on StarRocks than on Elasticsearch, because Elasticsearch can directly read the metadata related to the specified number of documents that meet the query conditions without the need to filter the requested data. --- ### Hive catalog A Hive catalog is a kind of external catalog that is supported by StarRocks from v2.4 onwards. Within Hive catalogs, you can: * Directly query data stored in Hive without the need to manually create tables. * Use [INSERT INTO](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) or asynchronous materialized views (which are supported from v2.5 onwards) to process data stored in Hive and load the data into StarRocks. * Perform operations on StarRocks to create or drop Hive databases and tables, or sink data from StarRocks tables to Parquet-formatted (supported from v3.2 onwards) and ORC- or Textfile-formatted (supported from v3.3 onwards) Hive tables by using [INSERT INTO](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md). To ensure successful SQL workloads on your Hive cluster, your StarRocks cluster must be able to access the storage system and metastore of your Hive cluster. StarRocks supports the following storage systems and metastores: * Distributed file system (HDFS) or object storage like AWS S3, Microsoft Azure Storage, Google GCS, or other S3-compatible storage system (for example, MinIO) * Metastore like Hive metastore or AWS Glue note If you choose AWS S3 as storage, you can use HMS or AWS Glue as metastore. If you choose any other storage system, you can only use HMS as metastore. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * StarRocks supports queries against Hive tables in Parquet, ORC, Textfile, Avro, RCFile, and SequenceFile file formats: * Parquet files support the following compression formats: SNAPPY, LZ4, ZSTD, GZIP, and NO\_COMPRESSION. From v3.1.5 onwards, Parquet files also support the LZO compression format. * ORC files support the following compression formats: ZLIB, SNAPPY, LZO, LZ4, ZSTD, and NO\_COMPRESSION. * Textfile files support the LZO compression format from v3.1.5 onwards. * The data types of Hive that StarRocks does not support are INTERVAL, BINARY, and UNION. Additionally, StarRocks does not support the MAP and STRUCT data types for Textfile-formatted Hive tables. * StarRocks supports sinking data to Parquet-formatted (supported from v3.2 onwards) and ORC- or Textfile-formatted (supported from v3.3 onwards) Hive tables: * Parquet and ORC files support the following compression formats: NO\_COMPRESSION, SNAPPY, LZ4, ZSTD, and GZIP. * Textfile files support the NO\_COMPRESSION compression format. You can use the table property [`compression_codec`](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md#properties) or the system variable [`connector_sink_compression_codec`](https://docs.starrocks.io/docs/sql-reference/System_variable.md#connector_sink_compression_codec) to specify the compression algorithm used for sinking data to Hive tables. When writing to a Hive table, if the table's properties include a compression codec, StarRocks will preferentially use that algorithm to compress the written data. Otherwise, it will use the compression algorithm set in the system variable `connector_sink_compression_codec`. #### Integration preparations[​](#integration-preparations "Direct link to Integration preparations") Before you create a Hive catalog, make sure your StarRocks cluster can integrate with the storage system and metastore of your Hive cluster. ##### AWS IAM[​](#aws-iam "Direct link to AWS IAM") If your Hive cluster uses AWS S3 as storage or AWS Glue as metastore, choose your suitable authentication method and make the required preparations to ensure that your StarRocks cluster can access the related AWS cloud resources. The following authentication methods are recommended: * Instance profile * Assumed role * IAM user Of the above-mentioned three authentication methods, instance profile is the most widely used. For more information, see [Preparation for authentication in AWS IAM](https://docs.starrocks.io/docs/integrations/authenticate_to_aws_resources.md#preparations). ##### HDFS[​](#hdfs "Direct link to HDFS") If you choose HDFS as storage, configure your StarRocks cluster as follows: * (Optional) Set the username that is used to access your HDFS cluster and Hive metastore. By default, StarRocks uses the username of the FE and BE or CN processes to access your HDFS cluster and Hive metastore. You can also set the username by adding `export HADOOP_USER_NAME=""` at the beginning of the **fe/conf/hadoop\_env.sh** file of each FE and at the beginning of the **be/conf/hadoop\_env.sh** file of each BE or the **cn/conf/hadoop\_env.sh** file of each CN. After you set the username in these files, restart each FE and each BE or CN to make the parameter settings take effect. You can set only one username for each StarRocks cluster. * When you query Hive data, the FEs and BEs or CNs of your StarRocks cluster use the HDFS client to access your HDFS cluster. In most cases, you do not need to configure your StarRocks cluster to achieve that purpose, and StarRocks starts the HDFS client using the default configurations. You need to configure your StarRocks cluster only in the following situations: * High availability (HA) is enabled for your HDFS cluster: Add the **hdfs-site.xml** file of your HDFS cluster to the **$FE\_HOME/conf** path of each FE and to the **$BE\_HOME/conf** path of each BE or the **$CN\_HOME/conf** path of each CN. * View File System (ViewFs) is enabled for your HDFS cluster: Add the **core-site.xml** file of your HDFS cluster to the **$FE\_HOME/conf** path of each FE and to the **$BE\_HOME/conf** path of each BE or the **$CN\_HOME/conf** path of each CN. note If an error indicating an unknown host is returned when you send a query, you must add the mapping between the host names and IP addresses of your HDFS cluster nodes to the **/etc/hosts** path. ##### Kerberos authentication[​](#kerberos-authentication "Direct link to Kerberos authentication") If Kerberos authentication is enabled for your HDFS cluster or Hive metastore, configure your StarRocks cluster as follows: * Run the `kinit -kt keytab_path principal` command on each FE and each BE or CN to obtain Ticket Granting Ticket (TGT) from Key Distribution Center (KDC). To run this command, you must have the permissions to access your HDFS cluster and Hive metastore. Note that accessing KDC with this command is time-sensitive. Therefore, you need to use cron to run this command periodically. * Add `JAVA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf"` to the **$FE\_HOME/conf/fe.conf** file of each FE and to the **$BE\_HOME/conf/be.conf** file of each BE or the **$CN\_HOME/conf/cn.conf** file of each CN. In this example, `/etc/krb5.conf` is the save path of the **krb5.conf** file. You can modify the path based on your needs. #### Create a Hive catalog[​](#create-a-hive-catalog "Direct link to Create a Hive catalog") ##### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE EXTERNAL CATALOG [COMMENT ] PROPERTIES ( "type" = "hive", GeneralParams, MetastoreParams, StorageCredentialParams, MetadataUpdateParams ) ``` ##### Parameters[​](#parameters "Direct link to Parameters") ###### catalog\_name[​](#catalog_name "Direct link to catalog_name") The name of the Hive catalog. The naming conventions are as follows: * The name can contain letters, digits (0-9), and underscores (\_). It must start with a letter. * The name is case-sensitive and cannot exceed 1023 characters in length. ###### comment[​](#comment "Direct link to comment") The description of the Hive catalog. This parameter is optional. ###### type[​](#type "Direct link to type") The type of your data source. Set the value to `hive`. ###### GeneralParams[​](#generalparams "Direct link to GeneralParams") A set of general parameters. The following table describes the parameters you can configure in `GeneralParams`. | Parameter | Required | Description | | -------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | enable\_recursive\_listing | No | Specifies whether StarRocks reads data from a table and its partitions and from the subdirectories within the physical locations of the table and its partitions. Valid values: `true` and `false`. Default value: `true`. The value `true` specifies to recursively list subdirectories, and the value `false` specifies to ignore subdirectories. | ###### MetastoreParams[​](#metastoreparams "Direct link to MetastoreParams") A set of parameters about how StarRocks integrates with the metastore of your data source. ###### Hive metastore[​](#hive-metastore "Direct link to Hive metastore") If you choose Hive metastore as the metastore of your data source, configure `MetastoreParams` as follows: ```sql "hive.metastore.type" = "hive", "hive.metastore.uris" = "" ``` note Before querying Hive data, you must add the mapping between the host names and IP addresses of your Hive metastore nodes to the `/etc/hosts` path. Otherwise, StarRocks may fail to access your Hive metastore when you start a query. The following table describes the parameter you need to configure in `MetastoreParams`. | Parameter | Required | Description | | ------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | hive.metastore.type | Yes | The type of metastore that you use for your Hive cluster. Set the value to `hive`. | | hive.metastore.uris | Yes | The URI of your Hive metastore. Format: `thrift://:`.
If high availability (HA) is enabled for your Hive metastore, you can specify multiple metastore URIs and separate them with commas (`,`), for example, `"thrift://:,thrift://:,thrift://:"`. | ###### AWS Glue[​](#aws-glue "Direct link to AWS Glue") If you choose AWS Glue as the metastore of your data source, which is supported only when you choose AWS S3 as storage, take one of the following actions: * To choose the instance profile-based authentication method, configure `MetastoreParams` as follows: ```sql "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.region" = "" ``` * To choose the assumed role-based authentication method, configure `MetastoreParams` as follows: ```sql "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.iam_role_arn" = "", "aws.glue.region" = "" ``` * To choose the IAM user-based authentication method, configure `MetastoreParams` as follows: ```sql "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "false", "aws.glue.access_key" = "", "aws.glue.secret_key" = "", "aws.glue.region" = "" ``` The following table describes the parameters you need to configure in `MetastoreParams`. | Parameter | Required | Description | | ------------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | hive.metastore.type | Yes | The type of metastore that you use for your Hive cluster. Set the value to `glue`. | | aws.glue.use\_instance\_profile | Yes | Specifies whether to enable the instance profile-based authentication method and the assumed role-based authentication. Valid values: `true` and `false`. Default value: `false`. | | aws.glue.iam\_role\_arn | No | The ARN of the IAM role that has privileges on your AWS Glue Data Catalog. If you use the assumed role-based authentication method to access AWS Glue, you must specify this parameter. | | aws.glue.region | Yes | The region in which your AWS Glue Data Catalog resides. Example: `us-west-1`. | | aws.glue.access\_key | No | The access key of your AWS IAM user. If you use the IAM user-based authentication method to access AWS Glue, you must specify this parameter. | | aws.glue.secret\_key | No | The secret key of your AWS IAM user. If you use the IAM user-based authentication method to access AWS Glue, you must specify this parameter. | | hive.metastore.glue.catalogid | No | The ID of the AWS Glue Data Catalog to use. When not specified, the catalog in the current AWS account is used. You must specify this parameter when you need to access a Glue Data Catalog in a different AWS account (cross-account access). | | aws.glue.resource\_share\_type | No | Controls which databases `SHOW DATABASES` lists, by setting the `ResourceShareType` sent to the AWS Glue `GetDatabases` API. Case-insensitive. Valid values: `FOREIGN` (databases shared with your account from another account through AWS Resource Access Manager), `FEDERATED` (databases that reference an external data source such as a JDBC connection), and `ALL` (local databases plus both of the above). When unset, only local databases are listed. This parameter affects listing only. StarRocks still queries a database using the single account set by `hive.metastore.glue.catalogid`, so a listed `FOREIGN` database remains unqueryable unless you also set `hive.metastore.glue.catalogid` to the owner account or create a [resource link](https://docs.aws.amazon.com/lake-formation/latest/dg/resource-links-about.html) for it in your own Data Catalog; a `FEDERATED` database is not a Hive-compatible database and cannot be queried through this catalog. | For information about how to choose an authentication method for accessing AWS Glue and how to configure an access control policy in the AWS IAM Console, see [Authentication parameters for accessing AWS Glue](https://docs.starrocks.io/docs/integrations/authenticate_to_aws_resources.md#authentication-parameters-for-accessing-aws-glue). ###### StorageCredentialParams[​](#storagecredentialparams "Direct link to StorageCredentialParams") A set of parameters about how StarRocks integrates with your storage system. This parameter set is optional. If you use HDFS as storage, you do not need to configure `StorageCredentialParams`. If you use AWS S3, other S3-compatible storage system, Microsoft Azure Storage, or Google GCS as storage, you must configure `StorageCredentialParams`. ###### AWS S3[​](#aws-s3 "Direct link to AWS S3") If you choose AWS S3 as storage for your Hive cluster, take one of the following actions: * To choose the instance profile-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "" ``` * To choose the assumed role-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "", "aws.s3.region" = "" ``` * To choose the IAM user-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ----------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | aws.s3.use\_instance\_profile | Yes | Specifies whether to enable the instance profile-based authentication method and the assumed role-based authentication method. Valid values: `true` and `false`. Default value: `false`. | | aws.s3.iam\_role\_arn | No | The ARN of the IAM role that has privileges on your AWS S3 bucket. If you use the assumed role-based authentication method to access AWS S3, you must specify this parameter. | | aws.s3.region | Yes | The region in which your AWS S3 bucket resides. Example: `us-west-1`. | | aws.s3.access\_key | No | The access key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter. | | aws.s3.secret\_key | No | The secret key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter. | For information about how to choose an authentication method for accessing AWS S3 and how to configure an access control policy in AWS IAM Console, see [Authentication parameters for accessing AWS S3](https://docs.starrocks.io/docs/integrations/authenticate_to_aws_resources.md#authentication-parameters-for-accessing-aws-s3). ###### S3-compatible storage system[​](#s3-compatible-storage-system "Direct link to S3-compatible storage system") Hive catalogs support S3-compatible storage systems from v2.5 onwards. If you choose an S3-compatible storage system, such as MinIO, as storage for your Hive cluster, configure `StorageCredentialParams` as follows to ensure a successful integration: ```sql "aws.s3.enable_ssl" = "false", "aws.s3.enable_path_style_access" = "true", "aws.s3.endpoint" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ---------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | aws.s3.enable\_ssl | Yes | Specifies whether to enable SSL connection.
Valid values: `true` and `false`. Default value: `true`. | | aws.s3.enable\_path\_style\_access | Yes | Specifies whether to enable path-style access.
Valid values: `true` and `false`. Default value: `false`. For MinIO, you must set the value to `true`.
Path-style URLs use the following format: `https://s3..amazonaws.com//`. For example, if you create a bucket named `DOC-EXAMPLE-BUCKET1` in the US West (Oregon) Region, and you want to access the `alice.jpg` object in that bucket, you can use the following path-style URL: `https://s3.us-west-2.amazonaws.com/DOC-EXAMPLE-BUCKET1/alice.jpg`. | | aws.s3.endpoint | Yes | The endpoint that is used to connect to your S3-compatible storage system instead of AWS S3. | | aws.s3.access\_key | Yes | The access key of your IAM user. | | aws.s3.secret\_key | Yes | The secret key of your IAM user. | ###### Microsoft Azure Storage[​](#microsoft-azure-storage "Direct link to Microsoft Azure Storage") Hive catalogs support Microsoft Azure Storage from v3.0 onwards. ###### Azure Blob Storage[​](#azure-blob-storage "Direct link to Azure Blob Storage") If you choose Blob Storage as storage for your Hive cluster, take one of the following actions: * To choose the Shared Key authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.blob.storage_account" = "", "azure.blob.shared_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | --------------------------- | ------------ | -------------------------------------------- | | azure.blob.storage\_account | Yes | The username of your Blob Storage account. | | azure.blob.shared\_key | Yes | The shared key of your Blob Storage account. | * To choose the SAS Token authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.blob.storage_account" = "", "azure.blob.container" = "", "azure.blob.sas_token" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | --------------------------- | ------------ | --------------------------------------------------------------- | | azure.blob.storage\_account | Yes | The username of your Blob Storage account. | | azure.blob.container | Yes | The name of the blob container that stores your data. | | azure.blob.sas\_token | Yes | The SAS token that is used to access your Blob Storage account. | ###### Azure Data Lake Storage Gen2[​](#azure-data-lake-storage-gen2 "Direct link to Azure Data Lake Storage Gen2") If you choose Data Lake Storage Gen2 as storage for your Hive cluster, take one of the following actions: * To choose the Managed Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------------------ | ------------ | ------------------------------------------------------------------------------------------------ | | azure.adls2.oauth2\_use\_managed\_identity | Yes | Specifies whether to enable the Managed Identity authentication method. Set the value to `true`. | | azure.adls2.oauth2\_tenant\_id | Yes | The ID of the tenant whose data you want to access. | | azure.adls2.oauth2\_client\_id | Yes | The client (application) ID of the managed identity. | * To choose the Shared Key authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.storage_account" = "", "azure.adls2.shared_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ---------------------------- | ------------ | -------------------------------------------------------------- | | azure.adls2.storage\_account | Yes | The username of your Data Lake Storage Gen2 storage account. | | azure.adls2.shared\_key | Yes | The shared key of your Data Lake Storage Gen2 storage account. | * To choose the Service Principal authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_client_secret" = "", "azure.adls2.oauth2_client_endpoint" = "" ``` The following table describes the parameters you need to configure `in StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------------ | ------------ | -------------------------------------------------------------------------- | | azure.adls2.oauth2\_client\_id | Yes | The client (application) ID of the service principal. | | azure.adls2.oauth2\_client\_secret | Yes | The value of the new client (application) secret created. | | azure.adls2.oauth2\_client\_endpoint | Yes | The OAuth 2.0 token endpoint (v1) of the service principal or application. | * To choose the Workload Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_token_file" = "", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | azure.adls2.oauth2\_token\_file | Yes | The absolute file path to the OAuth2 token file projected into the pod by the Azure Workload Identity webhook. | | azure.adls2.oauth2\_tenant\_id | Yes | The ID of the tenant whose data you want to access. | | azure.adls2.oauth2\_client\_id | Yes | The client ID (application ID) of the Azure AD application (user-assigned managed identity or app registration) associated with the workload identity. | ###### Azure Data Lake Storage Gen1[​](#azure-data-lake-storage-gen1 "Direct link to Azure Data Lake Storage Gen1") If you choose Data Lake Storage Gen1 as storage for your Hive cluster, take one of the following actions: * To choose the Managed Service Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls1.use_managed_service_identity" = "true" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------------------- | ------------ | -------------------------------------------------------------------------------------------------------- | | azure.adls1.use\_managed\_service\_identity | Yes | Specifies whether to enable the Managed Service Identity authentication method. Set the value to `true`. | * To choose the Service Principal authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls1.oauth2_client_id" = "", "azure.adls1.oauth2_credential" = "", "azure.adls1.oauth2_endpoint" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------ | ------------ | -------------------------------------------------------------------------- | | azure.adls1.oauth2\_client\_id | Yes | The client (application) ID of the service principal. | | azure.adls1.oauth2\_credential | Yes | The value of the new client (application) secret created. | | azure.adls1.oauth2\_endpoint | Yes | The OAuth 2.0 token endpoint (v1) of the service principal or application. | ###### Google GCS[​](#google-gcs "Direct link to Google GCS") Hive catalogs support Google GCS from v3.0 onwards. If you choose Google GCS as storage for your Hive cluster, take one of the following actions: * To choose the VM-based authentication method, configure `StorageCredentialParams` as follows: ```sql "gcp.gcs.use_compute_engine_service_account" = "true" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ---------------------------------------------- | ----------------- | --------------------- | ------------------------------------------------------------------------------------------- | | gcp.gcs.use\_compute\_engine\_service\_account | false | true | Specifies whether to directly use the service account that is bound to your Compute Engine. | * To choose the service account-based authentication method, configure `StorageCredentialParams` as follows: ```sql "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ------------------------------------------ | ----------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------- | | gcp.gcs.service\_account\_email | "" | "" | The email address in the JSON file generated at the creation of the service account. | | gcp.gcs.service\_account\_private\_key\_id | "" | "61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea" | The private key ID in the JSON file generated at the creation of the service account. | | gcp.gcs.service\_account\_private\_key | "" | "-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n" | The private key in the JSON file generated at the creation of the service account. | * To choose the impersonation-based authentication method, configure `StorageCredentialParams` as follows: * Make a VM instance impersonate a service account: ```sql "gcp.gcs.use_compute_engine_service_account" = "true", "gcp.gcs.impersonation_service_account" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ---------------------------------------------- | ----------------- | --------------------- | ------------------------------------------------------------------------------------------- | | gcp.gcs.use\_compute\_engine\_service\_account | false | true | Specifies whether to directly use the service account that is bound to your Compute Engine. | | gcp.gcs.impersonation\_service\_account | "" | "hello" | The service account that you want to impersonate. | * Make a service account (temporarily named as meta service account) impersonate another service account (temporarily named as data service account): ```sql "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "", "gcp.gcs.impersonation_service_account" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ------------------------------------------ | ----------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------ | | gcp.gcs.service\_account\_email | "" | "" | The email address in the JSON file generated at the creation of the meta service account. | | gcp.gcs.service\_account\_private\_key\_id | "" | "61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea" | The private key ID in the JSON file generated at the creation of the meta service account. | | gcp.gcs.service\_account\_private\_key | "" | "-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n" | The private key in the JSON file generated at the creation of the meta service account. | | gcp.gcs.impersonation\_service\_account | "" | "hello" | The data service account that you want to impersonate. | ###### MetadataUpdateParams[​](#metadataupdateparams "Direct link to MetadataUpdateParams") A set of parameters about how StarRocks updates the cached metadata of Hive. This parameter set is optional. StarRocks implements the [automatic asynchronous update policy](#appendix-understand-metadata-automatic-asynchronous-update) by default. In most cases, you can ignore `MetadataUpdateParams` and do not need to tune the policy parameters in it, because the default values of these parameters already provide you with an out-of-the-box performance. However, if the frequency of data updates in Hive is high, you can tune these parameters to further optimize the performance of automatic asynchronous updates. note In most cases, if your Hive data is updated at a granularity of 1 hour or less, the data update frequency is considered high. | Parameter | Required | Description | | ------------------------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | enable\_metastore\_cache | No | Specifies whether StarRocks caches the metadata of Hive tables. Valid values: `true` and `false`. Default value: `true`. The value `true` enables the cache, and the value `false` disables the cache. | | enable\_remote\_file\_cache | No | Specifies whether StarRocks caches the metadata of the underlying data files of Hive tables or partitions. Valid values: `true` and `false`. Default value: `true`. The value `true` enables the cache, and the value `false` disables the cache. | | metastore\_cache\_refresh\_interval\_sec | No | The time interval at which StarRocks asynchronously updates the metadata of Hive tables or partitions cached in itself. Unit: seconds. Default value: `60`, which is one minute. Since v3.3.0, the default value of this property is changed from `7200` to `60`. | | remote\_file\_cache\_refresh\_interval\_sec | No | The time interval at which StarRocks asynchronously updates the metadata of the underlying data files of Hive tables or partitions cached in itself. Unit: seconds. Default value: `60`. | | metastore\_cache\_ttl\_sec | No | The time interval at which StarRocks automatically discards the metadata of Hive tables or partitions cached in itself. Unit: seconds. Default value: `86400`, which is 24 hours. | | remote\_file\_cache\_ttl\_sec | No | The time interval at which StarRocks automatically discards the metadata of the underlying data files of Hive tables or partitions cached in itself. Unit: seconds. Default value: `129600`, which is 36 hours. | | enable\_cache\_list\_names | No | Specifies whether StarRocks caches Hive partition names. Valid values: `true` and `false`. Default value: `true`. The value `true` enables the cache, and the value `false` disables the cache. | | remote\_file\_cache\_memory\_ratio | No | The maximum memory usage ratio for the remote file cache. Default value: `0.1`, which is 10%. Supported from v3.5.6 onwards. | ##### Examples[​](#examples "Direct link to Examples") The following examples create a Hive catalog named `hive_catalog_hms` or `hive_catalog_glue`, depending on the type of metastore you use, to query data from your Hive cluster. ###### HDFS[​](#hdfs-1 "Direct link to HDFS") If you use HDFS as storage, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083" ); ``` ###### AWS S3[​](#aws-s3-1 "Direct link to AWS S3") ###### Instance profile-based authentication[​](#instance-profile-based-authentication "Direct link to Instance profile-based authentication") * If you use Hive metastore in your Hive cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "us-west-2" ); ``` * If you use AWS Glue in your Amazon EMR Hive cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_glue PROPERTIES ( "type" = "hive", "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.region" = "us-west-2", "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "us-west-2" ); ``` ###### Assumed role-based authentication[​](#assumed-role-based-authentication "Direct link to Assumed role-based authentication") * If you use Hive metastore in your Hive cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::081976408565:role/test_s3_role", "aws.s3.region" = "us-west-2" ); ``` * If you use AWS Glue in your Amazon EMR Hive cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_glue PROPERTIES ( "type" = "hive", "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.iam_role_arn" = "arn:aws:iam::081976408565:role/test_glue_role", "aws.glue.region" = "us-west-2", "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::081976408565:role/test_s3_role", "aws.s3.region" = "us-west-2" ); ``` ###### IAM user-based authentication[​](#iam-user-based-authentication "Direct link to IAM user-based authentication") * If you use Hive metastore in your Hive cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "us-west-2" ); ``` * If you use AWS Glue in your Amazon EMR Hive cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_glue PROPERTIES ( "type" = "hive", "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "false", "aws.glue.access_key" = "", "aws.glue.secret_key" = "", "aws.glue.region" = "us-west-2", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "us-west-2" ); ``` ###### S3-compatible storage system[​](#s3-compatible-storage-system-1 "Direct link to S3-compatible storage system") Use MinIO as an example. Run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.enable_ssl" = "true", "aws.s3.enable_path_style_access" = "true", "aws.s3.endpoint" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ); ``` ###### Microsoft Azure Storage[​](#microsoft-azure-storage-1 "Direct link to Microsoft Azure Storage") ###### Azure Blob Storage[​](#azure-blob-storage-1 "Direct link to Azure Blob Storage") * If you choose the Shared Key authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.blob.storage_account" = "", "azure.blob.shared_key" = "" ); ``` * If you choose the SAS Token authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.blob.storage_account" = "", "azure.blob.container" = "", "azure.blob.sas_token" = "" ); ``` ###### Azure Data Lake Storage Gen1[​](#azure-data-lake-storage-gen1-1 "Direct link to Azure Data Lake Storage Gen1") * If you choose the Managed Service Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls1.use_managed_service_identity" = "true" ); ``` * If you choose the Service Principal authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls1.oauth2_client_id" = "", "azure.adls1.oauth2_credential" = "", "azure.adls1.oauth2_endpoint" = "" ); ``` ###### Azure Data Lake Storage Gen2[​](#azure-data-lake-storage-gen2-1 "Direct link to Azure Data Lake Storage Gen2") * If you choose the Managed Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` * If you choose the Shared Key authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.storage_account" = "", "azure.adls2.shared_key" = "" ); ``` * If you choose the Service Principal authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_client_secret" = "", "azure.adls2.oauth2_client_endpoint" = "" ); ``` * If you choose the Workload Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_token_file" = "/var/run/secrets/azure/tokens/azure-identity-token", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` ###### Google GCS[​](#google-gcs-1 "Direct link to Google GCS") * If you choose the VM-based authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.use_compute_engine_service_account" = "true" ); ``` * If you choose the service account-based authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "" ); ``` * If you choose the impersonation-based authentication method: * If you make a VM instance impersonate a service account, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.use_compute_engine_service_account" = "true", "gcp.gcs.impersonation_service_account" = "" ); ``` * If you make a service account impersonate another service account, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "", "gcp.gcs.impersonation_service_account" = "" ); ``` #### View Hive catalogs[​](#view-hive-catalogs "Direct link to View Hive catalogs") You can use [SHOW CATALOGS](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CATALOGS.md) to query all catalogs in the current StarRocks cluster: ```sql SHOW CATALOGS; ``` You can also use [SHOW CREATE CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CREATE_CATALOG.md) to query the creation statement of an external catalog. The following example queries the creation statement of a Hive catalog named `hive_catalog_glue`: ```sql SHOW CREATE CATALOG hive_catalog_glue; ``` #### Switch to a Hive Catalog and a database in it[​](#switch-to-a-hive-catalog-and-a-database-in-it "Direct link to Switch to a Hive Catalog and a database in it") You can use one of the following methods to switch to a Hive catalog and a database in it: * Use [SET CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SET_CATALOG.md) to specify a Hive catalog in the current session, and then use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to specify an active database: ```sql -- Switch to a specified catalog in the current session: SET CATALOG -- Specify the active database in the current session: USE ``` * Directly use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to switch to a Hive catalog and a database in it: ```sql USE . ``` #### Drop a Hive catalog[​](#drop-a-hive-catalog "Direct link to Drop a Hive catalog") You can use [DROP CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/DROP_CATALOG.md) to drop an external catalog. The following example drops a Hive catalog named `hive_catalog_glue`: ```sql DROP Catalog hive_catalog_glue; ``` #### View the schema of a Hive table[​](#view-the-schema-of-a-hive-table "Direct link to View the schema of a Hive table") You can use one of the following syntaxes to view the schema of a Hive table: * View schema ```sql DESC[RIBE] .. ``` * View schema and location from the CREATE statement ```sql SHOW CREATE TABLE .. ``` #### Query a Hive table[​](#query-a-hive-table "Direct link to Query a Hive table") 1. Use [SHOW DATABASES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATABASES.md) to view the databases in your Hive cluster: ```sql SHOW DATABASES FROM ``` 2. [Switch to a Hive Catalog and a database in it](#switch-to-a-hive-catalog-and-a-database-in-it). 3. Use [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md) to query the destination table in the specified database: ```sql SELECT count(*) FROM LIMIT 10 ``` #### Load data from Hive[​](#load-data-from-hive "Direct link to Load data from Hive") Suppose you have an OLAP table named `olap_tbl`, you can transform and load data like below: ```sql INSERT INTO default_catalog.olap_db.olap_tbl SELECT * FROM hive_table ``` #### Grant privileges on Hive tables and views[​](#grant-privileges-on-hive-tables-and-views "Direct link to Grant privileges on Hive tables and views") You can use the [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) statement to grant the privileges on all tables and views within a Hive catalog to a specific role. The command syntax is as follows: ```sql GRANT SELECT ON ALL TABLES IN ALL DATABASES TO ROLE ``` For example, use the following commands to create a role named `hive_role_table`, switch to the Hive catalog `hive_catalog`, and then grant the role `hive_role_table` the privilege to query all tables and views within the Hive catalog `hive_catalog`: ```sql -- Create a role named hive_role_table. CREATE ROLE hive_role_table; -- Switch to the Hive catalog hive_catalog. SET CATALOG hive_catalog; -- Grant the role hive_role_table the privilege to query all tables and views within the Hive catalog hive_catalog. GRANT SELECT ON ALL TABLES IN ALL DATABASES TO ROLE hive_role_table; ``` #### Create a Hive database[​](#create-a-hive-database "Direct link to Create a Hive database") Similar to the internal catalog of StarRocks, if you have the [CREATE DATABASE](https://docs.starrocks.io/docs/administration/user_privs/authorization/privilege_item.md#catalog) privilege on a Hive catalog, you can use the [CREATE DATABASE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/CREATE_DATABASE.md) statement to create a database in that Hive catalog. This feature is supported from v3.2 onwards. note You can grant and revoke privileges by using [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) and [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE.md). note [Switch to a Hive catalog](#switch-to-a-hive-catalog-and-a-database-in-it), and then use the following statement to create a Hive database in that catalog: ```sql CREATE DATABASE [PROPERTIES ("location" = ":///")] ``` The `location` parameter specifies the file path in which you want to create the database, which can be in either HDFS or cloud storage. * When you use Hive metastore as the metastore of your Hive cluster, the `location` parameter defaults to `/`, which is supported by Hive metastore if you do not specify that parameter at database creation. * When you use AWS Glue as the metastore of your Hive cluster, the `location` parameter does not have a default value, and therefore you must specify that parameter at database creation. The `prefix` varies based on the storage system you use: | **Storage system** | **`Prefix`** **value** | | ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | HDFS | `hdfs` | | Google GCS | `gs` | | Azure Blob Storage | - If your storage account allows access over HTTP, the `prefix` is `wasb`.
- If your storage account allows access over HTTPS, the `prefix` is `wasbs`. | | Azure Data Lake Storage Gen1 | `adl` | | Azure Data Lake Storage Gen2 | - If your storage account allows access over HTTP, the`prefix` is `abfs`.
- If your storage account allows access over HTTPS, the `prefix` is `abfss`. | | AWS S3 or other S3-compatible storage (for example, MinIO) | `s3` | #### Drop a Hive database[​](#drop-a-hive-database "Direct link to Drop a Hive database") Similar to the internal databases of StarRocks, if you have the [DROP](https://docs.starrocks.io/docs/administration/user_privs/authorization/privilege_item.md#database) privilege on a Hive database, you can use the [DROP DATABASE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/DROP_DATABASE.md) statement to drop that Hive database. This feature is supported from v3.2 onwards. You can only drop empty databases. note You can grant and revoke privileges by using [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) and [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE.md). When you drop a Hive database, the database's file path on your HDFS cluster or cloud storage will not be dropped along with the database. [Switch to a Hive catalog](#switch-to-a-hive-catalog-and-a-database-in-it), and then use the following statement to drop a Hive database in that catalog: ```sql DROP DATABASE ``` #### Create a Hive table[​](#create-a-hive-table "Direct link to Create a Hive table") Similar to the internal databases of StarRocks, if you have the [CREATE TABLE](https://docs.starrocks.io/docs/administration/user_privs/authorization/privilege_item.md#database) privilege on a Hive database, you can use the [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md), [CREATE TABLE AS SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE_AS_SELECT.md), or [CREATE TABLE LIKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE_LIKE.md) statement to create a managed table in that Hive database. This feature is supported since v3.2 in which version StarRocks only supports creating Parquet-formatted Hive tables. From v3.3 onwards, StarRocks also supports creating ORC- and Textfile-formatted Hive tables. note * You can grant and revoke privileges by using [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) and [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE.md). * Hive catalogs support CREATE TABLE LIKE from v3.2.4 onwards. [Switch to a Hive catalog and a database in it](#switch-to-a-hive-catalog-and-a-database-in-it), and then use the following syntax to create a Hive managed table in that database. ##### Syntax[​](#syntax-1 "Direct link to Syntax") ```sql CREATE TABLE [IF NOT EXISTS] [database.]table_name (column_definition1[, column_definition2, ... partition_column_definition1,partition_column_definition2...]) [partition_desc] [PROPERTIES ("key" = "value", ...)] [AS SELECT query] [LIKE [database.]] ``` ##### Parameters[​](#parameters-1 "Direct link to Parameters") ###### column\_definition[​](#column_definition "Direct link to column_definition") The syntax of `column_definition` is as follows: ```sql col_name col_type [COMMENT 'comment'] ``` The following table describes the parameters. | Parameter | Description | | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | col\_name | The name of the column. | | col\_type | The data type of the column. The following data types are supported: TINYINT, SMALLINT, INT, BIGINT, FLOAT, DOUBLE, DECIMAL, DATE, DATETIME, CHAR, VARCHAR\[(length)], ARRAY, MAP, and STRUCT. The LARGEINT, HLL, and BITMAP data types are not supported. | > **NOTICE** > > All non-partition columns must use `NULL` as the default value. This means that you must specify `DEFAULT "NULL"` for each of the non-partition columns in the table creation statement. Additionally, partition columns must be defined following non-partition columns and cannot use `NULL` as the default value. ###### partition\_desc[​](#partition_desc "Direct link to partition_desc") The syntax of `partition_desc` is as follows: ```sql PARTITION BY (par_col1[, par_col2...]) ``` Currently StarRocks only supports identity transforms, which means that StarRocks creates a partition for each unique partition value. > **NOTICE** > > Partition columns must be defined following non-partition columns. Partition columns support all data types excluding FLOAT, DOUBLE, DECIMAL, and DATETIME and cannot use `NULL` as the default value. Additionally, the sequence of the partition columns declared in `partition_desc` must be consistent with the sequence of the columns defined in `column_definition`. ###### PROPERTIES[​](#properties "Direct link to PROPERTIES") You can specify the table attributes in the `"key" = "value"` format in `properties`. The following table describes a few key properties. | **Property** | **Description** | | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | location | The file path in which you want to create the managed table. When you use HMS as metastore, you do not need to specify the `location` parameter, because StarRocks will create the table in the default file path of the current Hive catalog. When you use AWS Glue as metadata service:- If you have specified the `location` parameter for the database in which you want to create the table, you do not need to specify the `location` parameter for the table. As such, the table defaults to the file path of the database to which it belongs.
- If you have not specified the `location` for the database in which you want to create the table, you must specify the `location` parameter for the table. | | file\_format | The file format of the managed table. Supported file formats are Parquet, ORC, and Textfile. ORC and Textfile formats are supported from v3.3 onwards. Valid values: `parquet`, `orc`, and `textfile`. Default value: `parquet`. | | compression\_codec | The compression algorithm used for the managed table. | ##### Examples[​](#examples-1 "Direct link to Examples") The following DDLs use the default file format Parquet as an example. 1. Create a non-partitioned table named `unpartition_tbl`. The table consists of two columns, `id` and `score`, as shown below: ```sql CREATE TABLE unpartition_tbl ( id int, score double ); ``` 2. Create a partitioned table named `partition_tbl_1`. The table consists of three columns, `action`, `id`, and `dt`, of which `id` and `dt` are defined as partition columns, as shown below: ```sql CREATE TABLE partition_tbl_1 ( action varchar(20), id int, dt date ) PARTITION BY (id,dt); ``` 3. Query an existing table named `partition_tbl_1`, and create a partitioned table named `partition_tbl_2` based on the query result of `partition_tbl_1`. For `partition_tbl_2`, `id` and `dt` are defined as partition columns, as shown below: ```sql CREATE TABLE partition_tbl_2 PARTITION BY (k1, k2) AS SELECT * from partition_tbl_1; ``` #### Sink data to a Hive table[​](#sink-data-to-a-hive-table "Direct link to Sink data to a Hive table") Similar to the internal tables of StarRocks, if you have the [INSERT](https://docs.starrocks.io/docs/administration/user_privs/authorization/privilege_item.md#table) privilege on a Hive table (which can be a managed table or an external table), you can use the [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) statement to sink the data of a StarRocks table to that Hive table. This feature is supported since v3.2 in which version data can be sunk only to Parquet-formatted Hive tables. From v3.3 onwards, StarRocks also supports sinking data to ORC- and Textfile-formatted Hive tables. Note that sinking data to external tables is disabled by default. To sink data to external tables, you must set the [system variable `ENABLE_WRITE_HIVE_EXTERNAL_TABLE`](https://docs.starrocks.io/docs/sql-reference/System_variable.md) to `true`. note * You can grant and revoke privileges by using [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) and [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE.md). * You can use the table property [`compression_codec`](#properties) or the system variable [`connector_sink_compression_codec`](https://docs.starrocks.io/docs/sql-reference/System_variable.md#connector_sink_compression_codec) to specify the compression algorithm used for sinking data to Hive tables. StarRocks will prioritize using the compression codec specified in the table property. [Switch to a Hive catalog and a database in it](#switch-to-a-hive-catalog-and-a-database-in-it), and then use the following syntax to sink the data of StarRocks table to a Parquet-formatted Hive table in that database. ##### Syntax[​](#syntax-2 "Direct link to Syntax") ```sql INSERT {INTO | OVERWRITE} [ (column_name [, ...]) ] { VALUES ( { expression | DEFAULT } [, ...] ) [, ...] | query } -- If you want to sink data to specified partitions, use the following syntax: INSERT {INTO | OVERWRITE} PARTITION (par_col1= [, par_col2=...]) { VALUES ( { expression | DEFAULT } [, ...] ) [, ...] | query } ``` > **NOTICE** > > Partition columns do not allow `NULL` values. Therefore, you must make sure that no empty values are loaded into the partition columns of the Hive table. ##### Parameters[​](#parameters-2 "Direct link to Parameters") | Parameter | Description | | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | INTO | To append the data of the StarRocks table to the Hive table. | | OVERWRITE | To overwrite the existing data of the Hive table with the data of the StarRocks table. | | column\_name | The name of the destination column to which you want to load data. You can specify one or more columns. If you specify multiple columns, separate them with commas (`,`). You can only specify columns that actually exist in the Hive table, and the destination columns that you specify must include the partition columns of the Hive table. The destination columns you specify are mapped one on one in sequence to the columns of the StarRocks table, regardless of what the destination column names are. If no destination columns are specified, the data is loaded into all columns of the Hive table. If a non-partition column of the StarRocks table cannot be mapped to any column of the Hive table, StarRocks writes the default value `NULL` to the Hive table column. If the INSERT statement contains a query statement whose returned column types differ from the data types of the destination columns, StarRocks performs an implicit conversion on the mismatched columns. If the conversion fails, a syntax parsing error will be returned. | | expression | Expression that assigns values to the destination column. | | DEFAULT | Assigns a default value to the destination column. | | query | Query statement whose result will be loaded into the Hive table. It can be any SQL statement supported by StarRocks. | | PARTITION | The partitions into which you want to load data. You must specify all partition columns of the Hive table in this property. The partition columns that you specify in this property can be in a different sequence than the partition columns that you have defined in the table creation statement. If you specify this property, you cannot specify the `column_name` property. | ##### Examples[​](#examples-2 "Direct link to Examples") The following DMLs use the default file format Parquet as an example. 1. Insert three data rows into the `partition_tbl_1` table: ```sql INSERT INTO partition_tbl_1 VALUES ("buy", 1, "2023-09-01"), ("sell", 2, "2023-09-02"), ("buy", 3, "2023-09-03"); ``` 2. Insert the result of a SELECT query, which contains simple computations, into the `partition_tbl_1` table: ```sql INSERT INTO partition_tbl_1 (id, action, dt) SELECT 1+1, 'buy', '2023-09-03'; ``` 3. Insert the result of a SELECT query, which reads data from the `partition_tbl_1` table, into the same table: ```sql INSERT INTO partition_tbl_1 SELECT 'buy', 1, date_add(dt, INTERVAL 2 DAY) FROM partition_tbl_1 WHERE id=1; ``` 4. Insert the result of a SELECT query into the partitions that meet two conditions, `dt='2023-09-01'` and `id=1`, of the `partition_tbl_2` table: ```sql INSERT INTO partition_tbl_2 SELECT 'order', 1, '2023-09-01'; ``` Or ```sql INSERT INTO partition_tbl_2 partition(dt='2023-09-01',id=1) SELECT 'order'; ``` 5. Overwrite all `action` column values in the partitions that meet two conditions, `dt='2023-09-01'` and `id=1`, of the `partition_tbl_1` table with `close`: ```sql INSERT OVERWRITE partition_tbl_1 SELECT 'close', 1, '2023-09-01'; ``` Or ```sql INSERT OVERWRITE partition_tbl_1 partition(dt='2023-09-01',id=1) SELECT 'close'; ``` #### Truncate a Hive table[​](#truncate-a-hive-table "Direct link to Truncate a Hive table") You can use the [TRUNCATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/TRUNCATE_TABLE.md) statement to quickly delete all data from Hive managed tables. This operation supports: * Truncating all data in non-partitioned tables * Truncating all partitions in a partitioned table * Truncating specific partitions in a partitioned table ##### Syntax[​](#syntax-3 "Direct link to Syntax") ```sql TRUNCATE TABLE TRUNCATE TABLE PARTITION (partition_name = partition_value [, ...]) ``` ##### Parameters[​](#parameters-3 "Direct link to Parameters") * `table_name`: The name of the Hive table that you want to truncate data from. You need to [Switch to a Hive catalog and a database in it](#switch-to-a-hive-catalog-and-a-database-in-it) before you truncate the table in that database. * `partition_name = partition_value`: The name and value of the partition column(s) to identify which partition(s) to truncate. ##### Examples[​](#examples-3 "Direct link to Examples") Switch to a Hive catalog and a database in it, and then use the following statements to truncate a Hive table in that database. 1. Truncate a non-partitioned table: ```sql TRUNCATE TABLE my_table; ``` 2. Truncate all partitions of a partitioned table: ```sql TRUNCATE TABLE my_partitioned_table; ``` 3. Truncate a single-partition partitioned table: ```sql TRUNCATE TABLE my_partitioned_table PARTITION (dt='2023-09-01'); ``` 4. Truncate specific partitions of a multi-partition partitioned table: ```sql TRUNCATE TABLE my_partitioned_table PARTITION (dt='2023-09-01', id=1); TRUNCATE TABLE my_multi_part_table PARTITION (k2='2020-01-02', k3='b'); ``` #### Drop a Hive table[​](#drop-a-hive-table "Direct link to Drop a Hive table") Similar to the internal tables of StarRocks, if you have the [DROP](https://docs.starrocks.io/docs/administration/user_privs/authorization/privilege_item.md#table) privilege on a Hive table, you can use the [DROP TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DROP_TABLE.md) statement to drop that Hive table. This feature is supported from v3.1 onwards. Note that currently StarRocks supports dropping only managed tables of Hive. note You can grant and revoke privileges by using [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) and [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE.md). When you drop a Hive table, you must specify the `FORCE` keyword in the DROP TABLE statement. After the operation is complete, the table's file path is retained, but the table's data on your HDFS cluster or cloud storage is all dropped along with the table. Exercise caution when you perform this operation to drop a Hive table. [Switch to a Hive catalog and a database in it](#switch-to-a-hive-catalog-and-a-database-in-it), and then use the following statement to drop a Hive table in that database. ```sql DROP TABLE FORCE ``` #### Manually or automatically update metadata cache[​](#manually-or-automatically-update-metadata-cache "Direct link to Manually or automatically update metadata cache") ##### Manual update[​](#manual-update "Direct link to Manual update") By default, StarRocks caches the metadata of Hive and automatically updates the metadata in asynchronous mode to deliver better performance. Additionally, after some schema changes or table updates are made on a Hive table, you can also use [REFRESH EXTERNAL TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/REFRESH_EXTERNAL_TABLE.md) to manually update its metadata, thereby ensuring that StarRocks can obtain up-to-date metadata at its earliest opportunity and generate appropriate execution plans: ```sql REFRESH EXTERNAL TABLE [PARTITION ('partition_name', ...)] ``` You need to manually update metadata in the following situations: * A data file in an existing partition is changed, for example, by running the `INSERT OVERWRITE ... PARTITION ...` command. * Schema changes are made on a Hive table. * An existing Hive table is deleted by using the DROP statement, and a new Hive table with the same name as the deleted Hive table is created. * You have specified `"enable_cache_list_names" = "true"` in `PROPERTIES` at the creation of your Hive catalog, and you want to query new partitions that you just created on your Hive cluster. note From v2.5.5 onwards, StarRocks provides the periodic Hive metadata cache refresh feature. For more information, see the below "[Periodically refresh metadata cache](#periodically-refresh-metadata-cache)" section of this topic. After you enable this feature, StarRocks refreshes your Hive metadata cache every 10 minutes by default. Therefore, manual updates are not needed in most cases. You need to perform a manual update only when you want to query new partitions immediately after the new partitions are created on your Hive cluster. Note that the REFRESH EXTERNAL TABLE refreshes only the tables and partitions cached in your FEs. #### Periodically refresh metadata cache[​](#periodically-refresh-metadata-cache "Direct link to Periodically refresh metadata cache") From v2.5.5 onwards, StarRocks can periodically refresh the cached metadata of the frequently accessed Hive catalogs to perceive data changes. You can configure the Hive metadata cache refresh through the following [FE parameters](https://docs.starrocks.io/docs/administration/management/FE_configuration.md): | Configuration item | Default | Description | | -------------------------------------------------------------------- | ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | enable\_background\_refresh\_connector\_metadata | `true` in v3.0
`false` in v2.5 | Whether to enable the periodic Hive metadata cache refresh. After it is enabled, StarRocks polls the metastore (Hive Metastore or AWS Glue) of your Hive cluster, and refreshes the cached metadata of the frequently accessed Hive catalogs to perceive data changes. `true` indicates to enable the Hive metadata cache refresh, and `false` indicates to disable it. This item is an [FE dynamic parameter](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#configure-fe-dynamic-parameters). You can modify it using the [ADMIN SET FRONTEND CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md) command. | | background\_refresh\_metadata\_interval\_millis | `600000` (10 minutes) | The interval between two consecutive Hive metadata cache refreshes. Unit: millisecond. This item is an [FE dynamic parameter](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#configure-fe-dynamic-parameters). You can modify it using the [ADMIN SET FRONTEND CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md) command. | | background\_refresh\_metadata\_time\_secs\_since\_last\_access\_secs | `86400` (24 hours) | The expiration time of a Hive metadata cache refresh task. For the Hive catalog that has been accessed, if it has not been accessed for more than the specified time, StarRocks stops refreshing its cached metadata. For the Hive catalog that has not been accessed, StarRocks will not refresh its cached metadata. Unit: second. This item is an [FE dynamic parameter](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#configure-fe-dynamic-parameters). You can modify it using the [ADMIN SET FRONTEND CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md) command. | Using the periodic Hive metadata cache refresh feature and the metadata automatic asynchronous update policy together significantly accelerates data access, reduces the read load from external data sources, and improves query performance. #### Appendix: Understand metadata automatic asynchronous update[​](#appendix-understand-metadata-automatic-asynchronous-update "Direct link to Appendix: Understand metadata automatic asynchronous update") Automatic asynchronous update is the default policy that StarRocks uses to update the metadata in Hive catalogs. By default (namely, when the `enable_metastore_cache` and `enable_remote_file_cache` parameters are both set to `true`), if a query hits a partition of a Hive table, StarRocks automatically caches the metadata of the partition and the metadata of the underlying data files of the partition. The cached metadata is updated by using the lazy update policy. For example, there is a Hive table named `table2`, which has four partitions: `p1`, `p2`, `p3`, and `p4`. A query hits `p1`, and StarRocks caches the metadata of `p1` and the metadata of the underlying data files of `p1`. Assume that the default time intervals to update and discard the cached metadata are as follows: * The time interval (specified by the `metastore_cache_refresh_interval_sec` parameter) to asynchronously update the cached metadata of `p1` is 60 seconds. * The time interval (specified by the `remote_file_cache_refresh_interval_sec` parameter) to asynchronously update the cached metadata of the underlying data files of `p1` is 60 seconds. * The time interval (specified by the `metastore_cache_ttl_sec` parameter) to automatically discard the cached metadata of `p1` is 24 hours. * The time interval (specified by the `remote_file_cache_ttl_sec` parameter) to automatically discard the cached metadata of the underlying data files of `p1` is 36 hours. The following figure shows the time intervals on a timeline for easier understanding. ![Timeline for updating and discarding cached metadata](/assets/images/hive_catalog_timeline-231c618c403c423e20192c9d3a34c70d.png) Then StarRocks updates or discards the metadata in compliance with the following rules: * If another query hits `p1` again and the current time from the last update is less than 60 seconds, StarRocks does not update the cached metadata of `p1` or the cached metadata of the underlying data files of `p1`. * If another query hits `p1` again and the current time from the last update is more than 60 seconds, StarRocks updates the cached metadata of `p1` and the cached metadata of the underlying data files of `p1`. * If the table has been accessed within 24 hours, the related cache will be refreshed every 10 minutes in the background. * If `p1` has not been accessed within 24 hours from the last update, StarRocks discards the cached metadata of `p1`. The metadata will be cached at the next query. * If `p1` has not been accessed within 36 hours from the last update, StarRocks discards the cached metadata of the underlying data files of `p1`. The metadata will be cached at the next query. --- ### Hudi catalog A Hudi catalog is a kind of external catalog that enables you to query data from Apache Hudi without ingestion. Also, you can directly transform and load data from Hudi by using [INSERT INTO](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) based on Hudi catalogs. StarRocks supports Hudi catalogs from v2.4 onwards. To ensure successful SQL workloads on your Hudi cluster, your StarRocks cluster must be able to access the storage system and metastore of your Hudi cluster. StarRocks supports the following storage systems and metastores: * Distributed file system (HDFS) or object storage like AWS S3, Microsoft Azure Storage, Google GCS, or other S3-compatible storage system (for example, MinIO) * Metastore like Hive metastore or AWS Glue > **NOTE** > > If you choose AWS S3 as storage, you can use HMS or AWS Glue as metastore. If you choose any other storage system, you can only use HMS as metastore. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * The file format of Hudi that StarRocks supports is Parquet. Parquet files support the following compression formats: SNAPPY, LZ4, ZSTD, GZIP, and NO\_COMPRESSION. * StarRocks provides complete support for Copy On Write (COW) tables and Merge On Read (MOR) tables from Hudi. #### Integration preparations[​](#integration-preparations "Direct link to Integration preparations") Before you create a Hudi catalog, make sure your StarRocks cluster can integrate with the storage system and metastore of your Hudi cluster. ##### AWS IAM[​](#aws-iam "Direct link to AWS IAM") If your Hudi cluster uses AWS S3 as storage or AWS Glue as metastore, choose your suitable authentication method and make the required preparations to ensure that your StarRocks cluster can access the related AWS cloud resources. The following authentication methods are recommended: * Instance profile * Assumed role * IAM user Of the above-mentioned three authentication methods, instance profile is the most widely used. For more information, see [Preparation for authentication in AWS IAM](https://docs.starrocks.io/docs/integrations/authenticate_to_aws_resources.md#preparations). ##### HDFS[​](#hdfs "Direct link to HDFS") If you choose HDFS as storage, configure your StarRocks cluster as follows: * (Optional) Set the username that is used to access your HDFS cluster and Hive metastore. By default, StarRocks uses the username of the FE and BE or CN processes to access your HDFS cluster and Hive metastore. You can also set the username by adding `export HADOOP_USER_NAME=""` at the beginning of the **fe/conf/hadoop\_env.sh** file of each FE and at the beginning of the **be/conf/hadoop\_env.sh** file of each BE or the **cn/conf/hadoop\_env.sh** file of each CN. After you set the username in these files, restart each FE and each BE or CN to make the parameter settings take effect. You can set only one username for each StarRocks cluster. * When you query Hudi data, the FEs and BEs or CNs of your StarRocks cluster use the HDFS client to access your HDFS cluster. In most cases, you do not need to configure your StarRocks cluster to achieve that purpose, and StarRocks starts the HDFS client using the default configurations. You need to configure your StarRocks cluster only in the following situations: * High availability (HA) is enabled for your HDFS cluster: Add the **hdfs-site.xml** file of your HDFS cluster to the **$FE\_HOME/conf** path of each FE and to the **$BE\_HOME/conf** path of each BE or the **$CN\_HOME/conf** path of each CN. * View File System (ViewFs) is enabled for your HDFS cluster: Add the **core-site.xml** file of your HDFS cluster to the **$FE\_HOME/conf** path of each FE and to the **$BE\_HOME/conf** path of each BE or the **$CN\_HOME/conf** path of each CN. > **NOTE** > > If an error indicating an unknown host is returned when you send a query, you must add the mapping between the host names and IP addresses of your HDFS cluster nodes to the **/etc/hosts** path. ##### Kerberos authentication[​](#kerberos-authentication "Direct link to Kerberos authentication") If Kerberos authentication is enabled for your HDFS cluster or Hive metastore, configure your StarRocks cluster as follows: * Run the `kinit -kt keytab_path principal` command on each FE and each BE or CN to obtain Ticket Granting Ticket (TGT) from Key Distribution Center (KDC). To run this command, you must have the permissions to access your HDFS cluster and Hive metastore. Note that accessing KDC with this command is time-sensitive. Therefore, you need to use cron to run this command periodically. * Add `JAVA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf"` to the **$FE\_HOME/conf/fe.conf** file of each FE and to the **$BE\_HOME/conf/be.conf** file of each BE or the **$CN\_HOME/conf/cn.conf** file of each CN. In this example, `/etc/krb5.conf` is the save path of the **krb5.conf** file. You can modify the path based on your needs. #### Create a Hudi catalog[​](#create-a-hudi-catalog "Direct link to Create a Hudi catalog") ##### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE EXTERNAL CATALOG [COMMENT ] PROPERTIES ( "type" = "hudi", MetastoreParams, StorageCredentialParams, MetadataUpdateParams ) ``` ##### Parameters[​](#parameters "Direct link to Parameters") ###### catalog\_name[​](#catalog_name "Direct link to catalog_name") The name of the Hudi catalog. The naming conventions are as follows: * The name can contain letters, digits (0-9), and underscores (\_). It must start with a letter. * The name is case-sensitive and cannot exceed 1023 characters in length. ###### comment[​](#comment "Direct link to comment") The description of the Hudi catalog. This parameter is optional. ###### type[​](#type "Direct link to type") The type of your data source. Set the value to `hudi`. ###### MetastoreParams[​](#metastoreparams "Direct link to MetastoreParams") A set of parameters about how StarRocks integrates with the metastore of your data source. ###### Hive metastore[​](#hive-metastore "Direct link to Hive metastore") If you choose Hive metastore as the metastore of your data source, configure `MetastoreParams` as follows: ```sql "hive.metastore.type" = "hive", "hive.metastore.uris" = "" ``` > **NOTE** > > Before querying Hudi data, you must add the mapping between the host names and IP addresses of your Hive metastore nodes to the `/etc/hosts` path. Otherwise, StarRocks may fail to access your Hive metastore when you start a query. The following table describes the parameter you need to configure in `MetastoreParams`. | Parameter | Required | Description | | ------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | hive.metastore.type | Yes | The type of metastore that you use for your Hudi cluster. Set the value to `hive`. | | hive.metastore.uris | Yes | The URI of your Hive metastore. Format: `thrift://:`.
If high availability (HA) is enabled for your Hive metastore, you can specify multiple metastore URIs and separate them with commas (`,`), for example, `"thrift://:,thrift://:,thrift://:"`. | ###### AWS Glue[​](#aws-glue "Direct link to AWS Glue") If you choose AWS Glue as the metastore of your data source, which is supported only when you choose AWS S3 as storage, take one of the following actions: * To choose the instance profile-based authentication method, configure `MetastoreParams` as follows: ```sql "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.region" = "" ``` * To choose the assumed role-based authentication method, configure `MetastoreParams` as follows: ```sql "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.iam_role_arn" = "", "aws.glue.region" = "" ``` * To choose the IAM user-based authentication method, configure `MetastoreParams` as follows: ```sql "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "false", "aws.glue.access_key" = "", "aws.glue.secret_key" = "", "aws.glue.region" = "" ``` The following table describes the parameters you need to configure in `MetastoreParams`. | Parameter | Required | Description | | ------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | hive.metastore.type | Yes | The type of metastore that you use for your Hudi cluster. Set the value to `glue`. | | aws.glue.use\_instance\_profile | Yes | Specifies whether to enable the instance profile-based authentication method and the assumed role-based authentication method. Valid values: `true` and `false`. Default value: `false`. | | aws.glue.iam\_role\_arn | No | The ARN of the IAM role that has privileges on your AWS Glue Data Catalog. If you use the assumed role-based authentication method to access AWS Glue, you must specify this parameter. | | aws.glue.region | Yes | The region in which your AWS Glue Data Catalog resides. Example: `us-west-1`. | | aws.glue.access\_key | No | The access key of your AWS IAM user. If you use the IAM user-based authentication method to access AWS Glue, you must specify this parameter. | | aws.glue.secret\_key | No | The secret key of your AWS IAM user. If you use the IAM user-based authentication method to access AWS Glue, you must specify this parameter. | For information about how to choose an authentication method for accessing AWS Glue and how to configure an access control policy in the AWS IAM Console, see [Authentication parameters for accessing AWS Glue](https://docs.starrocks.io/docs/integrations/authenticate_to_aws_resources.md#authentication-parameters-for-accessing-aws-glue). ###### StorageCredentialParams[​](#storagecredentialparams "Direct link to StorageCredentialParams") A set of parameters about how StarRocks integrates with your storage system. This parameter set is optional. If you use HDFS as storage, you do not need to configure `StorageCredentialParams`. If you use AWS S3, other S3-compatible storage system, Microsoft Azure Storage, or Google GCS as storage, you must configure `StorageCredentialParams`. ###### AWS S3[​](#aws-s3 "Direct link to AWS S3") If you choose AWS S3 as storage for your Hudi cluster, take one of the following actions: * To choose the instance profile-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "" ``` * To choose the assumed role-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "", "aws.s3.region" = "" ``` * To choose the IAM user-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ----------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | aws.s3.use\_instance\_profile | Yes | Specifies whether to enable the instance profile-based authentication method and the assumed role-based authentication method. Valid values: `true` and `false`. Default value: `false`. | | aws.s3.iam\_role\_arn | No | The ARN of the IAM role that has privileges on your AWS S3 bucket. If you use the assumed role-based authentication method to access AWS S3, you must specify this parameter. | | aws.s3.region | Yes | The region in which your AWS S3 bucket resides. Example: `us-west-1`. | | aws.s3.access\_key | No | The access key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter. | | aws.s3.secret\_key | No | The secret key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter. | For information about how to choose an authentication method for accessing AWS S3 and how to configure an access control policy in AWS IAM Console, see [Authentication parameters for accessing AWS S3](https://docs.starrocks.io/docs/integrations/authenticate_to_aws_resources.md#authentication-parameters-for-accessing-aws-s3). ###### S3-compatible storage system[​](#s3-compatible-storage-system "Direct link to S3-compatible storage system") Hudi catalogs support S3-compatible storage systems from v2.5 onwards. If you choose an S3-compatible storage system, such as MinIO, as storage for your Hudi cluster, configure `StorageCredentialParams` as follows to ensure a successful integration: ```sql "aws.s3.enable_ssl" = "false", "aws.s3.enable_path_style_access" = "true", "aws.s3.endpoint" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ---------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | aws.s3.enable\_ssl | Yes | Specifies whether to enable SSL connection.
Valid values: `true` and `false`. Default value: `true`. | | aws.s3.enable\_path\_style\_access | Yes | Specifies whether to enable path-style access.
Valid values: `true` and `false`. Default value: `false`. For MinIO, you must set the value to `true`.
Path-style URLs use the following format: `https://s3..amazonaws.com//`. For example, if you create a bucket named `DOC-EXAMPLE-BUCKET1` in the US West (Oregon) Region, and you want to access the `alice.jpg` object in that bucket, you can use the following path-style URL: `https://s3.us-west-2.amazonaws.com/DOC-EXAMPLE-BUCKET1/alice.jpg`. | | aws.s3.endpoint | Yes | The endpoint that is used to connect to your S3-compatible storage system instead of AWS S3. | | aws.s3.access\_key | Yes | The access key of your IAM user. | | aws.s3.secret\_key | Yes | The secret key of your IAM user. | ###### Microsoft Azure Storage[​](#microsoft-azure-storage "Direct link to Microsoft Azure Storage") Hudi catalogs support Microsoft Azure Storage from v3.0 onwards. ###### Azure Blob Storage[​](#azure-blob-storage "Direct link to Azure Blob Storage") If you choose Blob Storage as storage for your Hudi cluster, take one of the following actions: * To choose the Shared Key authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.blob.storage_account" = "", "azure.blob.shared_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | --------------------------- | ------------ | -------------------------------------------- | | azure.blob.storage\_account | Yes | The username of your Blob Storage account. | | azure.blob.shared\_key | Yes | The shared key of your Blob Storage account. | * To choose the SAS Token authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.blob.storage_account" = "", "azure.blob.container" = "", "azure.blob.sas_token" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | --------------------------- | ------------ | --------------------------------------------------------------- | | azure.blob.storage\_account | Yes | The username of your Blob Storage account. | | azure.blob.container | Yes | The name of the blob container that stores your data. | | azure.blob.sas\_token | Yes | The SAS token that is used to access your Blob Storage account. | ###### Azure Data Lake Storage Gen2[​](#azure-data-lake-storage-gen2 "Direct link to Azure Data Lake Storage Gen2") If you choose Data Lake Storage Gen2 as storage for your Hudi cluster, take one of the following actions: * To choose the Managed Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------------------ | ------------ | ------------------------------------------------------------------------------------------------ | | azure.adls2.oauth2\_use\_managed\_identity | Yes | Specifies whether to enable the Managed Identity authentication method. Set the value to `true`. | | azure.adls2.oauth2\_tenant\_id | Yes | The ID of the tenant whose data you want to access. | | azure.adls2.oauth2\_client\_id | Yes | The client (application) ID of the managed identity. | * To choose the Shared Key authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.storage_account" = "", "azure.adls2.shared_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ---------------------------- | ------------ | -------------------------------------------------------------- | | azure.adls2.storage\_account | Yes | The username of your Data Lake Storage Gen2 storage account. | | azure.adls2.shared\_key | Yes | The shared key of your Data Lake Storage Gen2 storage account. | * To choose the Service Principal authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_client_secret" = "", "azure.adls2.oauth2_client_endpoint" = "" ``` The following table describes the parameters you need to configure `in StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------------ | ------------ | -------------------------------------------------------------------------- | | azure.adls2.oauth2\_client\_id | Yes | The client (application) ID of the service principal. | | azure.adls2.oauth2\_client\_secret | Yes | The value of the new client (application) secret created. | | azure.adls2.oauth2\_client\_endpoint | Yes | The OAuth 2.0 token endpoint (v1) of the service principal or application. | * To choose the Workload Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_token_file" = "", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | azure.adls2.oauth2\_token\_file | Yes | The absolute file path to the OAuth2 token file projected into the pod by the Azure Workload Identity webhook. | | azure.adls2.oauth2\_tenant\_id | Yes | The ID of the tenant whose data you want to access. | | azure.adls2.oauth2\_client\_id | Yes | The client ID (application ID) of the Azure AD application (user-assigned managed identity or app registration) associated with the workload identity. | ###### Azure Data Lake Storage Gen1[​](#azure-data-lake-storage-gen1 "Direct link to Azure Data Lake Storage Gen1") If you choose Data Lake Storage Gen1 as storage for your Hudi cluster, take one of the following actions: * To choose the Managed Service Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls1.use_managed_service_identity" = "true" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------------------- | ------------ | -------------------------------------------------------------------------------------------------------- | | azure.adls1.use\_managed\_service\_identity | Yes | Specifies whether to enable the Managed Service Identity authentication method. Set the value to `true`. | * To choose the Service Principal authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls1.oauth2_client_id" = "", "azure.adls1.oauth2_credential" = "", "azure.adls1.oauth2_endpoint" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------ | ------------ | -------------------------------------------------------------------------- | | azure.adls1.oauth2\_client\_id | Yes | The client (application) ID of the service principal. | | azure.adls1.oauth2\_credential | Yes | The value of the new client (application) secret created. | | azure.adls1.oauth2\_endpoint | Yes | The OAuth 2.0 token endpoint (v1) of the service principal or application. | ###### Google GCS[​](#google-gcs "Direct link to Google GCS") Hudi catalogs support Google GCS from v3.0 onwards. If you choose Google GCS as storage for your Hudi cluster, take one of the following actions: * To choose the VM-based authentication method, configure `StorageCredentialParams` as follows: ```sql "gcp.gcs.use_compute_engine_service_account" = "true" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ---------------------------------------------- | ----------------- | --------------------- | ------------------------------------------------------------------------------------------- | | gcp.gcs.use\_compute\_engine\_service\_account | false | true | Specifies whether to directly use the service account that is bound to your Compute Engine. | * To choose the service account-based authentication method, configure `StorageCredentialParams` as follows: ```sql "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ------------------------------------------ | ----------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------- | | gcp.gcs.service\_account\_email | "" | "" | The email address in the JSON file generated at the creation of the service account. | | gcp.gcs.service\_account\_private\_key\_id | "" | "61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea" | The private key ID in the JSON file generated at the creation of the service account. | | gcp.gcs.service\_account\_private\_key | "" | "-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n" | The private key in the JSON file generated at the creation of the service account. | * To choose the impersonation-based authentication method, configure `StorageCredentialParams` as follows: * Make a VM instance impersonate a service account: ```sql "gcp.gcs.use_compute_engine_service_account" = "true", "gcp.gcs.impersonation_service_account" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ---------------------------------------------- | ----------------- | --------------------- | ------------------------------------------------------------------------------------------- | | gcp.gcs.use\_compute\_engine\_service\_account | false | true | Specifies whether to directly use the service account that is bound to your Compute Engine. | | gcp.gcs.impersonation\_service\_account | "" | "hello" | The service account that you want to impersonate. | * Make a service account (temporarily named as meta service account) impersonate another service account (temporarily named as data service account): ```sql "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "", "gcp.gcs.impersonation_service_account" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ------------------------------------------ | ----------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------ | | gcp.gcs.service\_account\_email | "" | "" | The email address in the JSON file generated at the creation of the meta service account. | | gcp.gcs.service\_account\_private\_key\_id | "" | "61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea" | The private key ID in the JSON file generated at the creation of the meta service account. | | gcp.gcs.service\_account\_private\_key | "" | "-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n" | The private key in the JSON file generated at the creation of the meta service account. | | gcp.gcs.impersonation\_service\_account | "" | "hello" | The data service account that you want to impersonate. | ###### MetadataUpdateParams[​](#metadataupdateparams "Direct link to MetadataUpdateParams") A set of parameters about how StarRocks updates the cached metadata of Hudi. This parameter set is optional. StarRocks implements the [automatic asynchronous update policy](#appendix-understand-metadata-automatic-asynchronous-update) by default. In most cases, you can ignore `MetadataUpdateParams` and do not need to tune the policy parameters in it, because the default values of these parameters already provide you with an out-of-the-box performance. However, if the frequency of data updates in Hudi is high, you can tune these parameters to further optimize the performance of automatic asynchronous updates. > **NOTE** > > In most cases, if your Hudi data is updated at a granularity of 1 hour or less, the data update frequency is considered high. | Parameter | Required | Description | | ------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | enable\_metastore\_cache | No | Specifies whether StarRocks caches the metadata of Hudi tables. Valid values: `true` and `false`. Default value: `true`. The value `true` enables the cache, and the value `false` disables the cache. | | enable\_remote\_file\_cache | No | Specifies whether StarRocks caches the metadata of the underlying data files of Hudi tables or partitions. Valid values: `true` and `false`. Default value: `true`. The value `true` enables the cache, and the value `false` disables the cache. | | metastore\_cache\_refresh\_interval\_sec | No | The time interval at which StarRocks asynchronously updates the metadata of Hudi tables or partitions cached in itself. Unit: seconds. Default value: `7200`, which is 2 hours. | | remote\_file\_cache\_refresh\_interval\_sec | No | The time interval at which StarRocks asynchronously updates the metadata of the underlying data files of Hudi tables or partitions cached in itself. Unit: seconds. Default value: `60`. | | metastore\_cache\_ttl\_sec | No | The time interval at which StarRocks automatically discards the metadata of Hudi tables or partitions cached in itself. Unit: seconds. Default value: `86400`, which is 24 hours. | | remote\_file\_cache\_ttl\_sec | No | The time interval at which StarRocks automatically discards the metadata of the underlying data files of Hudi tables or partitions cached in itself. Unit: seconds. Default value: `129600`, which is 36 hours. | ##### Examples[​](#examples "Direct link to Examples") The following examples create a Hudi catalog named `hudi_catalog_hms` or `hudi_catalog_glue`, depending on the type of metastore you use, to query data from your Hudi cluster. ###### HDFS[​](#hdfs-1 "Direct link to HDFS") If you use HDFS as storage, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083" ); ``` ###### AWS S3[​](#aws-s3-1 "Direct link to AWS S3") ###### If you choose instance profile-based credential[​](#if-you-choose-instance-profile-based-credential "Direct link to If you choose instance profile-based credential") * If you use Hive metastore in your Hudi cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "us-west-2" ); ``` * If you use AWS Glue in your Amazon EMR Hudi cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_glue PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.region" = "us-west-2", "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "us-west-2" ); ``` ###### If you choose assumed role-based credential[​](#if-you-choose-assumed-role-based-credential "Direct link to If you choose assumed role-based credential") * If you use Hive metastore in your Hudi cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::081976408565:role/test_s3_role", "aws.s3.region" = "us-west-2" ); ``` * If you use AWS Glue in your Amazon EMR Hudi cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_glue PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.iam_role_arn" = "arn:aws:iam::081976408565:role/test_glue_role", "aws.glue.region" = "us-west-2", "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::081976408565:role/test_s3_role", "aws.s3.region" = "us-west-2" ); ``` ###### If you choose IAM user-based credential[​](#if-you-choose-iam-user-based-credential "Direct link to If you choose IAM user-based credential") * If you use Hive metastore in your Hudi cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "us-west-2" ); ``` * If you use AWS Glue in your Amazon EMR Hudi cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_glue PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "false", "aws.glue.access_key" = "", "aws.glue.secret_key" = "", "aws.glue.region" = "us-west-2", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "us-west-2" ); ``` ###### S3-compatible storage system[​](#s3-compatible-storage-system-1 "Direct link to S3-compatible storage system") Use MinIO as an example. Run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.enable_ssl" = "true", "aws.s3.enable_path_style_access" = "true", "aws.s3.endpoint" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ); ``` ###### Microsoft Azure Storage[​](#microsoft-azure-storage-1 "Direct link to Microsoft Azure Storage") ###### Azure Blob Storage[​](#azure-blob-storage-1 "Direct link to Azure Blob Storage") * If you choose the Shared Key authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.blob.storage_account" = "", "azure.blob.shared_key" = "" ); ``` * If you choose the SAS Token authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.blob.storage_account" = "", "azure.blob.container" = "", "azure.blob.sas_token" = "" ); ``` ###### Azure Data Lake Storage Gen1[​](#azure-data-lake-storage-gen1-1 "Direct link to Azure Data Lake Storage Gen1") * If you choose the Managed Service Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls1.use_managed_service_identity" = "true" ); ``` * If you choose the Service Principal authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls1.oauth2_client_id" = "", "azure.adls1.oauth2_credential" = "", "azure.adls1.oauth2_endpoint" = "" ); ``` ###### Azure Data Lake Storage Gen2[​](#azure-data-lake-storage-gen2-1 "Direct link to Azure Data Lake Storage Gen2") * If you choose the Managed Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` * If you choose the Shared Key authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.storage_account" = "", "azure.adls2.shared_key" = "" ); ``` * If you choose the Service Principal authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_client_secret" = "", "azure.adls2.oauth2_client_endpoint" = "" ); ``` * If you choose the Workload Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_token_file" = "/var/run/secrets/azure/tokens/azure-identity-token", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` ###### Google GCS[​](#google-gcs-1 "Direct link to Google GCS") * If you choose the VM-based authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.use_compute_engine_service_account" = "true" ); ``` * If you choose the service account-based authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "" ); ``` * If you choose the impersonation-based authentication method: * If you make a VM instance impersonate a service account, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.use_compute_engine_service_account" = "true", "gcp.gcs.impersonation_service_account" = "" ); ``` * If you make a service account impersonate another service account, run a command like below: ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "", "gcp.gcs.impersonation_service_account" = "" ); ``` #### View Hudi catalogs[​](#view-hudi-catalogs "Direct link to View Hudi catalogs") You can use [SHOW CATALOGS](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CATALOGS.md) to query all catalogs in the current StarRocks cluster: ```sql SHOW CATALOGS; ``` You can also use [SHOW CREATE CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CREATE_CATALOG.md) to query the creation statement of an external catalog. The following example queries the creation statement of a Hudi catalog named `hudi_catalog_glue`: ```sql SHOW CREATE CATALOG hudi_catalog_glue; ``` #### Switch to a Hudi Catalog and a database in it[​](#switch-to-a-hudi-catalog-and-a-database-in-it "Direct link to Switch to a Hudi Catalog and a database in it") You can use one of the following methods to switch to a Hudi catalog and a database in it: * Use [SET CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SET_CATALOG.md) to specify a Hudi catalog in the current session, and then use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to specify an active database: ```sql -- Switch to a specified catalog in the current session: SET CATALOG -- Specify the active database in the current session: USE ``` * Directly use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to switch to a Hudi catalog and a database in it: ```sql USE . ``` #### Drop a Hudi catalog[​](#drop-a-hudi-catalog "Direct link to Drop a Hudi catalog") You can use [DROP CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/DROP_CATALOG.md) to drop an external catalog. The following example drops a Hudi catalog named `hudi_catalog_glue`: ```sql DROP Catalog hudi_catalog_glue; ``` #### View the schema of a Hudi table[​](#view-the-schema-of-a-hudi-table "Direct link to View the schema of a Hudi table") You can use one of the following syntaxes to view the schema of a Hudi table: * View schema ```sql DESC[RIBE] .. ``` * View schema and location from the CREATE statement ```sql SHOW CREATE TABLE .. ``` #### Query a Hudi table[​](#query-a-hudi-table "Direct link to Query a Hudi table") 1. Use [SHOW DATABASES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATABASES.md) to view the databases in your Hudi cluster: ```sql SHOW DATABASES FROM ``` 2. [Switch to a Hudi Catalog and a database in it](#switch-to-a-hudi-catalog-and-a-database-in-it). 3. Use [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md) to query the destination table in the specified database: ```sql SELECT count(*) FROM LIMIT 10 ``` #### Load data from Hudi[​](#load-data-from-hudi "Direct link to Load data from Hudi") Suppose you have an OLAP table named `olap_tbl`, you can transform and load data like below: ```sql INSERT INTO default_catalog.olap_db.olap_tbl SELECT * FROM hudi_table ``` #### Manually or automatically update metadata cache[​](#manually-or-automatically-update-metadata-cache "Direct link to Manually or automatically update metadata cache") ##### Manual update[​](#manual-update "Direct link to Manual update") By default, StarRocks caches the metadata of Hudi and automatically updates the metadata in asynchronous mode to deliver better performance. Additionally, after some schema changes or table updates are made on a Hudi table, you can also use [REFRESH EXTERNAL TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/REFRESH_EXTERNAL_TABLE.md) to manually update its metadata, thereby ensuring that StarRocks can obtain up-to-date metadata at its earliest opportunity and generate appropriate execution plans: ```sql REFRESH EXTERNAL TABLE [PARTITION ('partition_name', ...)] ``` #### Appendix: Understand metadata automatic asynchronous update[​](#appendix-understand-metadata-automatic-asynchronous-update "Direct link to Appendix: Understand metadata automatic asynchronous update") Automatic asynchronous update is the default policy that StarRocks uses to update the metadata in Hudi catalogs. By default (namely, when the `enable_metastore_cache` and `enable_remote_file_cache` parameters are both set to `true`), if a query hits a partition of a Hudi table, StarRocks automatically caches the metadata of the partition and the metadata of the underlying data files of the partition. The cached metadata is updated by using the lazy update policy. For example, there is a Hudi table named `table2`, which has four partitions: `p1`, `p2`, `p3`, and `p4`. A query hits `p1`, and StarRocks caches the metadata of `p1` and the metadata of the underlying data files of `p1`. Assume that the default time intervals to update and discard the cached metadata are as follows: * The time interval (specified by the `metastore_cache_refresh_interval_sec` parameter) to asynchronously update the cached metadata of `p1` is 2 hours. * The time interval (specified by the `remote_file_cache_refresh_interval_sec` parameter) to asynchronously update the cached metadata of the underlying data files of `p1` is 60 seconds. * The time interval (specified by the `metastore_cache_ttl_sec` parameter) to automatically discard the cached metadata of `p1` is 24 hours. * The time interval (specified by the `remote_file_cache_ttl_sec` parameter) to automatically discard the cached metadata of the underlying data files of `p1` is 36 hours. The following figure shows the time intervals on a timeline for easier understanding. ![Timeline for updating and discarding cached metadata](/assets/images/catalog_timeline-152d44a61a8d93f0f6ea1442dfe9b91f.png) Then StarRocks updates or discards the metadata in compliance with the following rules: * If another query hits `p1` again and the current time from the last update is less than 60 seconds, StarRocks does not update the cached metadata of `p1` or the cached metadata of the underlying data files of `p1`. * If another query hits `p1` again and the current time from the last update is more than 60 seconds, StarRocks updates the cached metadata of the underlying data files of `p1`. * If another query hits `p1` again and the current time from the last update is more than 2 hours, StarRocks updates the cached metadata of `p1`. * If `p1` has not been accessed within 24 hours from the last update, StarRocks discards the cached metadata of `p1`. The metadata will be cached at the next query. * If `p1` has not been accessed within 36 hours from the last update, StarRocks discards the cached metadata of the underlying data files of `p1`. The metadata will be cached at the next query. --- ### Iceberg DDL operations StarRocks Iceberg Catalog supports a variety of Data Definition Language (DDL) operations, including creating and managing databases, tables, and views. You must have the appropriate privileges to perform DDL operations. For more information about privileges, see [Privileges](https://docs.starrocks.io/docs/administration/user_privs/authorization/privilege_item.md). #### CREATE DATABASE[​](#create-database "Direct link to CREATE DATABASE") Creates a database in an Iceberg catalog. This feature is supported from v3.1 onwards. ##### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE DATABASE [IF NOT EXISTS] [PROPERTIES ("location" = ":////")] ``` ##### Parameters[​](#parameters "Direct link to Parameters") `location`: Specifies the file path where the database will be created. Both HDFS and cloud storage are supported. If not specified, the database is created in the default file path of the Iceberg catalog. The `prefix` varies based on the storage system: * HDFS: `hdfs` * Google GCS: `gs` * Azure Blob Storage (HTTP): `wasb` * Azure Blob Storage (HTTPS): `wasbs` * Azure Data Lake Storage Gen1: `adl` * Azure Data Lake Storage Gen2 (HTTP): `abfs` * Azure Data Lake Storage Gen2 (HTTPS): `abfss` * AWS S3 or S3-compatible storage: `s3` ##### Example[​](#example "Direct link to Example") ```sql CREATE DATABASE iceberg_db PROPERTIES ("location" = "s3://my_bucket/iceberg_db/"); ``` #### DROP DATABASE[​](#drop-database "Direct link to DROP DATABASE") Drops an empty database from an Iceberg catalog. This feature is supported from v3.1 onwards. note Only empty databases can be dropped. When you drop a database, the file path in the remote storage is not deleted. ##### Syntax[​](#syntax-1 "Direct link to Syntax") ```sql DROP DATABASE [IF EXISTS] ``` ##### Example[​](#example-1 "Direct link to Example") ```sql DROP DATABASE iceberg_db; ``` #### CREATE TABLE[​](#create-table "Direct link to CREATE TABLE") Creates a table in an Iceberg database. This feature is supported from v3.1 onwards. ##### Syntax[​](#syntax-2 "Direct link to Syntax") ```sql CREATE TABLE [IF NOT EXISTS] [database.]table_name ( column_definition1[, column_definition2, ...], partition_column_definition1, partition_column_definition2, ... ) [partition_desc] [ORDER BY sort_desc] [PROPERTIES ("key" = "value", ...)] [AS SELECT query] ``` ##### Parameters[​](#parameters-1 "Direct link to Parameters") ###### `column_definition`[​](#column_definition "Direct link to column_definition") ```sql col_name col_type [COMMENT 'comment'] [DEFAULT default_value] ``` note All non-partition columns must use `NULL` as the default value. Partition columns must be defined after non-partition columns and cannot use `NULL` as the default value. ###### Default values[​](#default-values "Direct link to Default values") From v4.1 onwards, StarRocks supports setting default values for columns in Iceberg tables. This feature requires Iceberg format version 3 (`"format-version" = "3"`). **Usage:** * **Write-time filling**: When executing an INSERT statement, if a value is not specified for a column, the system automatically uses the column's default value. * **Schema Evolution filling**: When a new column is added to an existing table, reading old data files (that do not contain the new column) will use the new column's default value. **Syntax:** ```sql col_name col_type DEFAULT default_value ``` **Requirements:** * The table must use Iceberg format version 3 (`"format-version" = "3"`). * Default values for numeric types (INT, BIGINT, FLOAT, DOUBLE), BOOLEAN, STRING, and DATE/TIMESTAMP types must be wrapped in quotes. For example: `DEFAULT "18"`, `DEFAULT "100.0"`, `DEFAULT "true"`. **Examples:** * **Create a table with default values:** ```sql CREATE TABLE user_info ( id INT, name STRING, age INT DEFAULT "18", score DOUBLE DEFAULT "100.0", status STRING DEFAULT 'active', is_active BOOLEAN DEFAULT "true" ) PROPERTIES ("format-version" = "3"); ``` * **Add a column with a default value:** ```sql ALTER TABLE user_info ADD COLUMN bonus DOUBLE DEFAULT "50.5"; ``` * **Modify a column's default value:** ```sql ALTER TABLE user_info MODIFY COLUMN status STRING DEFAULT "inactive"; ``` ###### `partition_desc`[​](#partition_desc "Direct link to partition_desc") ```sql PARTITION BY (partition_expr[, partition_expr...]) ``` Each `partition_expr` can be: * `column_name` (identity transform) * `transform_expr(column_name)` * `transform_expr(column_name, parameter)` StarRocks supports partition transformation expressions defined in the Apache Iceberg specification. note Partition columns support all data types except FLOAT, DOUBLE, DECIMAL, and DATETIME. ###### `ORDER BY`[​](#order-by "Direct link to order-by") Specifies sort keys for the Iceberg table. This feature is supported from v4.0 onwards. ```sql ORDER BY (column_name [ASC | DESC] [NULLS FIRST | NULLS LAST], ...) ``` ###### `PROPERTIES`[​](#properties "Direct link to properties") Key table properties: * `location`: File path for the table. Required when using AWS Glue without database-level location. * `file_format`: File format. Only `parquet` (Default) is supported. * `compression_codec`: Compression algorithm. Options: SNAPPY, GZIP, ZSTD, LZ4 (Default: `zstd`). ##### Examples[​](#examples "Direct link to Examples") * **Create a non-partitioned table:** ```sql CREATE TABLE unpartition_tbl ( id int, score double ); ``` * **Create a partitioned table:** ```sql CREATE TABLE partition_tbl ( action varchar(20), id int, dt date ) PARTITION BY (id, dt); ``` * **Create a table with hidden partitions:** ```sql CREATE TABLE hidden_partition_tbl ( action VARCHAR(20), id INT, dt DATE ) PARTITION BY bucket(id, 10), year(dt); ``` * **CREATE TABLE AS SELECT:** ```sql CREATE TABLE new_tbl PARTITION BY (id, dt) AS SELECT * FROM existing_tbl; ``` #### ALTER TABLE to evolve partition spec[​](#alter-table-to-evolve-partition-spec "Direct link to ALTER TABLE to evolve partition spec") Modifies an Iceberg table's partition spec by adding or dropping partition columns. ##### Syntax[​](#syntax-3 "Direct link to Syntax") ```sql ALTER TABLE [catalog.][database.]table_name ADD PARTITION COLUMN partition_expr [, partition_expr ...]; ALTER TABLE [catalog.][database.]table_name DROP PARTITION COLUMN partition_expr [, partition_expr ...]; ``` Supported `partition_expr` formats: * Column name (identity transform) * Transform expressions: `year()`, `month()`, `day()`, `hour()`, `truncate()`, `bucket()` ##### Examples[​](#examples-1 "Direct link to Examples") * **Add partition columns:** ```sql ALTER TABLE sales_data ADD PARTITION COLUMN month(sale_date), bucket(customer_id, 10); ``` * **Drop partition column:** ```sql ALTER TABLE sales_data DROP PARTITION COLUMN day(sale_date); ``` #### DROP TABLE[​](#drop-table "Direct link to DROP TABLE") Drops an Iceberg table. This feature is supported from v3.1 onwards. When you drop a table, the file path and data in the remote storage are not deleted by default. ##### Syntax[​](#syntax-4 "Direct link to Syntax") ```sql DROP TABLE [IF EXISTS] [FORCE] ``` ##### Parameters[​](#parameters-2 "Direct link to Parameters") * `FORCE`: When specified, the table data in the remote storage is deleted, while the file path is retained. ##### Example[​](#example-2 "Direct link to Example") ```sql DROP TABLE iceberg_db.sales_data; -- Force drop the table with its data DROP TABLE iceberg_db.temp_data FORCE; ``` #### CREATE VIEW[​](#create-view "Direct link to CREATE VIEW") Creates an Iceberg view. This feature is supported from v3.5 onwards. Creating an Iceberg view with PROPERTIES is supported from v4.0.3 onwards. ##### Syntax[​](#syntax-5 "Direct link to Syntax") ```sql CREATE VIEW [IF NOT EXISTS] [..] ( [COMMENT 'column comment'] [, [COMMENT 'column comment'], ...] ) [COMMENT 'view comment'] [PROPERTIES ("key" = "value", ...)] AS ``` ##### Example[​](#example-3 "Direct link to Example") * **Create a regular Iceberg view:** ```sql CREATE VIEW IF NOT EXISTS iceberg_db.sales_summary AS SELECT region, SUM(amount) as total_sales FROM iceberg_db.sales GROUP BY region; ``` * **Create an Iceberg view with properties:** ```sql CREATE VIEW IF NOT EXISTS iceberg_db.sales_summary PROPERTIES ( "key1" = "value1" ) AS SELECT region, SUM(amount) as total_sales FROM iceberg_db.sales GROUP BY region; ``` #### ALTER VIEW to update StarRocks dialect[​](#alter-view-to-update-starrocks-dialect "Direct link to ALTER VIEW to update StarRocks dialect") Adds or modifies StarRocks dialect for an existing Iceberg view. This feature is supported from v3.5 onwards. note You can define only one StarRocks dialect for each Iceberg view. ##### Syntax[​](#syntax-6 "Direct link to Syntax") ```sql ALTER VIEW [..] ( [, ] ) { ADD | MODIFY } DIALECT ``` ##### Examples[​](#examples-2 "Direct link to Examples") * **Add StarRocks dialect:** ```sql ALTER VIEW iceberg_db.spark_view ADD DIALECT SELECT k1, k2 FROM iceberg_db.source_table; ``` * **Modify StarRocks dialect:** ```sql ALTER VIEW iceberg_db.spark_view MODIFY DIALECT SELECT k1, k2, k3 FROM iceberg_db.source_table; ``` --- ### Iceberg DML operations StarRocks Iceberg Catalog supports a variety of Data Manipulation Language (DML) operations, including inserting data into Iceberg tables. You must have the appropriate privileges to perform DML operations. For more information about privileges, see [Privileges](https://docs.starrocks.io/docs/administration/user_privs/authorization/privilege_item.md). #### INSERT[​](#insert "Direct link to INSERT") Inserts data into an Iceberg table. This feature is supported from v3.1 onwards. Similar to loading data into StarRocks native tables, if you have the [INSERT privilege](https://docs.starrocks.io/docs/administration/user_privs/authorization/privilege_item.md#table) on an Iceberg table, you can use the [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) statement to sink the data to the Iceberg table. Currently, only Parquet-formatted Iceberg tables are supported. ##### Syntax[​](#syntax "Direct link to Syntax") ```sql INSERT {INTO | OVERWRITE} [ (column_name [, ...]) ] { VALUES ( { expression | DEFAULT } [, ...] ) [, ...] | query } -- If you want to sink data to specified partitions, use the following syntax: INSERT {INTO | OVERWRITE} PARTITION (par_col1= [, par_col2=...]) { VALUES ( { expression | DEFAULT } [, ...] ) [, ...] | query } ``` note `NULL` values are not allowed in partition columns. Therefore, you must make sure that no empty values are loaded into the partition columns of the Iceberg table. ##### Parameters[​](#parameters "Direct link to Parameters") ###### INTO[​](#into "Direct link to INTO") Appends the data to the Iceberg table. ###### OVERWRITE[​](#overwrite "Direct link to OVERWRITE") Overwrites the existing data of the Iceberg table. ###### column\_name[​](#column_name "Direct link to column_name") The name of the destination column to which you want to load data. You can specify one or more columns. Multiple columns are separated with commas (`,`). * You can only specify columns that actually exist in the Iceberg table. * The destination columns must include the partition columns of the Iceberg table. * The destination columns are mapped one on one in sequence to the columns in the SELECT statement (source columns), regardless of what the destination column names are. * If no destination columns are specified, the data is loaded into all columns of the Iceberg table. * If a non-partition source column cannot be mapped to any destination column, StarRocks writes the default value `NULL` to the destination column. * If the data types of the source and destination columns mismatch, StarRocks performs an implicit conversion on the mismatched columns. If the conversion fails, a syntax parsing error will be returned. note You cannot specify the `column_name` property if you have specified the PARTITION clause. ###### expression[​](#expression "Direct link to expression") Expression that assigns values to the destination column. ###### DEFAULT[​](#default "Direct link to DEFAULT") Assigns a default value to the destination column. ###### query[​](#query "Direct link to query") Query statement whose result will be loaded into the Iceberg table. It can be any SQL statement supported by StarRocks. ###### PARTITION[​](#partition "Direct link to PARTITION") The partitions into which you want to load data. You must specify all partition columns of the Iceberg table in this property. The partition columns that you specify in this property can be in a different sequence than the partition columns that you have defined in the table creation statement. note You cannot specify the `column_name` property if you have specified the PARTITION clause. ##### Examples[​](#examples "Direct link to Examples") 1. Insert three data rows into the `partition_tbl_1` table: ```sql INSERT INTO partition_tbl_1 VALUES ("buy", 1, "2023-09-01"), ("sell", 2, "2023-09-02"), ("buy", 3, "2023-09-03"); ``` 2. Insert the result of a SELECT query, which contains simple computations, into the `partition_tbl_1` table: ```sql INSERT INTO partition_tbl_1 (id, action, dt) SELECT 1+1, 'buy', '2023-09-03'; ``` 3. Insert the result of a SELECT query, which reads data from the `partition_tbl_1` table, into the same table: ```sql INSERT INTO partition_tbl_1 SELECT 'buy', 1, date_add(dt, INTERVAL 2 DAY) FROM partition_tbl_1 WHERE id=1; ``` 4. Insert the result of a SELECT query into the partitions that meet two conditions, `dt='2023-09-01'` and `id=1`, of the `partition_tbl_2` table: ```sql INSERT INTO partition_tbl_2 SELECT 'order', 1, '2023-09-01'; ``` Or ```sql INSERT INTO partition_tbl_2 partition(dt='2023-09-01',id=1) SELECT 'order'; ``` 5. Overwrite all `action` column values in the partitions that meet two conditions, `dt='2023-09-01'` and `id=1`, of the `partition_tbl_1` table with `close`: ```sql INSERT OVERWRITE partition_tbl_1 SELECT 'close', 1, '2023-09-01'; ``` Or ```sql INSERT OVERWRITE partition_tbl_1 partition(dt='2023-09-01',id=1) SELECT 'close'; ``` #### DELETE[​](#delete "Direct link to DELETE") You can use the DELETE statement to delete data from Iceberg tables based on specified conditions. This feature is supported from v4.1 and later. ##### Syntax[​](#syntax-1 "Direct link to Syntax") ```sql DELETE FROM WHERE ``` ##### Parameters[​](#parameters-1 "Direct link to Parameters") * `table_name`: The name of the Iceberg table you want to delete data from. You can use: * Fully qualified name: `catalog_name.database_name.table_name` * Database-qualified name (after setting catalog): `database_name.table_name` * Table name only (after setting catalog and database): `table_name` * `condition`: The condition to identify which rows to delete. It can include: * Comparison operators: `=`, `!=`, `>`, `<`, `>=`, `<=`, `<>` * Logical operators: `AND`, `OR`, `NOT` * `IN` and `NOT IN` clauses * `BETWEEN` and `LIKE` operators * `IS NULL` and `IS NOT NULL` * Sub-queries with `IN` or `EXISTS` ##### Examples[​](#examples-1 "Direct link to Examples") ###### Basic DELETE operations[​](#basic-delete-operations "Direct link to Basic DELETE operations") Delete rows matching a simple condition: ```sql DELETE FROM iceberg_catalog.db.table1 WHERE id = 3; ``` ###### DELETE with IN and NOT IN[​](#delete-with-in-and-not-in "Direct link to DELETE with IN and NOT IN") Delete multiple rows using IN clause: ```sql DELETE FROM iceberg_catalog.db.table1 WHERE id IN (18, 20, 22); DELETE FROM iceberg_catalog.db.table1 WHERE id NOT IN (100, 101, 102); ``` ###### DELETE with logical operators[​](#delete-with-logical-operators "Direct link to DELETE with logical operators") Combine multiple conditions: ```sql DELETE FROM iceberg_catalog.db.table1 WHERE age > 30 AND salary < 70000; DELETE FROM iceberg_catalog.db.table1 WHERE status = 'inactive' OR last_login < '2023-01-01'; ``` ###### DELETE with pattern matching[​](#delete-with-pattern-matching "Direct link to DELETE with pattern matching") Use LIKE for pattern-based deletion: ```sql DELETE FROM iceberg_catalog.db.table1 WHERE name LIKE 'A%'; DELETE FROM iceberg_catalog.db.table1 WHERE email LIKE '%@example.com'; ``` ###### DELETE with range conditions[​](#delete-with-range-conditions "Direct link to DELETE with range conditions") Use BETWEEN for range-based deletion: ```sql DELETE FROM iceberg_catalog.db.table1 WHERE age BETWEEN 30 AND 40; DELETE FROM iceberg_catalog.db.table1 WHERE created_date BETWEEN '2023-01-01' AND '2023-12-31'; ``` ###### DELETE with NULL checks[​](#delete-with-null-checks "Direct link to DELETE with NULL checks") Delete rows with or without NULL values: ```sql DELETE FROM iceberg_catalog.db.table1 WHERE name IS NULL; DELETE FROM iceberg_catalog.db.table1 WHERE email IS NULL AND phone IS NULL; DELETE FROM iceberg_catalog.db.table1 WHERE age IS NOT NULL; ``` ###### DELETE with sub-queries[​](#delete-with-sub-queries "Direct link to DELETE with sub-queries") Use sub-queries to identify rows to delete: ```sql -- DELETE with IN sub-query DELETE FROM iceberg_catalog.db.table1 WHERE id IN (SELECT id FROM temp_table WHERE expired = true); -- DELETE with EXISTS sub-query DELETE FROM iceberg_catalog.db.table1 t1 WHERE EXISTS (SELECT user_id FROM inactive_users t2 WHERE t2.user_id = t1.user_id); ``` #### TRUNCATE[​](#truncate "Direct link to TRUNCATE") You can use the TRUNCATE TABLE statement to quickly delete all data from Iceberg tables. ##### Syntax[​](#syntax-2 "Direct link to Syntax") ```sql TRUNCATE TABLE ``` ##### Parameters[​](#parameters-2 "Direct link to Parameters") * `table_name`: The name of the Iceberg table that you want to truncate data from. You can use: * Fully qualified name: `catalog_name.database_name.table_name` * Database-qualified name (after setting catalog): `database_name.table_name` * Table name only (after setting catalog and database): `table_name` ##### Examples[​](#examples-2 "Direct link to Examples") ###### Example 1: Truncate a table using fully qualified name[​](#example-1-truncate-a-table-using-fully-qualified-name "Direct link to Example 1: Truncate a table using fully qualified name") ```sql TRUNCATE TABLE iceberg_catalog.my_db.my_table; ``` ###### Example 2: Truncate a table after setting catalog[​](#example-2-truncate-a-table-after-setting-catalog "Direct link to Example 2: Truncate a table after setting catalog") ```sql SET CATALOG iceberg_catalog; TRUNCATE TABLE my_db.my_table; ``` ###### Example 3: Truncate a table after setting catalog and database[​](#example-3-truncate-a-table-after-setting-catalog-and-database "Direct link to Example 3: Truncate a table after setting catalog and database") ```sql SET CATALOG iceberg_catalog; USE my_db; TRUNCATE TABLE my_table; ``` --- ### Iceberg catalog tip This example uses the Local Climatological Data(LCD) dataset featured in the [StarRocks Basics](https://docs.starrocks.io/docs/quick_start/shared-nothing.md) Quick Start. You can load the data and try the example yourself. An Iceberg catalog is a type of external catalog that is supported by StarRocks from v2.4 onwards. With Iceberg catalogs, you can: * Directly query data stored in Iceberg without the need to manually create tables. * Use [INSERT INTO](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) or asynchronous materialized views (which are supported from v2.5 onwards) to process data stored in Iceberg and load the data into StarRocks. * Perform operations on StarRocks to create or drop Iceberg databases and tables, or sink data from StarRocks tables to Parquet-formatted Iceberg tables by using [INSERT INTO](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) (this feature is supported from v3.1 onwards). To ensure successful SQL workloads on your Iceberg cluster, your StarRocks cluster must be able to access the storage system and metastore of your Iceberg cluster. StarRocks supports the following storage systems and metastores: * Distributed file system (HDFS) or object storage like AWS S3, Microsoft Azure Storage, Google GCS, or other S3-compatible storage system (for example, MinIO) * Metastore like Hive metastore, AWS Glue, or Tabular note * If you choose AWS S3 as storage, you can use HMS or AWS Glue as metastore. If you choose any other storage system, you can only use HMS as metastore. * If you choose Tabular as metastore, you need to use the Iceberg REST catalog. #### Usage notes[​](#usage-notes "Direct link to Usage notes") Take note of the following points when you use StarRocks to query data from Iceberg: | **File format** | **Compression format** | **Iceberg table version** | | --------------- | ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Parquet | SNAPPY, LZ4, ZSTD, GZIP, and NO\_COMPRESSION | - v1 tables: supported.
- v2 tables: supported from StarRocks v3.1 onwards in which queries on these v2 tables support position deletes. In v3.1.10, v3.2.5, v3.3 and their later versions, queries on v2 tables also support equality deletes. | | ORC | ZLIB, SNAPPY, LZO, LZ4, ZSTD, and NO\_COMPRESSION | - v1 tables: supported.
- v2 tables: supported from StarRocks v3.0 onwards in which queries on these v2 tables support position deletes. In v3.1.8, v3.2.3, v3.3 and their later versions, queries on v2 tables also support equality deletes. | #### Integration preparation[​](#integration-preparation "Direct link to Integration preparation") Before you create an Iceberg catalog, make sure your StarRocks cluster can integrate with the storage system and metastore of your Iceberg cluster. *** ##### Storage[​](#storage "Direct link to Storage") Select the tab that matches your storage type: * AWS S3 * HDFS If your Iceberg cluster uses AWS S3 as storage or AWS Glue as metastore, choose your suitable authentication method and make the required preparations to ensure that your StarRocks cluster can access the related AWS cloud resources. The following authentication methods are recommended: * Instance profile * Assumed role * IAM user Of the above-mentioned three authentication methods, instance profile is the most widely used. For more information, see [Preparation for authentication in AWS IAM](https://docs.starrocks.io/docs/integrations/authenticate_to_aws_resources.md#preparations). If you choose HDFS as storage, configure your StarRocks cluster as follows: * (Optional) Set the username that is used to access your HDFS cluster and Hive metastore. By default, StarRocks uses the username of the FE and BE or CN processes to access your HDFS cluster and Hive metastore. You can also set the username by adding `export HADOOP_USER_NAME=""` at the beginning of the **fe/conf/hadoop\_env.sh** file of each FE and at the beginning of the **be/conf/hadoop\_env.sh** file of each BE or the **cn/conf/hadoop\_env.sh** file of each CN. After you set the username in these files, restart each FE and each BE or CN to make the parameter settings take effect. You can set only one username for each StarRocks cluster. * When you query Iceberg data, the FEs and BEs or CNs of your StarRocks cluster use the HDFS client to access your HDFS cluster. In most cases, you do not need to configure your StarRocks cluster to achieve that purpose, and StarRocks starts the HDFS client using the default configurations. You need to configure your StarRocks cluster only in the following situations: * High availability (HA) is enabled for your HDFS cluster: Add the **hdfs-site.xml** file of your HDFS cluster to the **$FE\_HOME/conf** path of each FE and to the **$BE\_HOME/conf** path of each BE or the **$CN\_HOME/conf** path of each CN. * View File System (ViewFs) is enabled for your HDFS cluster: Add the **core-site.xml** file of your HDFS cluster to the **$FE\_HOME/conf** path of each FE and to the **$BE\_HOME/conf** path of each BE or the **$CN\_HOME/conf** path of each CN. tip If an error indicating an unknown host is returned when you send a query, you must add the mapping between the host names and IP addresses of your HDFS cluster nodes to the **/etc/hosts** path. ###### Pass HDFS client configurations through Catalog PROPERTIES[​](#pass-hdfs-client-configurations-through-catalog-properties "Direct link to Pass HDFS client configurations through Catalog PROPERTIES") In addition to placing **hdfs-site.xml** under the **conf** directories of FEs/BEs/CNs, you can declare HDFS client configurations directly in the `PROPERTIES` of `CREATE EXTERNAL CATALOG` (for example, the HA-related keys `dfs.nameservices`, `dfs.ha.namenodes.`, `dfs.namenode.rpc-address..`, `dfs.client.failover.proxy.provider.`, `fs.defaultFS`, etc.). Both FEs and BEs/CNs receive these properties and apply them to the HDFS client. The main advantage of this approach is that **a single StarRocks cluster can access multiple independent HDFS HA clusters at the same time through different Iceberg Catalogs**. Because the **conf** directory of each FE/BE/CN can hold only one **hdfs-site.xml**, multiple HDFS HA clusters cannot coexist there. Passing the configuration through Catalog PROPERTIES lets each Catalog carry its own HA configuration, with no interference between Catalogs. For a complete HA example, see the [Examples - HDFS](#examples) section below. *** ###### Kerberos authentication[​](#kerberos-authentication "Direct link to Kerberos authentication") If Kerberos authentication is enabled for your HDFS cluster or Hive metastore, configure your StarRocks cluster as follows: * Run the `kinit -kt keytab_path principal` command on each FE and each BE or CN to obtain Ticket Granting Ticket (TGT) from Key Distribution Center (KDC). To run this command, you must have the permissions to access your HDFS cluster and Hive metastore. Note that accessing KDC with this command is time-sensitive. Therefore, you need to use cron to run this command periodically. * Add `JAVA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf"` to the **$FE\_HOME/conf/fe.conf** file of each FE and to the **$BE\_HOME/conf/be.conf** file of each BE or the **$CN\_HOME/conf/cn.conf** file of each CN. In this example, `/etc/krb5.conf` is the save path of the **krb5.conf** file. You can modify the path based on your needs. *** #### Create an Iceberg catalog[​](#create-an-iceberg-catalog "Direct link to Create an Iceberg catalog") ##### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE EXTERNAL CATALOG [COMMENT ] PROPERTIES ( "type" = "iceberg", [SecurityParams], MetastoreParams, StorageCredentialParams, MetadataRelatedParams ) ``` *** ##### Parameters[​](#parameters "Direct link to Parameters") ###### catalog\_name[​](#catalog_name "Direct link to catalog_name") The name of the Iceberg catalog. The naming conventions are as follows: * The name can contain letters, digits (0-9), and underscores (\_). It must start with a letter. * The name is case-sensitive and cannot exceed 1023 characters in length. ###### comment[​](#comment "Direct link to comment") The description of the Iceberg catalog. This parameter is optional. ###### type[​](#type "Direct link to type") The type of your data source. Set the value to `iceberg`. ###### SecurityParams[​](#securityparams "Direct link to SecurityParams") Parameter(s) about how StarRocks manages data access to the catalog. For detailed instructions on managing data access for Iceberg Catalogs, see [Security Setup for Iceberg REST Catalog](https://docs.starrocks.io/docs/data_source/catalog/iceberg/iceberg_rest_security.md). ###### catalog.access.control[​](#catalogaccesscontrol "Direct link to catalog.access.control") The data access control policy. Valid values: * `native` (Default): The StarRocks built-in data access control system is used. * `allowall`: All data access checks are delegated to the Catalog itself. * `ranger`: Data access checks are delegated to Apache Ranger. ###### MetastoreParams[​](#metastoreparams "Direct link to MetastoreParams") A set of parameters about how StarRocks integrates with the metastore of your data source. Choose the tab that matches your metastore type: * Hive metastore * AWS Glue * REST * JDBC ###### Hive metastore[​](#hive-metastore "Direct link to Hive metastore") If you choose Hive metastore as the metastore of your data source, configure `MetastoreParams` as follows: ```sql "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "" ``` note Before querying Iceberg data, you must add the mapping between the host names and IP addresses of your Hive metastore nodes to the `/etc/hosts` path. Otherwise, StarRocks may fail to access your Hive metastore when you start a query. The following table describes the parameter you need to configure in `MetastoreParams`. * `iceberg.catalog.type` * Required: Yes * Description: The type of metastore that you use for your Iceberg cluster. Set the value to `hive`. * `hive.metastore.uris` * Required: Yes * Description: The URI of your Hive metastore. Format: `thrift://:`.
If high availability (HA) is enabled for your Hive metastore, you can specify multiple metastore URIs and separate them with commas (`,`), for example, `"thrift://:,thrift://:,thrift://:"`. ###### AWS Glue[​](#aws-glue "Direct link to AWS Glue") If you choose AWS Glue as the metastore of your data source, which is supported only when you choose AWS S3 as storage, take one of the following actions: * To choose the instance profile-based authentication method, configure `MetastoreParams` as follows: ```sql "iceberg.catalog.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.region" = "" ``` * To choose the assumed role-based authentication method, configure `MetastoreParams` as follows: ```sql "iceberg.catalog.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.iam_role_arn" = "", "aws.glue.region" = "" ``` * To choose the IAM user-based authentication method, configure `MetastoreParams` as follows: ```sql "iceberg.catalog.type" = "glue", "aws.glue.use_instance_profile" = "false", "aws.glue.access_key" = "", "aws.glue.secret_key" = "", "aws.glue.region" = "" ``` `MetastoreParams` for AWS Glue: * `iceberg.catalog.type` * Required: Yes * Description: The type of metastore that you use for your Iceberg cluster. Set the value to `glue`. * `aws.glue.use_instance_profile` * Required: Yes * Description: Specifies whether to enable the instance profile-based authentication method and the assumed role-based authentication method. Valid values: `true` and `false`. Default value: `false`. * `aws.glue.iam_role_arn` * Required: No * Description: The ARN of the IAM role that has privileges on your AWS Glue Data Catalog. If you use the assumed role-based authentication method to access AWS Glue, you must specify this parameter. * `aws.glue.region` * Required: Yes * Description: The region in which your AWS Glue Data Catalog resides. Example: `us-west-1`. * `aws.glue.access_key` * Required: No * Description: The access key of your AWS IAM user. If you use the IAM user-based authentication method to access AWS Glue, you must specify this parameter. * `aws.glue.secret_key` * Required: No * Description: The secret key of your AWS IAM user. If you use the IAM user-based authentication method to access AWS Glue, you must specify this parameter. * `aws.glue.catalog_id` * Required: No * Description: The ID of the AWS Glue Data Catalog to use. When not specified, the catalog in the current AWS account is used. You must specify this parameter when you need to access a Glue Data Catalog in a different AWS account (cross-account access). For information about how to choose an authentication method for accessing AWS Glue and how to configure an access control policy in the AWS IAM Console, see [Authentication parameters for accessing AWS Glue](https://docs.starrocks.io/docs/integrations/authenticate_to_aws_resources.md#authentication-parameters-for-accessing-aws-glue). ###### REST[​](#rest "Direct link to REST") note For detailed instructions on creating an Iceberg REST catalog for S3 tables, see [Create Iceberg REST Catalog for AWS S3 tables](https://docs.starrocks.io/docs/data_source/catalog/iceberg/iceberg_rest_s3.md). If you use REST as metastore, you must specify the metastore type as REST (`"iceberg.catalog.type" = "rest"`). Configure `MetastoreParams` as follows: ```sql "iceberg.catalog.type" = "rest", "iceberg.catalog.uri" = "", "iceberg.catalog.security" = "oauth2", "iceberg.catalog.oauth2.credential" = "", "iceberg.catalog.warehouse" = "" ``` `MetastoreParams` for REST catalog: * `iceberg.catalog.type` * Required: Yes * Description: The type of metastore that you use for your Iceberg cluster. Set the value to `rest`. - * `iceberg.catalog.uri` * Required: Yes * Description: The URI of the REST service endpoint. Example: `https://api.tabular.io/ws`. - * `iceberg.catalog.view-endpoints-supported` * Required: No * Description: Whether to use the view endpoints to support view-related operations when the REST service of earlier versions that does not return endpoints in `CatalogConfig`. This parameter is used for backwards compatibility with REST servers of early versions. Default: `false`. * `iceberg.catalog.security` * Required: No * Description: The type of authorization protocol to use. Default: `NONE`. Valid values: `OAUTH2` and `JWT`. When this item is set to `OAUTH2`, either `token` or `credential` is required. When this item is set to `JWT`, the user is required to log in to the StarRocks cluster using the `JWT` method. You can omit `token` or `credential` and StarRocks will use the logged in user's JWT to access the catalog. * `iceberg.catalog.oauth2.token` * Required: No * Description: The bearer token used for interactions with the server. A `token` or `credential` is required for `OAUTH2` authorization protocol. Example: `AbCdEf123456`. * `iceberg.catalog.oauth2.credential` * Required: No * Description: The credential to exchange for a token in the OAuth2 client credentials flow with the server. A `token` or `credential` is required for `OAUTH2` authorization protocol. Example: `AbCdEf123456`. * `iceberg.catalog.oauth2.scope` * Required: No * Description: Scope to be used when communicating with the REST Catalog. Applicable only when `credential` is used. * `iceberg.catalog.oauth2.server-uri` * Required: No * Description: The endpoint to retrieve access token from OAuth2 Server. * `iceberg.catalog.vended-credentials-enabled` * Required: No * Description: Whether to use credentials provided by REST backend for file system access. Default: `true`. * `iceberg.catalog.warehouse` * Required: No * Description: The warehouse location or identifier of the Iceberg catalog. Example: `s3://my_bucket/warehouse_location` or `sandbox`. - * `iceberg.catalog.rest.nested-namespace-enabled` * Required: No * Description: Whether to support querying objects under nested namespace. Default: `false`. * `iceberg.catalog.rest.view-endpoints-enabled` * Required: No * Description: Whether to enable view endpoints for view-related operations. If set to `false`, view operations like `getView` will be disabled. Default: `true`. The following example creates an Iceberg catalog named `tabular` that uses Tabular as metastore: ```sql CREATE EXTERNAL CATALOG tabular PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "rest", "iceberg.catalog.uri" = "https://api.tabular.io/ws", "iceberg.catalog.oauth2.credential" = "t-5Ii8e3FIbT9m0:aaaa-3bbbbbbbbbbbbbbbbbbb", "iceberg.catalog.warehouse" = "sandbox" ); ``` The following example creates an Iceberg catalog named `smith_polaris` that uses Polaris as metastore: ```sql CREATE EXTERNAL CATALOG smith_polaris PROPERTIES ( "iceberg.catalog.uri" = "http://xxx.xx.xx.xxx:8181/api/catalog", "type" = "iceberg", "iceberg.catalog.type" = "rest", "iceberg.catalog.warehouse" = "starrocks_catalog", "iceberg.catalog.security" = "oauth2", "iceberg.catalog.oauth2.credential" = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "iceberg.catalog.oauth2.scope"='PRINCIPAL_ROLE:ALL' ); # `ns1.ns2.tpch_namespace` is a nested namespace create table smith_polaris.`ns1.ns2.tpch_namespace`.tbl (c1 string); mysql> select * from smith_polaris.`ns1.ns2.tpch_namespace`.tbl; +------+ | c1 | +------+ | 1 | | 2 | | 3 | +------+ 3 rows in set (0.34 sec) ``` The following example creates an Iceberg catalog named `r2` that uses Cloudflare R2 Data Catalog as metastore: ```sql CREATE EXTERNAL CATALOG r2 PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "rest", "iceberg.catalog.uri" = "", "iceberg.catalog.security" = "oauth2", "iceberg.catalog.oauth2.token" = "", "iceberg.catalog.warehouse" = "" ); SET CATALOG r2; CREATE DATABASE testdb; SHOW DATABASES FROM r2; +--------------------+ | Database | +--------------------+ | information_schema | | testdb | +--------------------+ 2 rows in set (0.66 sec) ``` The ``,``, and `` values are obtained from the [Cloudflare Dashboard as detailed here](https://developers.cloudflare.com/r2/data-catalog/get-started/#prerequisites). ###### JDBC[​](#jdbc "Direct link to JDBC") If you choose JDBC as the metastore of your data source, configure `MetastoreParams` as follows: ```sql "iceberg.catalog.type" = "jdbc", "iceberg.catalog.uri" = "", "iceberg.catalog.warehouse" = "" ``` The following table describes the parameter you need to configure in `MetastoreParams`. * `iceberg.catalog.type` * Required: Yes * Description: The type of metastore that you use for your Iceberg cluster. Set the value to `jdbc`. * `iceberg.catalog.uri` * Required: Yes * Description: The URI of your database. Format: `jdbc:[mysql\|postgresql]://:/`. * `iceberg.catalog.warehouse` * Required: Yes * Description: The warehouse location or identifier of the Iceberg catalog. Example: `s3://my_bucket/warehouse_location` . * `iceberg.catalog.jdbc.user` * Required: No * Description: The username for the database. * `iceberg.catalog.jdbc.password` * Required: No * Description: The password for the database. * `iceberg.catalog.jdbc.init-catalog-tables` * Required: No * Description: Whether to create the tables `iceberg_namespace_properties` and `iceberg_tables` for storing metadata in the database specified by `iceberg.catalog.uri`. The default value is `false`. Specify `true` if these two tables have not yet been created in the database specified by `iceberg.catalog.uri`. The following example creates an Iceberg catalog named `iceberg_jdbc` and uses JDBC as metastore: ```sql CREATE EXTERNAL CATALOG iceberg_jdbc PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "jdbc", "iceberg.catalog.warehouse" = "s3://my_bucket/warehouse_location", "iceberg.catalog.uri" = "jdbc:mysql://ip:port/db_name", "iceberg.catalog.jdbc.user" = "username", "iceberg.catalog.jdbc.password" = "password", "aws.s3.endpoint" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ); ``` If using MySQL or other custom JDBC drivers, the corresponding JAR files need to be placed in the `fe/lib` and `be/lib/jni-packages` directories. *** ###### `StorageCredentialParams`[​](#storagecredentialparams "Direct link to storagecredentialparams") A set of parameters about how StarRocks integrates with your storage system. This parameter set is optional. Note the following points: * If you use HDFS as storage, you do not need to configure `StorageCredentialParams` and can skip this section. If you use AWS S3, other S3-compatible storage system, Microsoft Azure Storage, or Google GCS as storage, you must configure `StorageCredentialParams`. * If you use Tabular as metastore, you do not need to configure `StorageCredentialParams` and can skip this section. If you use HMS or AWS Glue as metastore, you must configure `StorageCredentialParams`. Choose the tab that matches your storage type: * AWS S3 * HDFS * MinIO * Microsoft Azure Blob Storage * Google GCS ###### AWS S3[​](#aws-s3 "Direct link to AWS S3") If you choose AWS S3 as storage for your Iceberg cluster, take one of the following actions: * To choose the instance profile-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "" ``` * To choose the assumed role-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "", "aws.s3.region" = "" ``` * To choose the IAM user-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "" ``` * To choose vended credential (supported from v4.0 onwards) with the REST catalog, configure `StorageCredentialParams` as follows: ```sql "aws.s3.region" = "" ``` `StorageCredentialParams` for AWS S3: ###### aws.s3.use\_instance\_profile[​](#awss3use_instance_profile "Direct link to aws.s3.use_instance_profile") * Required: Yes * Description: Specifies whether to enable the instance profile-based authentication method and the assumed role-based authentication method. Valid values: `true` and `false`. Default value: `false`. ###### aws.s3.iam\_role\_arn[​](#awss3iam_role_arn "Direct link to aws.s3.iam_role_arn") * Required: No * Description: The ARN of the IAM role that has privileges on your AWS S3 bucket. If you use the assumed role-based authentication method to access AWS S3, you must specify this parameter. ###### aws.s3.region[​](#awss3region "Direct link to aws.s3.region") * Required: Yes * Description: The region in which your AWS S3 bucket resides. Example: `us-west-1`. ###### aws.s3.access\_key[​](#awss3access_key "Direct link to aws.s3.access_key") * Required: No * Description: The access key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter. ###### aws.s3.secret\_key[​](#awss3secret_key "Direct link to aws.s3.secret_key") * Required: No * Description: The secret key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter. For information about how to choose an authentication method for accessing AWS S3 and how to configure an access control policy in AWS IAM Console, see [Authentication parameters for accessing AWS S3](https://docs.starrocks.io/docs/integrations/authenticate_to_aws_resources.md#authentication-parameters-for-accessing-aws-s3). When using HDFS storage skip the storage credentials. ###### S3-compatible storage system[​](#s3-compatible-storage-system "Direct link to S3-compatible storage system") Iceberg catalogs support S3-compatible storage systems from v2.5 onwards. If you choose an S3-compatible storage system, such as MinIO, as storage for your Iceberg cluster, configure `StorageCredentialParams` as follows to ensure a successful integration: ```sql "aws.s3.enable_ssl" = "false", "aws.s3.enable_path_style_access" = "true", "aws.s3.endpoint" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ``` `StorageCredentialParams` for MinIO and other S3 compatible systems: ###### aws.s3.enable\_ssl[​](#awss3enable_ssl "Direct link to aws.s3.enable_ssl") * Required: Yes * Description: Specifies whether to enable SSL connection.
Valid values: `true` and `false`. Default value: `true`. ###### aws.s3.enable\_path\_style\_access[​](#awss3enable_path_style_access "Direct link to aws.s3.enable_path_style_access") * Required: Yes * Description: Specifies whether to enable path-style access.
Valid values: `true` and `false`. Default value: `false`. For MinIO, you must set the value to `true`.
Path-style URLs use the following format: `https://s3..amazonaws.com//`. For example, if you create a bucket named `DOC-EXAMPLE-BUCKET1` in the US West (Oregon) Region, and you want to access the `alice.jpg` object in that bucket, you can use the following path-style URL: `https://s3.us-west-2.amazonaws.com/DOC-EXAMPLE-BUCKET1/alice.jpg`. ###### aws.s3.endpoint[​](#awss3endpoint "Direct link to aws.s3.endpoint") * Required: Yes * Description: The endpoint that is used to connect to your S3-compatible storage system instead of AWS S3. ###### aws.s3.access\_key[​](#awss3access_key-1 "Direct link to aws.s3.access_key") * Required: Yes * Description: The access key of your IAM user. ###### aws.s3.secret\_key[​](#awss3secret_key-1 "Direct link to aws.s3.secret_key") * Required: Yes * Description: The secret key of your IAM user. ###### Microsoft Azure Storage[​](#microsoft-azure-storage "Direct link to Microsoft Azure Storage") Iceberg catalogs support Microsoft Azure Storage from v3.0 onwards. ###### Azure Blob Storage[​](#azure-blob-storage "Direct link to Azure Blob Storage") If you choose Blob Storage as storage for your Iceberg cluster, take one of the following actions: * To choose the Shared Key authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.blob.storage_account" = "", "azure.blob.shared_key" = "" ``` * To choose the SAS Token authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.blob.storage_account" = "", "azure.blob.container" = "", "azure.blob.sas_token" = "" ``` * To choose REST catalog with vended credential (supported from v4.0 onwards), you do not need to configure `StorageCredentialParams`. `StorageCredentialParams` for Microsoft Azure: ###### azure.blob.storage\_account[​](#azureblobstorage_account "Direct link to azure.blob.storage_account") * Required: Yes * Description: The username of your Blob Storage account. ###### azure.blob.shared\_key[​](#azureblobshared_key "Direct link to azure.blob.shared_key") * Required: Yes * Description: The shared key of your Blob Storage account. ###### azure.blob.account\_name[​](#azureblobaccount_name "Direct link to azure.blob.account_name") * Required: Yes * Description: The username of your Blob Storage account. ###### azure.blob.container[​](#azureblobcontainer "Direct link to azure.blob.container") * Required: Yes * Description: The name of the blob container that stores your data. ###### azure.blob.sas\_token[​](#azureblobsas_token "Direct link to azure.blob.sas_token") * Required: Yes * Description: The SAS token that is used to access your Blob Storage account. ###### Azure Data Lake Storage Gen1[​](#azure-data-lake-storage-gen1 "Direct link to Azure Data Lake Storage Gen1") If you choose Data Lake Storage Gen1 as storage for your Iceberg cluster, take one of the following actions: * To choose the Managed Service Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls1.use_managed_service_identity" = "true" ``` Or: * To choose the Service Principal authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls1.oauth2_client_id" = "", "azure.adls1.oauth2_credential" = "", "azure.adls1.oauth2_endpoint" = "" ``` ###### Azure Data Lake Storage Gen2[​](#azure-data-lake-storage-gen2 "Direct link to Azure Data Lake Storage Gen2") If you choose Data Lake Storage Gen2 as storage for your Iceberg cluster, take one of the following actions: * To choose the Managed Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ``` Or: * To choose the Shared Key authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.storage_account" = "", "azure.adls2.shared_key" = "" ``` Or: * To choose the Service Principal authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_client_secret" = "", "azure.adls2.oauth2_client_endpoint" = "" ``` * To choose the Workload Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_token_file" = "", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | azure.adls2.oauth2\_token\_file | Yes | The absolute file path to the OAuth2 token file projected into the pod by the Azure Workload Identity webhook. | | azure.adls2.oauth2\_tenant\_id | Yes | The ID of the tenant whose data you want to access. | | azure.adls2.oauth2\_client\_id | Yes | The client ID (application ID) of the Azure AD application (user-assigned managed identity or app registration) associated with the workload identity. | * To choose REST catalog with vended credential (supported from v4.0 onwards), you do not need to configure `StorageCredentialParams`. ###### Google GCS[​](#google-gcs "Direct link to Google GCS") Iceberg catalogs support Google GCS from v3.0 onwards. If you choose Google GCS as storage for your Iceberg cluster, take one of the following actions: * To choose the VM-based authentication method, configure `StorageCredentialParams` as follows: ```sql "gcp.gcs.use_compute_engine_service_account" = "true" ``` * To choose the service account-based authentication method, configure `StorageCredentialParams` as follows: ```sql "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "" ``` * To choose the impersonation-based authentication method, configure `StorageCredentialParams` as follows: * Make a VM instance impersonate a service account: ```sql "gcp.gcs.use_compute_engine_service_account" = "true", "gcp.gcs.impersonation_service_account" = "" ``` * Make a service account (temporarily named as meta service account) impersonate another service account (temporarily named as data service account): ```sql "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "", "gcp.gcs.impersonation_service_account" = "" ``` * To choose REST catalog with vended credential (supported from v4.0 onwards), you do not need to configure `StorageCredentialParams`. note When vended credentials are used, StarRocks accesses GCS directly with the token vended by the REST catalog. Any `gcp.gcs.impersonation_service_account` configured on the catalog is ignored for that access. `StorageCredentialParams` for Google GCS: ###### gcp.gcs.service\_account\_email[​](#gcpgcsservice_account_email "Direct link to gcp.gcs.service_account_email") * Default value: "" * Example: "" * Description: The email address in the JSON file generated at the creation of the service account. ###### gcp.gcs.service\_account\_private\_key\_id[​](#gcpgcsservice_account_private_key_id "Direct link to gcp.gcs.service_account_private_key_id") * Default value: "" * Example: "61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea" * Description: The private key ID in the JSON file generated at the creation of the service account. ###### gcp.gcs.service\_account\_private\_key[​](#gcpgcsservice_account_private_key "Direct link to gcp.gcs.service_account_private_key") * Default value: "" * Example: "-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n" * Description: The private key in the JSON file generated at the creation of the service account. ###### gcp.gcs.impersonation\_service\_account[​](#gcpgcsimpersonation_service_account "Direct link to gcp.gcs.impersonation_service_account") * Default value: "" * Example: "hello" * Description: The service account that you want to impersonate. *** ###### MetadataRelatedParams[​](#metadatarelatedparams "Direct link to MetadataRelatedParams") A set of parameters about cache of the Iceberg metadata in StarRocks. This parameter set is optional. From v3.3.3 onwards, StarRocks supports the [periodic metadata refresh strategy](#appendix-a-periodic-metadata-refresh-strategy). In most cases, you can ignore the parameters below and do not need to tune the policy parameters in it, because the default values of these parameters already provide you with performance out-of-the-box. You can adjust the Iceberg metadata parsing mode using the system variable [`plan_mode`](https://docs.starrocks.io/docs/sql-reference/System_variable.md#plan_mode). | **Parameter** | **Default** | **Description** | | -------------------------------------------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | enable\_iceberg\_metadata\_cache | true | Whether to cache Iceberg-related metadata, including Table Cache, Partition Name Cache, and the Data File Cache and Delete Data File Cache in Manifest. | | iceberg\_manifest\_cache\_with\_column\_statistics | true | Whether to cache column statistics. When enabled, statistics are cached only for the columns where file-level min/max pruning is effective (partition source columns, sort key columns, and identifier columns), bounding the manifest cache footprint on wide tables. | | refresh\_iceberg\_manifest\_min\_length | 2 \* 1024 \* 1024 | The minimum Manifest file length that triggers a Data File Cache refresh. | | iceberg\_data\_file\_cache\_memory\_usage\_ratio | 0.1 | The maximum memory usage ratio for the data file Manifest cache. Supported from v3.5.6 onwards. | | iceberg\_delete\_file\_cache\_memory\_usage\_ratio | 0.1 | The maximum memory usage ratio for the delete file Manifest cache. Supported from v3.5.6 onwards. | | iceberg\_table\_cache\_refresh\_interval\_sec | 60 | The interval (in seconds) at which the asynchronous refresh of the Iceberg table cache is triggered. Supported from v3.5.7 onwards. | Starting from v3.4, StarRocks can obtain statistics of Iceberg tables by reading Iceberg metadata through setting the following parameters, without actively triggering the collection of Iceberg table statistics. | **Parameter** | **Default** | **Description** | | -------------------------------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | enable\_get\_stats\_from\_external\_metadata | false | Whether to obtain statistics from Iceberg metadata. When this item is set to `true`, you can further control which type of statistics to collect through the session variable [`enable_get_stats_from_external_metadata`](https://docs.starrocks.io/docs/sql-reference/System_variable.md#enable_get_stats_from_external_metadata). | ##### Examples[​](#examples "Direct link to Examples") The following examples create an Iceberg catalog named `iceberg_catalog_hms` or `iceberg_catalog_glue`, depending on the type of metastore you use, to query data from your Iceberg cluster. Chose the tab that matches your storage type: * AWS S3 * HDFS * MinIO * Microsoft Azure Blob Storage * Google GCS ###### AWS S3[​](#aws-s3-1 "Direct link to AWS S3") ###### If you choose instance profile-based credential[​](#if-you-choose-instance-profile-based-credential "Direct link to If you choose instance profile-based credential") * If you use Hive metastore in your Iceberg cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "us-west-2" ); ``` * If you use AWS Glue in your Amazon EMR Iceberg cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_glue PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.region" = "us-west-2", "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "us-west-2" ); ``` ###### If you choose assumed role-based credential[​](#if-you-choose-assumed-role-based-credential "Direct link to If you choose assumed role-based credential") * If you use Hive metastore in your HIceberg cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::081976408565:role/test_s3_role", "aws.s3.region" = "us-west-2" ); ``` * If you use AWS Glue in your Amazon EMR Iceberg cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_glue PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.iam_role_arn" = "arn:aws:iam::081976408565:role/test_glue_role", "aws.glue.region" = "us-west-2", "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::081976408565:role/test_s3_role", "aws.s3.region" = "us-west-2" ); ``` ###### If you choose IAM user-based credential[​](#if-you-choose-iam-user-based-credential "Direct link to If you choose IAM user-based credential") * If you use Hive metastore in your Iceberg cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "us-west-2" ); ``` * If you use AWS Glue in your Amazon EMR Iceberg cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_glue PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "glue", "aws.glue.use_instance_profile" = "false", "aws.glue.access_key" = "", "aws.glue.secret_key" = "", "aws.glue.region" = "us-west-2", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "us-west-2" ); ``` ###### If you choose vended credential[​](#if-you-choose-vended-credential "Direct link to If you choose vended credential") If you choose REST catalog with vended credential, run a command like below: ```sql CREATE EXTERNAL CATALOG polaris_s3 PROPERTIES ( "type" = "iceberg", "iceberg.catalog.uri" = "http://xxx:xxx/api/catalog", "iceberg.catalog.type" = "rest", "iceberg.catalog.rest.nested-namespace-enabled"="true", "iceberg.catalog.security" = "oauth2", "iceberg.catalog.oauth2.credential" = "xxxxx:xxxx", "iceberg.catalog.oauth2.scope"='PRINCIPAL_ROLE:ALL', "iceberg.catalog.warehouse" = "iceberg_catalog", "aws.s3.region" = "us-west-2" ); ``` ###### HDFS[​](#hdfs "Direct link to HDFS") If you use HDFS as storage, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083" ); ``` ###### Access an HA-enabled HDFS cluster[​](#access-an-ha-enabled-hdfs-cluster "Direct link to Access an HA-enabled HDFS cluster") If the target HDFS cluster has HA enabled, you can declare the HA configurations directly in `PROPERTIES`: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_ha PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "hadoop.security.authentication" = "simple", -- HDFS HA configurations "dfs.nameservices" = "my_cluster", "dfs.ha.namenodes.my_cluster" = "nn1,nn2", "dfs.namenode.rpc-address.my_cluster.nn1" = "host1:8020", "dfs.namenode.rpc-address.my_cluster.nn2" = "host2:8020", "dfs.client.failover.proxy.provider.my_cluster" = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider", "fs.defaultFS" = "hdfs://my_cluster" ); ``` ###### Access multiple HDFS HA clusters simultaneously[​](#access-multiple-hdfs-ha-clusters-simultaneously "Direct link to Access multiple HDFS HA clusters simultaneously") If you need to query Iceberg tables that live on multiple independent HDFS HA clusters from the same StarRocks cluster, create one Catalog per HDFS cluster and let each Catalog carry its own `dfs.nameservices` and related HA parameters. The Catalogs do not interfere with each other. ```sql -- Catalog A: access HDFS HA cluster cluster_a CREATE EXTERNAL CATALOG iceberg_catalog_a PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://hms-a.example.com:9083", "hadoop.security.authentication" = "simple", "username" = "hdfs", "dfs.nameservices" = "cluster_a", "dfs.ha.namenodes.cluster_a" = "nn1,nn2", "dfs.namenode.rpc-address.cluster_a.nn1" = "host-a-1:8020", "dfs.namenode.rpc-address.cluster_a.nn2" = "host-a-2:8020", "dfs.client.failover.proxy.provider.cluster_a" = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider", "fs.defaultFS" = "hdfs://cluster_a" ); -- Catalog B: access another HDFS HA cluster cluster_b CREATE EXTERNAL CATALOG iceberg_catalog_b PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://hms-b.example.com:9083", "hadoop.security.authentication" = "simple", "dfs.nameservices" = "cluster_b", "dfs.ha.namenodes.cluster_b" = "nn1,nn2", "dfs.namenode.rpc-address.cluster_b.nn1" = "host-b-1:8020", "dfs.namenode.rpc-address.cluster_b.nn2" = "host-b-2:8020", "dfs.client.failover.proxy.provider.cluster_b" = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider", "fs.defaultFS" = "hdfs://cluster_b" ); ``` ###### S3-compatible storage system[​](#s3-compatible-storage-system-1 "Direct link to S3-compatible storage system") Use MinIO as an example. Run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.enable_ssl" = "true", "aws.s3.enable_path_style_access" = "true", "aws.s3.endpoint" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ); ``` ###### Microsoft Azure Storage[​](#microsoft-azure-storage-1 "Direct link to Microsoft Azure Storage") ###### Azure Blob Storage[​](#azure-blob-storage-1 "Direct link to Azure Blob Storage") * If you choose the Shared Key authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.blob.storage_account" = "", "azure.blob.shared_key" = "" ); ``` * If you choose the SAS Token authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.blob.storage_account" = "", "azure.blob.container" = "", "azure.blob.sas_token" = "" ); ``` * If you choose REST catalog with vended credential, run a command like below: ```sql CREATE EXTERNAL CATALOG polaris_azure PROPERTIES ( "type" = "iceberg", "iceberg.catalog.uri" = "http://xxx:xxx/api/catalog", "iceberg.catalog.type" = "rest", "iceberg.catalog.rest.nested-namespace-enabled"="true", "iceberg.catalog.security" = "oauth2", "iceberg.catalog.oauth2.credential" = "xxxxx:xxxx", "iceberg.catalog.oauth2.scope"='PRINCIPAL_ROLE:ALL', "iceberg.catalog.warehouse" = "iceberg_catalog" ); ``` ###### Azure Data Lake Storage Gen1[​](#azure-data-lake-storage-gen1-1 "Direct link to Azure Data Lake Storage Gen1") * If you choose the Managed Service Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls1.use_managed_service_identity" = "true" ); ``` * If you choose the Service Principal authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls1.oauth2_client_id" = "", "azure.adls1.oauth2_credential" = "", "azure.adls1.oauth2_endpoint" = "" ); ``` ###### Azure Data Lake Storage Gen2[​](#azure-data-lake-storage-gen2-1 "Direct link to Azure Data Lake Storage Gen2") * If you choose the Managed Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` * If you choose the Shared Key authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.storage_account" = "", "azure.adls2.shared_key" = "" ); ``` * If you choose the Service Principal authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_client_secret" = "", "azure.adls2.oauth2_client_endpoint" = "" ); ``` * If you choose the Workload Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_token_file" = "/var/run/secrets/azure/tokens/azure-identity-token", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` * If you choose REST catalog with vended credential, run a command like below: ```sql CREATE EXTERNAL CATALOG polaris_azure PROPERTIES ( "type" = "iceberg", "iceberg.catalog.uri" = "http://xxx:xxx/api/catalog", "iceberg.catalog.type" = "rest", "iceberg.catalog.rest.nested-namespace-enabled"="true", "iceberg.catalog.security" = "oauth2", "iceberg.catalog.oauth2.credential" = "xxxxx:xxxx", "iceberg.catalog.oauth2.scope"='PRINCIPAL_ROLE:ALL', "iceberg.catalog.warehouse" = "iceberg_catalog" ); ``` ###### Google GCS[​](#google-gcs-1 "Direct link to Google GCS") * If you choose the VM-based authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.use_compute_engine_service_account" = "true" ); ``` * If you choose the service account-based authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "" ); ``` * If you choose the impersonation-based authentication method: * If you make a VM instance impersonate a service account, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.use_compute_engine_service_account" = "true", "gcp.gcs.impersonation_service_account" = "" ); ``` * If you make a service account impersonate another service account, run a command like below: ```sql CREATE EXTERNAL CATALOG iceberg_catalog_hms PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "", "gcp.gcs.impersonation_service_account" = "" ); ``` * If you choose REST catalog with vended credential, run a command like below: ```sql CREATE EXTERNAL CATALOG polaris_gcp PROPERTIES ( "type" = "iceberg", "iceberg.catalog.uri" = "http://xxx:xxx/api/catalog", "iceberg.catalog.type" = "rest", "iceberg.catalog.rest.nested-namespace-enabled"="true", "iceberg.catalog.security" = "oauth2", "iceberg.catalog.oauth2.credential" = "xxxxx:xxxx", "iceberg.catalog.oauth2.scope"='PRINCIPAL_ROLE:ALL', "iceberg.catalog.warehouse" = "iceberg_catalog" ); ``` *** #### Use your catalog[​](#use-your-catalog "Direct link to Use your catalog") ##### View Iceberg catalogs[​](#view-iceberg-catalogs "Direct link to View Iceberg catalogs") You can use [SHOW CATALOGS](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CATALOGS.md) to query all catalogs in the current StarRocks cluster: ```sql SHOW CATALOGS; ``` You can also use [SHOW CREATE CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CREATE_CATALOG.md) to query the creation statement of an external catalog. The following example queries the creation statement of an Iceberg catalog named `iceberg_catalog_glue`: ```sql SHOW CREATE CATALOG iceberg_catalog_glue; ``` *** ##### Switch to an Iceberg Catalog and a database in it[​](#switch-to-an-iceberg-catalog-and-a-database-in-it "Direct link to Switch to an Iceberg Catalog and a database in it") You can use one of the following methods to switch to an Iceberg catalog and a database in it: * Use [SET CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SET_CATALOG.md) to specify an Iceberg catalog in the current session, and then use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to specify an active database: ```sql -- Switch to a specified catalog in the current session: SET CATALOG -- Specify the active database in the current session: USE ``` * Directly use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to switch to an Iceberg catalog and a database in it: ```sql USE . ``` *** ##### Drop an Iceberg catalog[​](#drop-an-iceberg-catalog "Direct link to Drop an Iceberg catalog") You can use [DROP CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/DROP_CATALOG.md) to drop an external catalog. The following example drops an Iceberg catalog named `iceberg_catalog_glue`: ```sql DROP Catalog iceberg_catalog_glue; ``` *** ##### View the schema of an Iceberg table[​](#view-the-schema-of-an-iceberg-table "Direct link to View the schema of an Iceberg table") You can use one of the following syntaxes to view the schema of an Iceberg table: * View schema ```sql DESC[RIBE] .. ``` * View schema and location from the CREATE statement ```sql SHOW CREATE TABLE .. ``` *** ##### Query an Iceberg table[​](#query-an-iceberg-table "Direct link to Query an Iceberg table") 1. Use [SHOW DATABASES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATABASES.md) to view the databases in your Iceberg cluster: ```sql SHOW DATABASES FROM ``` 2. [Switch to an Iceberg catalog and a database in it](#switch-to-an-iceberg-catalog-and-a-database-in-it). 3. Use [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md) to query the destination table in the specified database: ```sql SELECT count(*) FROM LIMIT 10 ``` *** ##### Iceberg DDL Operations[​](#iceberg-ddl-operations "Direct link to Iceberg DDL Operations") For DDL operations (CREATE/DROP DATABASE, CREATE/DROP TABLE, and CREATE/ALTER VIEW), see [Iceberg DDL operations](https://docs.starrocks.io/docs/data_source/catalog/iceberg/DDL.md). *** ##### Iceberg DML Operations[​](#iceberg-dml-operations "Direct link to Iceberg DML Operations") For DML operations (INSERT), see [Iceberg DML operations](https://docs.starrocks.io/docs/data_source/catalog/iceberg/DML.md). *** ##### Iceberg Stored Procedures[​](#iceberg-stored-procedures "Direct link to Iceberg Stored Procedures") For Iceberg stored procedures (using snapshots, performing manual Compaction), see [Iceberg Stored Procedures](https://docs.starrocks.io/docs/data_source/catalog/iceberg/procedures.md). *** ##### Configure metadata caching[​](#configure-metadata-caching "Direct link to Configure metadata caching") The metadata files of your Iceberg cluster may be stored in remote storage such as AWS S3 or HDFS. By default, StarRocks caches Iceberg metadata in memory. To accelerate queries, StarRocks adopts a two-level metadata caching mechanism, with which it can cache metadata both in memory and on disk. For each initial query, StarRocks caches their computation results. If any subsequent query that is semantically equivalent to a previous query is issued, StarRocks first attempts to retrieve the requested metadata from its caches, and it retrieves the metadata from the remote storage only when the metadata cannot be hit in its caches. StarRocks uses the Least Recently Used (LRU) algorithm to cache and evict data. The basic rules are as follows: * StarRocks first attempts to retrieve the requested metadata from the memory. If the metadata cannot be hit in the memory, StarRock attempts to retrieve the metadata from the disks. The metadata that StarRocks has retrieved from the disks will be loaded into the memory. If the metadata cannot be hit in the disks either, StarRock retrieves the metadata from the remote storage and caches the retrieved metadata in the memory. * StarRocks writes the metadata evicted out of the memory into the disks, but it directly discards the metadata evicted out of the disks. From v3.3.3 onwards, StarRocks supports the [periodic metadata refresh strategy](#appendix-a-periodic-metadata-refresh-strategy). You can adjust the Iceberg metadata caching plan using the system variable [`plan_mode`](https://docs.starrocks.io/docs/sql-reference/System_variable.md#plan_mode). ###### FE Configurations on Iceberg metadata caching[​](#fe-configurations-on-iceberg-metadata-caching "Direct link to FE Configurations on Iceberg metadata caching") ###### enable\_iceberg\_metadata\_disk\_cache[​](#enable_iceberg_metadata_disk_cache "Direct link to enable_iceberg_metadata_disk_cache") * Unit: N/A * Default value: `false` * Description: Specifies whether to enable the disk cache. ###### iceberg\_metadata\_cache\_disk\_path[​](#iceberg_metadata_cache_disk_path "Direct link to iceberg_metadata_cache_disk_path") * Unit: N/A * Default value: `StarRocksFE.STARROCKS_HOME_DIR + "/caches/iceberg"` * Description: The save path of cached metadata files on disk. ###### iceberg\_metadata\_disk\_cache\_capacity[​](#iceberg_metadata_disk_cache_capacity "Direct link to iceberg_metadata_disk_cache_capacity") * Unit: Bytes * Default value: `2147483648`, equivalent to 2 GB * Description: The maximum size of cached metadata allowed on disk. ###### iceberg\_metadata\_memory\_cache\_capacity[​](#iceberg_metadata_memory_cache_capacity "Direct link to iceberg_metadata_memory_cache_capacity") * Unit: Bytes * Default value: `536870912`, equivalent to 512 MB * Description: The maximum size of cached metadata allowed in memory. ###### iceberg\_metadata\_memory\_cache\_expiration\_seconds[​](#iceberg_metadata_memory_cache_expiration_seconds "Direct link to iceberg_metadata_memory_cache_expiration_seconds") * Unit: Seconds * Default value: `86500` * Description: The amount of time after which a cache entry in memory expires counting from its last access. ###### iceberg\_metadata\_disk\_cache\_expiration\_seconds[​](#iceberg_metadata_disk_cache_expiration_seconds "Direct link to iceberg_metadata_disk_cache_expiration_seconds") * Unit: Seconds * Default value: `604800`, equivalent to one week * Description: The amount of time after which a cache entry on disk expires counting from its last access. ###### iceberg\_metadata\_cache\_max\_entry\_size[​](#iceberg_metadata_cache_max_entry_size "Direct link to iceberg_metadata_cache_max_entry_size") * Unit: Bytes * Default value: `8388608`, equivalent to 8 MB * Description: The maximum size of a file that can be cached. Files whose size exceeds the value of this parameter cannot be cached. If a query requests these files, StarRocks retrieves them from the remote storage. ###### enable\_background\_refresh\_connector\_metadata[​](#enable_background_refresh_connector_metadata "Direct link to enable_background_refresh_connector_metadata") * Unit: - * Default value: true * Description: Whether to enable the periodic Iceberg metadata cache refresh. After it is enabled, StarRocks polls the metastore (Hive Metastore or AWS Glue) of your Iceberg cluster, and refreshes the cached metadata of the frequently accessed Iceberg catalogs to perceive data changes. `true` indicates to enable the Iceberg metadata cache refresh, and `false` indicates to disable it. ###### background\_refresh\_metadata\_interval\_millis[​](#background_refresh_metadata_interval_millis "Direct link to background_refresh_metadata_interval_millis") * Unit: Millisecond * Default value: 600000 * Description: The interval between two consecutive Iceberg metadata cache refreshes. - Unit: millisecond. ###### background\_refresh\_metadata\_time\_secs\_since\_last\_access\_sec[​](#background_refresh_metadata_time_secs_since_last_access_sec "Direct link to background_refresh_metadata_time_secs_since_last_access_sec") * Unit: Second * Default value: 86400 * Description: The expiration time of an Iceberg metadata cache refresh task. For the Iceberg catalog that has been accessed, if it has not been accessed for more than the specified time, StarRocks stops refreshing its cached metadata. For the Iceberg catalog that has not been accessed, StarRocks will not refresh its cached metadata. #### Appendix A: Periodic Metadata Refresh Strategy[​](#appendix-a-periodic-metadata-refresh-strategy "Direct link to Appendix A: Periodic Metadata Refresh Strategy") Iceberg supports [snapshots](https://docs.starrocks.io/docs/data_source/catalog/iceberg/iceberg_timetravel.md). With the newest snapshot, you can get the newest result. Therefore, only cached snapshots can influence data freshness. As a result, you only need to pay attention to the refresh strategy of cache that contains snapshot. The following flowchart shows the time intervals on a timeline. ![Timeline for updating and discarding cached metadata](/assets/images/iceberg_catalog_timeline-7d661913987e269182bf4c22db626344.png) #### Appendix B: Metadata File Parsing[​](#appendix-b-metadata-file-parsing "Direct link to Appendix B: Metadata File Parsing") * **Distributed Plan for Large volume of Metadata** To handle large volume of metadata effectively, StarRocks employs a distributed approach using multiple BE and CN nodes. This method leverages the parallel computing capabilities of modern query engines, which can distribute tasks such as reading, decompressing, and filtering manifest files across multiple nodes. By processing these manifest files in parallel, the time required for metadata retrieval is significantly reduced, leading to faster job planning. This is particularly beneficial for large queries involving numerous manifest files, as it eliminates single-point bottlenecks and enhances overall query execution efficiency. * **Local Plan for Small volume of Metadata** For smaller queries, where the repeated decompression and parsing of manifest files can introduce unnecessary delays, a different strategy is employed. StarRocks caches deserialized memory objects, especially Avro files, to address this issue. By storing these deserialized files in memory, the system can bypass the decompression and parsing stages for subsequent queries. This caching mechanism allows direct access to the required metadata, significantly reducing retrieval times. As a result, the system becomes more responsive and better suited to meet high query demands and materialized view rewriting needs. * **Adaptive Metadata Retrieval Strategy** (Default) StarRocks is designed to automatically select the appropriate metadata retrieval method based on various factors, including the number of FE and BE/CN nodes, their CPU core counts, and the number of manifest files required for the current query. This adaptive approach ensures that the system dynamically optimizes metadata retrieval without the need for manual adjustment of metadata-related parameters. By doing so, StarRocks provides a seamless experience, balancing between distributed and local plans to achieve optimal query performance under different conditions. You can adjust the Iceberg metadata caching plan using the system variable [`plan_mode`](https://docs.starrocks.io/docs/sql-reference/System_variable.md#plan_mode). --- ### Iceberg Metadata Tables This topic describes how to inspect the metadata information of Iceberg tables in StarRocks. #### Overview[​](#overview "Direct link to Overview") From V3.4.1 onwards, StarRocks supports Iceberg metadata tables. These metadata tables contain a variety of information about Iceberg tables, such as table change history, snapshots, and manifests. You can query each metadata table by appending the metadata table name to the original table name. Currently, StarRocks supports the following Iceberg metadata tables: | Metadata table | Description | | ---------------------- | ----------------------------------------------------------------------------------------- | | `history` | Shows a log of metadata changes made to the table. | | `metadata_log_entries` | Shows the metadata log entries for the table. | | `snapshots` | Shows details about the table snapshots. | | `manifests` | Shows an overview of the manifests associated with the snapshots in the table’s log. | | `partitions` | Shows details about the partitions in the table. | | `files` | Shows details about the data files and delete files in the current snapshot of the table. | | `refs` | Shows details about the Iceberg references, including branches and tags. | #### Iceberg v3 Row Lineage Metadata Columns[​](#iceberg-v3-row-lineage-metadata-columns "Direct link to Iceberg v3 Row Lineage Metadata Columns") From v4.1 onwards, for Iceberg v3 tables (format-version = 3), StarRocks supports querying the following Row Lineage metadata columns: | Metadata Column | Description | | ------------------------------- | ------------------------------------------------------------------------------------- | | `_row_id` | Unique row identifier within the table (BIGINT). Format: `firstRowId + row_position`. | | `_last_updated_sequence_number` | The commit sequence number when the row was last updated (BIGINT). | Usage: ```sql SELECT _row_id, _last_updated_sequence_number, * FROM [.][.]table; ``` note * The `_row_id` column requires data files to have `firstRowId` metadata. If a data file is missing `firstRowId`, the query will fail with an error. * For newly inserted data, `_row_id` is computed as `firstRowId + row_position` and `_last_updated_sequence_number` is the file-level `dataSequenceNumber`. * After compaction (e.g., Iceberg OPTIMIZE / rewrite-data-files), if the compactor writes `_row_id` and `_last_updated_sequence_number` as physical columns in the data files (as required by the Iceberg v3 spec), StarRocks reads the per-row values from the physical columns, preserving row lineage across compaction. * These metadata columns are only available for Iceberg v3 tables (format-version = 3). #### `history` table[​](#history-table "Direct link to history-table") Usage: ```sql SELECT * FROM [.][.]table$history; ``` Output: | Field | Description | | --------------------- | ------------------------------------------------------------- | | made\_current\_at | The time when the snapshot became the current snapshot. | | snapshot\_id | The ID of the snapshot. | | parent\_id | The ID of the parent snapshot. | | is\_current\_ancestor | Whether this snapshot is an ancestor of the current snapshot. | #### `metadata_log_entries` table[​](#metadata_log_entries-table "Direct link to metadata_log_entries-table") Usage: ```sql SELECT * FROM [.][.]table$metadata_log_entries; ``` Output: | Field | Description | | ------------------------ | ------------------------------------------------------------ | | timestamp | The time when the metadata was recorded. | | file | The location of the metadata file. | | latest\_snapshot\_id | The ID of the latest snapshot when the metadata was updated. | | latest\_schema\_id | The ID of the latest schema when the metadata was updated. | | latest\_sequence\_number | The data sequence number of the metadata file. | #### `snapshots` table[​](#snapshots-table "Direct link to snapshots-table") Usage: ```sql SELECT * FROM [.][.]table$snapshots; ``` Output: | Field | Description | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | committed\_at | The time when the snapshot was committed. | | snapshot\_id | The ID for the snapshot. | | parent\_id | The ID for the parent snapshot. | | operation | The type of operation performed on the Iceberg table. Valid values:- `append`: New data is appended.
- `replace`: Files are removed and replaced without changing the data in the table.
- `overwrite`: Old data is overwritten by new data.
- `delete`: Data is deleted from the table. | | manifest\_list | The list of Avro manifest files that contain detailed information about snapshot changes. | | summary | A summary of the changes made from the previous snapshot to the current snapshot. | #### `manifests` table[​](#manifests-table "Direct link to manifests-table") Usage: ```sql SELECT * FROM [.][.]table$manifests; ``` Output: | Field | Description | | ---------------------------- | --------------------------------------------------------------------------------------- | | path | The location of the manifest file. | | length | The length of the manifest file. | | partition\_spec\_id | The ID for the partition specification that is used to write the manifest file. | | added\_snapshot\_id | The ID of the snapshot during which this manifest entry has been added. | | added\_data\_files\_count | The number of data files with status `ADDED` in the manifest file. | | added\_rows\_count | The total number of rows in all data files with status `ADDED` in the manifest file. | | existing\_data\_files\_count | The number of data files with status `EXISTING` in the manifest file. | | existing\_rows\_count | The total number of rows in all data files with status `EXISTING` in the manifest file. | | deleted\_data\_files\_count | The number of data files with status `DELETED` in the manifest file. | | deleted\_rows\_count | The total number of rows in all data files with status `DELETED` in the manifest file. | | partition\_summaries | Partition range metadata. | #### `partitions` table[​](#partitions-table "Direct link to partitions-table") Usage: ```sql SELECT * FROM [.][.]table$partitions; ``` Output: | Field | Description | | ---------------------------------- | ------------------------------------------------------------------------- | | partition\_value | The mapping of the partition column names to the partition column values. | | spec\_id | The partition Spec ID of files. | | record\_count | The number of records in the partition. | | file\_count | The number of files mapped in the partition. | | total\_data\_file\_size\_in\_bytes | The size of all the data files in the partition. | | position\_delete\_record\_count | The total row count of Position Delete files in the partition. | | position\_delete\_file\_count | The number of Position Delete files in the partition. | | equality\_delete\_record\_count | The total row count of Equality Delete files in the partition. | | equality\_delete\_file\_count | The number of Position Equality files in the partition. | | last\_updated\_at | The time when the partition was updated most recently. | #### `files` table[​](#files-table "Direct link to files-table") Usage: ```sql SELECT * FROM [.][.]table$files; ``` Output: | Field | Description | | --------------------- | ------------------------------------------------------------------------------------------------------------------ | | content | The type of content stored in the file. Valid values: `DATA(0)`, `POSITION_DELETES(1)`, and `EQUALITY_DELETES(2)`. | | file\_path | The location of the data file. | | file\_format | The format of the data file. | | spec\_id | The Spec ID that is used to track the file containing a row. | | record\_count | The number of entries contained in the data file. | | file\_size\_in\_bytes | The size of the data file. | | column\_sizes | The mapping between the Iceberg column ID and its corresponding size in the file. | | value\_counts | The mapping between the Iceberg column ID and its corresponding count of entries in the file. | | null\_value\_counts | The mapping between the Iceberg column ID and its corresponding count of `NULL` values in the file. | | nan\_value\_counts | The mapping between the Iceberg column ID and its corresponding count of non- numerical values in the file. | | lower\_bounds | The mapping between the Iceberg column ID and its corresponding lower bound in the file. | | upper\_bounds | The mapping between the Iceberg column ID and its corresponding upper bound in the file. | | split\_offsets | The list of recommended split locations. | | sort\_id | The ID representing sort order for this file. | | equality\_ids | The set of field IDs used for equality comparison in equality delete files. | | key\_metadata | The metadata about the encryption key that is used to encrypt this file, if applicable. | #### `refs` table[​](#refs-table "Direct link to refs-table") Usage: ```sql SELECT * FROM [.][.]table$refs; ``` Output: | Field | Description | | --------------------------- | --------------------------------------------------------------------------------------------------------- | | name | The name of the reference. | | type | The type of the reference. Valid values: `BRANCH` or `TAG`. | | snapshot\_id | The snapshot ID of the reference. | | max\_reference\_age\_in\_ms | The maximum age of the reference before it could be expired. | | min\_snapshots\_to\_keep | For branch only, the minimum number of snapshots to keep in a branch. | | max\_snapshot\_age\_in\_ms | For branch only, the max snapshot age allowed in a branch. Older snapshots in the branch will be expired. | --- ### Create Iceberg REST Catalog for AWS S3 Tables This article explains how to create Iceberg REST Catalog in StarRocks for access to data in AWS S3 tables through the AWS Glue Iceberg REST endpoint. The AWS Glue Iceberg REST endpoint implements the [Iceberg REST Catalog Open API specification](https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml), which provides a standardized interface for interacting with Iceberg tables. To access S3 tables using this endpoint, you need to configure credentials by combining IAM policies and AWS Lake Formation authorization. The following sections will guide you through the access permission setup, including defining required policies, establishing Lake Formation permissions at database and table levels, and using StarRocks to create an Iceberg REST catalog for accessing S3 tables. #### (Optional) Create a table bucket[​](#optional-create-a-table-bucket "Direct link to (Optional) Create a table bucket") You can skip this step if you already have a table bucket for S3 tables. 1. Sign in to the [Amazon S3 Console](https://console.aws.amazon.com/s3) as a user with administrator privileges. 2. In the upper-right corner of the page, select your AWS region. 3. In the left-side navigation pane, choose **Table buckets** from the navigation panel. 4. Click **Create table bucket** to create a table bucket. 5. After creating the table bucket, select it and click **Create table with Athena**. 6. Create a namespace. 7. After creating the namespace, click **Create table with Athena** again to create a table. > **NOTE** > > You can create a Database and Table using Athena, and then query them using StarRocks. Alternatively, you can just create a table bucket, and then use StarRocks to create the database and table. #### Create IAM Policy[​](#create-iam-policy "Direct link to Create IAM Policy") To access S3 tables via the AWS Glue endpoint, create an IAM Policy with permissions for AWS Glue and Lake Formation operations: 1. Sign in to the [Amazon IAM Console](https://console.aws.amazon.com/iam) as a user with administrator privileges. 2. In the upper-right corner of the page, select your AWS region. 3. In the left-side navigation pane, choose **Policies** from the navigation panel. 4. Choose **Create a policy**, select **JSON** in the policy editor. 5. Add the following policy to grant access to AWS Glue and Lake Formation actions. ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "glue:GetCatalog", "glue:GetDatabase", "glue:GetDatabases", "glue:GetTable", "glue:GetTables", "glue:CreateTable", "glue:UpdateTable" ], "Resource": [ "arn:aws:glue:::catalog", "arn:aws:glue:::catalog/s3tablescatalog", "arn:aws:glue:::catalog/s3tablescatalog/", "arn:aws:glue:::table/s3tablescatalog///*", "arn:aws:glue:::database/s3tablescatalog//" ] }, { "Effect": "Allow", "Action": [ "lakeformation:GetDataAccess" ], "Resource": "*" } ] } ``` After creating the IAM policy, attach it to the target IAM user: 1. Choose **Users** from the navigation panel. 2. Select the user that requires the S3 table access. 3. Click **Add permissions** and select **Attach policies directly**. 4. Attach the newly created policy. #### Manage Permissions via Lake Formation[​](#manage-permissions-via-lake-formation "Direct link to Manage Permissions via Lake Formation") To access S3 tables, StarRocks requires Lake Formation to first set up permissions that allow third-party query engines to access S3 tables. 1. Sign in to the [Lake Formation Console](https://console.aws.amazon.com/lakeformation) as a user with administrator privileges. 2. In the upper-right corner of the page, select your AWS region. 3. In the left-side navigation pane, choose **Application integration settings** from the navigation panel. 4. Choose **Allow external engines to access data in Amazon S3 locations with full table access.** Next, grant the above IAM User access permissions in Lake Formation. 1. In the left-side navigation pane of the [Lake Formation Console](https://console.aws.amazon.com/lakeformation), choose **Data permissions** from the navigation panel. 2. Click **Grant**. 3. In the **Principals** section, choose **IAM users and roles**, and select the authorized IAM user from the **IAM users and roles** drop-down list. 4. In the **LF-Tags or catalog resources** section, choose **Named Data Catalog resources**, and select your table bucket you created in the **Catalogs** drop-down list. 5. In the **Catalog permissions** section, choose **Super** for **Catalog permissions**. 6. Click **Grant**. > **NOTE** > > Here, Super permissions are granted for testing convenience. You need to assign appropriate permissions based on actual requirements in a production environment. #### Create Iceberg REST Catalog[​](#create-iceberg-rest-catalog "Direct link to Create Iceberg REST Catalog") Create an Iceberg REST catalog in StarRocks: ```sql CREATE EXTERNAL CATALOG starrocks_lakehouse_s3tables PROPERTIES( "type"="iceberg", "iceberg.catalog.type" = "rest", "iceberg.catalog.uri" = "https://glue..amazonaws.com/iceberg", "iceberg.catalog.rest.sigv4-enabled" = "true", "iceberg.catalog.rest.signing-name" = "glue", "iceberg.catalog.rest.access-key-id" = "", "iceberg.catalog.rest.secret-access-key" = "", "iceberg.catalog.warehouse" = ":s3tablescatalog/", "aws.s3.region" = "" ); ``` You can then create databases and tables and run queries in it. Example: ```sql -- Switch to the catalog StarRocks> SET CATALOG starrocks_lakehouse_s3tables; -- Create database StarRocks> CREATE DATABASE s3table_db; Query OK, 0 rows affected -- Switch database StarRocks> USE s3table_db; Database changed -- Create table StarRocks> CREATE TABLE taxis ( trip_id BIGINT, trip_distance FLOAT, fare_amount DOUBLE, store_and_fwd_flag STRING, vendor_id BIGINT ) PARTITION BY (vendor_id); Query OK, 0 rows affected -- Insert data StarRocks> INSERT INTO taxis VALUES (1000371, 1.8, 15.32, 'N', 1), (1000372, 2.5, 22.15, 'N', 2), (1000373, 0.9, 9.01, 'N', 2), (1000374, 8.4, 42.13, 'Y', 1); Query OK, 4 rows affected -- Query data StarRocks> SELECT * FROM taxis; +---------+---------------+-------------+--------------------+-----------+ | trip_id | trip_distance | fare_amount | store_and_fwd_flag | vendor_id | +---------+---------------+-------------+--------------------+-----------+ | 1000372 | 2.5 | 22.15 | N | 2 | | 1000373 | 0.9 | 9.01 | N | 2 | | 1000371 | 1.8 | 15.32 | N | 1 | | 1000374 | 8.4 | 42.13 | Y | 1 | +---------+---------------+-------------+--------------------+-----------+ 4 rows in set ``` --- ### Security Setup for Iceberg REST Catalog In scenarios where multiple nodes collaboratively access the same data lake, the core challenge is **how to achieve secure, unified, and auditable permission management**. In traditional models, users need to configure storage credentials and perform local permission control individually for each node, which not only increases maintenance costs but also exposes risks of credential leakage and inconsistent permissions. By integrating Iceberg REST Catalog with StarRocks, secure access and unified permission management can be achieved through a combination of JWT (JSON Web Token) authentication and Vended Credentials (temporary credentials). This setup helps to: * **Reduce credential risks**: No need to store high-privilege account information within StarRocks. Credentials are issued temporarily by the Catalog, avoiding leakage. * **Unified and simplified permissions**: Access control for all databases, tables, and views is centrally managed by the Catalog, ensuring consistency across different nodes and avoiding redundant configuration. * **Compliance and simplified operations**: User actions are traceable, facilitating audits. It also reduces the cost of maintaining permissions and storage credentials within StarRocks. #### Security mechanisms[​](#security-mechanisms "Direct link to Security mechanisms") * **JWT authentication** * When a user logs into StarRocks, the JWT Token obtained can be passed through to the Iceberg REST Session Catalog. * The Catalog will authenticate the user based on the JWT Token and execute queries under the real user’s identity. * The benefit is that StarRocks doesn’t need to store high-privilege accounts, which significantly lowers security risks. * **Vended credentials** * After authentication, the Catalog can generate temporary storage access credentials for the user. * Users do not need to configure credentials for the storage layer in StarRocks. * Each time object storage is accessed, StarRocks will use the temporary credentials issued by the Catalog. * This enhances security and simplifies credential management. #### Usage[​](#usage "Direct link to Usage") ##### Step 1. Set up JWT authentication[​](#step-1-set-up-jwt-authentication "Direct link to Step 1. Set up JWT authentication") In StarRocks, configure a **[JWT-based Security Integration](https://docs.starrocks.io/docs/administration/user_privs/authentication/security_integration.md#create-a-security-integration-with-jwt)** or [create a user with JWT authentication](https://docs.starrocks.io/en/docs/administration/user_privs/authentication/jwt_authentication/). ##### Step 2. Create Iceberg REST Catalog and configure security settings[​](#step-2-create-iceberg-rest-catalog-and-configure-security-settings "Direct link to Step 2. Create Iceberg REST Catalog and configure security settings") ```sql CREATE EXTERNAL CATALOG iceberg_rest_catalog PROPERTIES ( "iceberg.catalog.type" = "rest", "iceberg.catalog.uri" = "", "iceberg.catalog.security" = "jwt", "iceberg.catalog.warehouse" = "", "iceberg.catalog.vended-credentials-enabled" = "true" ); ``` Properties: * `iceberg.catalog.type`: Set this property to `rest`, indicating to use the REST Catalog. * `iceberg.catalog.uri`: API endpoint for the REST Catalog service. * `iceberg.catalog.security`: Set this property to `jwt` to enable JWT authentication. StarRocks will pass the current user’s authentication information to the Catalog. * `iceberg.catalog.warehouse`: Specifies the Iceberg data warehouse path or identifier. * `iceberg.catalog.vended-credentials-enabled`: Set this property to `true` to enable Vended Credentials and allow temporary credentials issuance. ##### Step 3. Grant permissions[​](#step-3-grant-permissions "Direct link to Step 3. Grant permissions") When a user queries the Catalog via StarRocks, permissions are handled at two levels: 1. **StarRocks Internal Object Permissions** Whether the user has permission to view the Catalog object within StarRocks and switch the session to that Catalog using `SET CATALOG`. 2. **Catalog Internal Permissions** Whether the user has appropriate access permissions when accessing databases, tables, or views within the Catalog. In the ideal workflow, StarRocks is responsible for managing the basic permissions required for a user to access Catalog objects after logging into the StarRocks cluster. Fine-grained data access permissions within the Catalog are completely managed by the Catalog itself. This means: * Users must have the `USAGE` permission to the Catalog object in StarRocks so that they can switch to the catalog using `SET CATALOG` or view the catalog information using `SHOW CATALOGS`. * Data-level permission checks are entirely handled by the Catalog. ###### StarRocks Internal Catalog Object Permissions[​](#starrocks-internal-catalog-object-permissions "Direct link to StarRocks Internal Catalog Object Permissions") To execute `SHOW CATALOGS` or `SET CATALOG` in StarRocks, the user needs `USAGE` permission on the corresponding Catalog object. ###### Option 1: Grant Access to All Users[​](#option-1-grant-access-to-all-users "Direct link to Option 1: Grant Access to All Users") If you allow all users to view and switch to a specified Catalog, you can grant the `USAGE` permission of the Catalog to the `public` role. ```sql GRANT USAGE ON CATALOG TO ROLE public; ``` This will automatically grant all users, upon logging into StarRocks, the `public` role and thus the `USAGE` permission of that Catalog. ###### Option 2: Granular Permission Management[​](#option-2-granular-permission-management "Direct link to Option 2: Granular Permission Management") If you want only specific users or groups to view and switch to a Catalog, you can use **Group Provider** and **Role** for finer-grained management: 1. Synchronize the external user group information to StarRocks using [Group Provider](https://docs.starrocks.io/docs/administration/user_privs/group_provider.md). 2. Create the corresponding StarRocks role. ```sql CREATE ROLE ; ``` 3. Bind the role to the external user group: ```sql GRANT TO EXTERNAL GROUP ; ``` 4. Grant the role `USAGE` permission of the Catalog: ```sql GRANT USAGE ON CATALOG TO ROLE ; ``` In this case, members of the external group will automatically inherit the assigned role upon logging into StarRocks and gain the `USAGE` permission of the Catalog. ###### Catalog Internal Data Permissions[​](#catalog-internal-data-permissions "Direct link to Catalog Internal Data Permissions") By configuring the Catalog’s properties, permission checks for all objects within the Catalog are delegated to the Catalog itself. The Catalog will manage authentication and permissions centrally, simplifying permission configuration in StarRocks, avoiding redundant authorizations, and ensuring consistent permission rules across different engines. ```sql ALTER CATALOG iceberg_rest_catalog SET PROPERTIES ( "catalog.access.control" = "allowall" ); ``` Properties: * `catalog.access.control`: Set this property to `allowall`. StarRocks will not perform any further permission checks on objects within the Catalog. Instead, all permission management will be handled by the Catalog itself. --- ### Time Travel with Iceberg Catalog Beta feature [Advice on use of Beta features](https://docs.starrocks.io/docs/introduction/maturity.md) This topic introduces StarRocks' Time Travel feature for Iceberg catalogs. This feature is supported from v3.4.0 onwards. #### Overview[​](#overview "Direct link to Overview") Each Iceberg table maintains a metadata snapshot log, which represents the changes applied to it. Databases can perform Time Travel queries against Iceberg tables by accessing these historical snapshots. Iceberg supports branching and tagging snapshots for sophisticated snapshot lifecycle management, allowing each branch or tag to maintain its own lifecycle based on customized retention policies. For more information on Iceberg's branching and tagging feature, see [Official Documentation](https://iceberg.apache.org/docs/latest/branching/). By integrating Iceberg's snapshot branching and tagging feature, StarRocks supports creating and managing branches and tags in Iceberg catalogs, and Time Travel queries against tables within. #### Manage branches, tags, and snapshots[​](#manage-branches-tags-and-snapshots "Direct link to Manage branches, tags, and snapshots") This section introduces how to manage branches, tags, and snapshots. For instructions on Iceberg stored procedures (using snapshots, performing manual Compaction), see [Iceberg Stored Procedures](https://docs.starrocks.io/docs/data_source/catalog/iceberg/procedures.md). ##### Create a branch[​](#create-a-branch "Direct link to Create a branch") ###### `CREATE BRANCH` Syntax[​](#create-branch-syntax "Direct link to create-branch-syntax") ```sql ALTER TABLE [catalog.][database.]table_name CREATE [OR REPLACE] BRANCH [IF NOT EXISTS] [AS OF VERSION ] [RETAIN { DAYS | HOURS | MINUTES }] [WITH SNAPSHOT RETENTION { minSnapshotsToKeep | maxSnapshotAge | minSnapshotsToKeep maxSnapshotAge }] minSnapshotsToKeep ::= SNAPSHOTS maxSnapshotAge ::= { DAYS | HOURS | MINUTES } ``` ###### Parameters[​](#parameters "Direct link to Parameters") * `branch_name`: Name of the branch to create. * `AS OF VERSION`: ID of the snapshot (version) on which to create the branch. * `RETAIN`: Time to retain the branch. Format: ` `. Supported units: `DAYS`, `HOURS`, and `MINUTES`. Example: `7 DAYS`, `12 HOURS`, or `30 MINUTES`. * `WITH SNAPSHOT RETENTION`: The minimum number of snapshots to keep and/or the maximum time to keep the snapshots. ###### Example[​](#example "Direct link to Example") Create a branch `test-branch` based on version (snapshot ID) `12345` of the table `iceberg.sales.order`, retain the branch for `7` days, and keep at least `2` snapshots on the branch. ```sql ALTER TABLE iceberg.sales.order CREATE BRANCH `test-branch` AS OF VERSION 12345 RETAIN 7 DAYS WITH SNAPSHOT RETENTION 2 SNAPSHOTS; ``` Create a branch `test-branch2` based on version (snapshot ID) `12345` of the table `iceberg.sales.order`, retain the branch for `7` days, and keep the snapshot on the branch for at most `2` days. ```sql ALTER TABLE iceberg.sales.order CREATE BRANCH `test-branch2` AS OF VERSION 12345 RETAIN 7 DAYS WITH SNAPSHOT RETENTION 2 DAYS; ``` Create a branch `test-branch3` based on version (snapshot ID) `12345` of the table `iceberg.sales.order`, retain the branch for `7` days, and keep at least `2` snapshots on the branch, each for at most `2` days. ```sql ALTER TABLE iceberg.sales.order CREATE BRANCH `test-branch3` AS OF VERSION 12345 RETAIN 7 DAYS WITH SNAPSHOT RETENTION 2 SNAPSHOTS 2 DAYS; ``` ##### Load data into a specific branch of a table[​](#load-data-into-a-specific-branch-of-a-table "Direct link to Load data into a specific branch of a table") ###### `VERSION AS OF` Syntax[​](#version-as-of-syntax "Direct link to version-as-of-syntax") ```sql INSERT INTO [catalog.][database.]table_name [FOR] VERSION AS OF ``` ###### Parameters[​](#parameters-1 "Direct link to Parameters") * `branch_name`: Name of the table branch into which the data is loaded. * `query_statement`: Query statement whose result will be loaded into the destination table. It can be any SQL statement supported by StarRocks. ###### Example[​](#example-1 "Direct link to Example") Load the result of a query into the branch `test-branch` of the table `iceberg.sales.order`. ```sql INSERT INTO iceberg.sales.order FOR VERSION AS OF `test-branch` SELECT c1, k1 FROM tbl; ``` ##### Create a tag[​](#create-a-tag "Direct link to Create a tag") ###### `CREATE TAG` Syntax[​](#create-tag-syntax "Direct link to create-tag-syntax") ```sql ALTER TABLE [catalog.][database.]table_name CREATE [OR REPLACE] TAG [IF NOT EXISTS] [AS OF VERSION ] [RETAIN { DAYS | HOURS | MINUTES }] ``` ###### Parameters[​](#parameters-2 "Direct link to Parameters") * `tag_name`: Name of the tag to create. * `AS OF VERSION`: ID of the snapshot (version) on which to create the tag. * `RETAIN`: Time to retain the tag. Format: ` `. Supported units: `DAYS`, `HOURS`, and `MINUTES`. Example: `7 DAYS`, `12 HOURS`, or `30 MINUTES`. ###### Example[​](#example-2 "Direct link to Example") Create a tag `test-tag` based on version (snapshot ID) `12345` of the table `iceberg.sales.order`, and retain the tag for `7` days. ```sql ALTER TABLE iceberg.sales.order CREATE TAG `test-tag` AS OF VERSION 12345 RETAIN 7 DAYS; ``` ##### Drop a branch or a tag[​](#drop-a-branch-or-a-tag "Direct link to Drop a branch or a tag") ###### `DROP BRANCH`, `DROP TAG` Syntax[​](#drop-branch-drop-tag-syntax "Direct link to drop-branch-drop-tag-syntax") ```sql ALTER TABLE [catalog.][database.]table_name DROP { BRANCH | TAG } ``` ###### Example[​](#example-3 "Direct link to Example") ```sql ALTER TABLE iceberg.sales.order DROP BRANCH `test-branch`; ALTER TABLE iceberg.sales.order DROP TAG `test-tag`; ``` #### Query with Time Travel[​](#query-with-time-travel "Direct link to Query with Time Travel") ##### Time Travel to a specific branch or tag[​](#time-travel-to-a-specific-branch-or-tag "Direct link to Time Travel to a specific branch or tag") ###### `VERSION AS OF` Syntax[​](#version-as-of-syntax-1 "Direct link to version-as-of-syntax-1") ```sql [FOR] VERSION AS OF '' ``` ###### Parameter[​](#parameter "Direct link to Parameter") `tag_or_branch`: Name of the branch or tag to which you want to Time Travel. If a branch name is specified, the query will Time Travel to the head snapshot of the branch. If a tag name is specified, the query will Time Travel to the snapshot that the tag referenced. ###### Example[​](#example-4 "Direct link to Example") ```sql -- Time Travel to the head snapshot of a branch. SELECT * FROM iceberg.sales.order VERSION AS OF 'test-branch'; -- Time Travel to the snapshot that the tag referenced. SELECT * FROM iceberg.sales.order VERSION AS OF 'test-tag'; ``` ##### Time Travel to a specific snapshot[​](#time-travel-to-a-specific-snapshot "Direct link to Time Travel to a specific snapshot") ###### `VERSION AS OF` Syntax[​](#version-as-of-syntax-2 "Direct link to version-as-of-syntax-2") ```sql [FOR] VERSION AS OF '' ``` ###### Parameter[​](#parameter-1 "Direct link to Parameter") `snapshot_id`: ID of the snapshot to which you want to Time Travel. ###### Example[​](#example-5 "Direct link to Example") ```sql SELECT * FROM iceberg.sales.order VERSION AS OF 12345; ``` ##### Time Travel to a specific datetime or date[​](#time-travel-to-a-specific-datetime-or-date "Direct link to Time Travel to a specific datetime or date") ###### `TIMESTAMP AS OF` Syntax[​](#timestamp-as-of-syntax "Direct link to timestamp-as-of-syntax") ```sql [FOR] TIMESTAMP AS OF { '' | '' | date_and_time_function } ``` ###### Parameter[​](#parameter-2 "Direct link to Parameter") `date_and_time_function`: Any [date and time functions](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/now.md) supported by StarRocks. ###### Example[​](#example-6 "Direct link to Example") ```sql SELECT * FROM iceberg.sales.order TIMESTAMP AS OF '1986-10-26 01:21:00'; SELECT * FROM iceberg.sales.order TIMESTAMP AS OF '1986-10-26'; SELECT * FROM iceberg.sales.order TIMESTAMP AS OF now(); ``` --- ### Iceberg Procedures StarRocks Iceberg Catalog supports a variety of procedures for managing Iceberg tables, including snapshot management, branch management, data maintenance, metadata management, and table management. You must have the appropriate privileges to execute procedures. For more information about privileges, see [Privileges](https://docs.starrocks.io/docs/administration/user_privs/authorization/privilege_item.md). #### Snapshot management[​](#snapshot-management "Direct link to Snapshot management") ##### Rollback to snapshot[​](#rollback-to-snapshot "Direct link to Rollback to snapshot") Rolls back the table to a specific snapshot. This operation sets the table's current snapshot to the specified snapshot ID. ###### `rollback_to_snapshot` Syntax[​](#rollback_to_snapshot-syntax "Direct link to rollback_to_snapshot-syntax") ```sql ALTER TABLE [catalog.][database.]table_name EXECUTE rollback_to_snapshot() ``` ###### Parameters[​](#parameters "Direct link to Parameters") `snapshot_id`: ID of the snapshot to which you want to roll back the table. ###### Example[​](#example "Direct link to Example") Roll back the table to snapshot with ID 98765: ```sql ALTER TABLE iceberg.sales.order EXECUTE rollback_to_snapshot(98765); ``` ##### Cherry pick a snapshot[​](#cherry-pick-a-snapshot "Direct link to Cherry pick a snapshot") Cherry picks a specific snapshot and applies it to the current state of the table. This operation creates a new snapshot based on an existing snapshot, while the original snapshot remains unchanged. ###### `cherrypick_snapshot` Syntax[​](#cherrypick_snapshot-syntax "Direct link to cherrypick_snapshot-syntax") ```sql ALTER TABLE [catalog.][database.]table_name EXECUTE cherrypick_snapshot() ``` ###### Parameters[​](#parameters-1 "Direct link to Parameters") `snapshot_id`: ID of the snapshot which you want to cherry pick. ###### Example[​](#example-1 "Direct link to Example") ```sql ALTER TABLE iceberg.sales.order EXECUTE cherrypick_snapshot(54321); ``` #### Branch management[​](#branch-management "Direct link to Branch management") ##### Fast forward a branch to another[​](#fast-forward-a-branch-to-another "Direct link to Fast forward a branch to another") Fast-forwards one branch to another branch's latest snapshot. This operation updates the source branch's snapshot to match the target branch's snapshot. ###### `fast_forward` Syntax[​](#fast_forward-syntax "Direct link to fast_forward-syntax") ```sql ALTER TABLE [catalog.][database.]table_name EXECUTE fast_forward('', '') ``` ###### Parameters[​](#parameters-2 "Direct link to Parameters") * `from_branch`: The branch you want to fast forward. Wrap the branch name in quotes. * `to_branch`: The branch to which you want to fast forward the `from_branch`. Wrap the branch name in quotes. ###### Example[​](#example-2 "Direct link to Example") Fast forward the `main` branch to the branch `test-branch`: ```sql ALTER TABLE iceberg.sales.order EXECUTE fast_forward('main', 'test-branch'); ``` #### Data maintenance[​](#data-maintenance "Direct link to Data maintenance") ##### Rewrite data files[​](#rewrite-data-files "Direct link to Rewrite data files") Rewrites data files to optimize file layout. This procedure merges small files to improve query performance and reduce metadata overhead. ###### `rewrite_data_files` Syntax[​](#rewrite_data_files-syntax "Direct link to rewrite_data_files-syntax") ```sql ALTER TABLE [catalog.][database.]table_name EXECUTE rewrite_data_files ("key"=value [,"key"=value, ...]) [WHERE ] ``` ###### Parameters[​](#parameters-3 "Direct link to Parameters") ###### `rewrite_data_files` properties[​](#rewrite_data_files-properties "Direct link to rewrite_data_files-properties") `"key"=value` pairs that declare the manual compaction behaviors. Note that you need to wrap the key in double quotes. ###### `min_file_size_bytes`[​](#min_file_size_bytes "Direct link to min_file_size_bytes") * Description: The upper limit of a small data file. Data files whose size is less than this value will be merged during the compaction. * Unit: Byte * Type: Int * Default: 268,435,456 (256 MB) ###### `batch_size`[​](#batch_size "Direct link to batch_size") * Description: The maximum size of data that can be processed in each batch. * Unit: Byte * Type: Int * Default: 10,737,418,240 (10 GB) ###### `rewrite_all`[​](#rewrite_all "Direct link to rewrite_all") * Description: Whether to rewrite all data files during the compaction, ignoring the parameters that filter data files with specific requirements. * Unit: - * Type: Boolean * Default: false ###### `batch_parallelism`[​](#batch_parallelism "Direct link to batch_parallelism") * Description: The number of parallel batches to process during the compaction. * Unit: - * Type: Int * Default: 1 ###### `WHERE` clause[​](#where-clause "Direct link to where-clause") * Description: The filter predicate used to specify the partition(s) to be involved in the compaction. ###### Example[​](#example-3 "Direct link to Example") The following example performs manual Compaction on specific partitions in the Iceberg table `t1`. The partitions are represented by the clause `WHERE part_col = 'p1'`. In these partitions, data files that are smaller than 134,217,728 bytes (128 MB) will be merged during the Compaction. ```sql ALTER TABLE t1 EXECUTE rewrite_data_files("min_file_size_bytes"= 134217728) WHERE part_col = 'p1'; ``` #### Metadata management[​](#metadata-management "Direct link to Metadata management") ##### Expire snapshots[​](#expire-snapshots "Direct link to Expire snapshots") Expires snapshots older than a specific timestamp. This operation deletes the data files of the expired snapshots, helping to manage storage usage. ###### `expire_snapshots` Syntax[​](#expire_snapshots-syntax "Direct link to expire_snapshots-syntax") ```sql ALTER TABLE [catalog.][database.]table_name EXECUTE expire_snapshots( [ [older_than =] '' ] [, [retain_last =] ] ) ``` ###### Parameters[​](#parameters-4 "Direct link to Parameters") ###### `older_than`[​](#older_than "Direct link to older_than") * Description: The timestamp before which snapshots will be removed. If not specified, files older than 5 days (from the current time) will be removed by default. Format: 'YYYY-MM-DD HH:MM :SS '. * Type: DATETIME * Required: No ###### `retain_last`[​](#retain_last "Direct link to retain_last") * Description: The maximum number of most recent snapshots to retain. The less recent snapshots will be removed when this threshold is reached. If not specified, only one snapshot will be retained by default. * Type: Integer * Required: No ###### Example[​](#example-4 "Direct link to Example") Expire snapshots before '2023-12-17 00:14:38' and retain two snapshots: ```sql -- With the parameter key specified: ALTER TABLE iceberg.sales.order EXECUTE expire_snapshots(older_than = '2023-12-17 00:14:38', retain_last = 2); -- With the parameter key unspecified: ALTER TABLE iceberg.sales.order EXECUTE expire_snapshots('2023-12-17 00:14:38', 2); ``` ##### Remove orphan files[​](#remove-orphan-files "Direct link to Remove orphan files") Removes orphan files from the table that are not referenced by any valid snapshot and are older than a specified timestamp. This operation helps clean up unused files and reclaim storage space. ###### `remove_orphan_files` Syntax[​](#remove_orphan_files-syntax "Direct link to remove_orphan_files-syntax") ```sql ALTER TABLE [catalog.][database.]table_name EXECUTE remove_orphan_files( [ [older_than =] '' ] [, [location =] '' ] ) ``` ###### Parameters[​](#parameters-5 "Direct link to Parameters") ###### `older_than`[​](#older_than-1 "Direct link to older_than-1") * Description: The timestamp before which orphan files will be removed. If not specified, files older than 7 days (from the current time) will be removed by default. Format: 'YYYY-MM-DD HH:MM :SS '. The value must be at least [`iceberg_remove_orphan_files_min_retention_seconds`](https://docs.starrocks.io/docs/administration/management/FE_parameters/shared_lake_other.md) (24 hours by default) before the current time. A more recent value is rejected, because deleting files that young can remove data that a concurrent write has not committed yet. * Type: DATETIME * Required: No ###### `location`[​](#location "Direct link to location") * Description: The directory from which you want to remove orphan files. It must be a sub-directory of the table location. If not specified, the table location will be used by default. * Type: STRING * Required: No ###### Example[​](#example-5 "Direct link to Example") Remove orphan files older than '2024-01-01 00:00:00' from the sub-directory `sub_dir` of the table location: ```sql -- With the parameter key specified: ALTER TABLE iceberg.sales.order EXECUTE remove_orphan_files(older_than = '2024-01-01 00:00:00', location = 's3://iceberg-bucket/iceberg_db/iceberg_table/sub_dir'); -- With the parameter key unspecified: ALTER TABLE iceberg.sales.order EXECUTE remove_orphan_files('2024-01-01 00:00:00', 's3://bucket-test/iceberg_db/iceberg_table/sub_dir'); ``` ##### Rewrite manifests[​](#rewrite-manifests "Direct link to Rewrite manifests") Rewrites data manifest files and merges them by partition to avoid performance degradation caused by excessive small manifests. note This operation rewrites the data manifests of the current snapshot only. ###### `rewrite_manifests` Syntax[​](#rewrite_manifests-syntax "Direct link to rewrite_manifests-syntax") ```sql ALTER TABLE [catalog.][database.]table_name EXECUTE rewrite_manifests() ``` ###### Example[​](#example-6 "Direct link to Example") Rewrite the data manifests of the current snapshot: ```sql ALTER TABLE iceberg.sales.order EXECUTE rewrite_manifests() ``` #### Table management[​](#table-management "Direct link to Table management") ##### Add files[​](#add-files "Direct link to Add files") Adds data files to an Iceberg table from either a source table or a specific location. This procedure supports Parquet and ORC file formats. ###### `add_files` Syntax[​](#add_files-syntax "Direct link to add_files-syntax") ```sql ALTER TABLE [catalog.][database.]table_name EXECUTE add_files( [source_table = '' | location = '', file_format = ''] [, recursive = ] ) ``` ###### Parameters[​](#parameters-6 "Direct link to Parameters") Either `source_table` or `location` must be provided, but not both. ###### `source_table`[​](#source_table "Direct link to source_table") * Description: The source table from which to add files. Format: 'catalog.database.table'. * Type: String * Required: No ###### `location`[​](#location-1 "Direct link to location-1") * Description: The directory path or file path from which to add files. * Type: String * Required: No ###### `file_format`[​](#file_format "Direct link to file_format") * Description: The format of the data files. Supported values: 'parquet', 'orc'. * Type: String * Required: No (required when using `location`) ###### `recursive`[​](#recursive "Direct link to recursive") * Description: Whether to recursively scan subdirectories when adding files from a location. * Type: Boolean * Default: true * Required: No ###### Example[​](#example-7 "Direct link to Example") Add files from a source table: ```sql ALTER TABLE iceberg.sales.order EXECUTE add_files(source_table = 'hive_catalog.sales.source_order'); ``` Add files from a specific location with Parquet format: ```sql ALTER TABLE iceberg.sales.order EXECUTE add_files(location = 's3://bucket/data/order/', file_format = 'parquet', recursive = true); ``` Add files from a single file: ```sql ALTER TABLE iceberg.sales.order EXECUTE add_files(location = 's3://bucket/data/order/data.parquet', file_format = 'parquet'); ``` ##### Register table[​](#register-table "Direct link to Register table") Registers an Iceberg table using a metadata file. This procedure allows you to add an existing Iceberg table to the catalog without migrating data. ###### `register_table` Syntax[​](#register_table-syntax "Direct link to register_table-syntax") ```sql CALL [catalog.]system.register_table( database_name = '', table_name = '', metadata_file = '' ) ``` ###### Parameters[​](#parameters-7 "Direct link to Parameters") ###### `database_name`[​](#database_name "Direct link to database_name") * Description: The name of the database in which to register the table. * Type: String * Required: Yes ###### `table_name`[​](#table_name "Direct link to table_name") * Description: The name of the table to register. * Type: String * Required: Yes ###### `metadata_file`[​](#metadata_file "Direct link to metadata_file") * Description: The path to the Iceberg table metadata file (e.g., metadata.json). * Type: String * Required: Yes ###### Example[​](#example-8 "Direct link to Example") Register a table using a metadata file: ```sql CALL iceberg_catalog.system.register_table( database_name = 'sales', table_name = 'order', metadata_file = 's3://bucket/metadata/sales/order/metadata/00001-xxxxx-xxxxx-xxxxx.metadata.json' ); ``` Or use the current catalog: ```sql CALL system.register_table( database_name = 'sales', table_name = 'order', metadata_file = 's3://bucket/metadata/sales/order/metadata/00001-xxxxx-xxxxx-xxxxx.metadata.json' ); ``` --- ### JDBC catalog Beta feature [Advice on use of Beta features](https://docs.starrocks.io/docs/introduction/maturity.md) StarRocks supports JDBC catalogs from v3.0 onwards. A JDBC catalog is a kind of external catalog that enables you to query data from data sources accessed through JDBC without ingestion. Also, you can directly transform and load data from JDBC data sources by using [INSERT INTO](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) based on JDBC catalogs. JDBC catalogs support MySQL and PostgreSQL from v3.0 onwards, Oracle and SQLServer since v3.2.9 and v3.3.1, and ClickHouse (Experimental) since v3.3.0. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") * The FEs and BEs or CNs in your StarRocks cluster can download the JDBC driver from the download URL specified by the `driver_url` parameter. * `JAVA_HOME` in the **$BE\_HOME/bin/start\_be.sh** file on each BE or CN node is properly configured as a path in the JDK environment instead of a path in the JRE environment. For example, you can configure `export JAVA_HOME = `. You must add this configuration at the beginning of the script and restart the BE or CN for the configuration to take effect. #### Create a JDBC catalog[​](#create-a-jdbc-catalog "Direct link to Create a JDBC catalog") ##### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE EXTERNAL CATALOG [COMMENT ] PROPERTIES ("key"="value", ...) ``` ##### Parameters[​](#parameters "Direct link to Parameters") ###### `catalog_name`[​](#catalog_name "Direct link to catalog_name") The name of the JDBC catalog. The naming conventions are as follows: * The name can contain letters, digits (0-9), and underscores (\_). It must start with a letter. * The name is case-sensitive and cannot exceed 1023 characters in length. ###### `comment`[​](#comment "Direct link to comment") The description of the JDBC catalog. This parameter is optional. ###### `PROPERTIES`[​](#properties "Direct link to properties") The properties of the JDBC Catalog. `PROPERTIES` must include the following parameters: | **Parameter** | **Description** | | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | type | The type of the resource. Set the value to `jdbc`. | | user | The username that is used to connect to the target database. | | password | The password that is used to connect to the target database. | | jdbc\_uri | The URI that the JDBC driver uses to connect to the target database. For MySQL, the URI is in the `"jdbc:mysql://ip:port"` format. For PostgreSQL, the URI is in the `"jdbc:postgresql://ip:port/db_name"` format. For more information: [PostgreSQL](https://jdbc.postgresql.org/documentation/head/connect.html). | | driver\_url | The download URL of the JDBC driver JAR package. An HTTP URL or file URL is supported, for example, `https://repo1.maven.org/maven2/org/postgresql/postgresql/42.3.3/postgresql-42.3.3.jar` and `file:///home/disk1/postgresql-42.3.3.jar`.
**NOTE**
You can also put the JDBC driver to any same path on the FE and BE or CN nodes and set `driver_url` to that path, which must be in the `file:////to/the/driver` format. | | driver\_class | The class name of the JDBC driver. The JDBC driver class names of common database engines are as follows:- MySQL: `com.mysql.jdbc.Driver` (MySQL v5.x and earlier) and `com.mysql.cj.jdbc.Driver` (MySQL v6.x and later)
- PostgreSQL: `org.postgresql.Driver`
- Oracle: `oracle.jdbc.driver.OracleDriver` | | schema\_resolver | (Optional) Explicitly specifies the schema resolver to use. Valid values: `postgresql`, `mysql`, `oracle`, `sqlserver`, `clickhouse`. Use this parameter when working with non-standard JDBC drivers that cannot be auto-detected by driver class name. If not specified, StarRocks will auto-detect the appropriate resolver based on the `driver_class` parameter. | ###### Optional Oracle properties[​](#optional-oracle-properties "Direct link to Optional Oracle properties") When `driver_class` is set to Oracle, you can configure the following optional properties: | **Parameter** | **Default** | **Description** | | ------------------------------ | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | oracle.number.default-scale | 6 | Set it when Oracle `NUMBER` metadata does not provide explicit precision and scale. Valid range: `0` to `38`. | | oracle.temporal.to-datetime | false | Controls Oracle `DATE`, `TIMESTAMP`, and `TIMESTAMP WITH LOCAL TIME ZONE` mapping. If it is set to `true`, these data types are mapped to StarRocks' `DATETIME` type; otherwise, `DATE` remains `DATE`, and `TIMESTAMP` / `TIMESTAMP WITH LOCAL TIME ZONE` are mapped to `VARCHAR(64)`. | | oracle.timestamptz.to-datetime | false | Controls Oracle `TIMESTAMP WITH TIME ZONE` mapping. If it is set to `true`, it is mapped to StarRocks' `DATETIME` type; otherwise, it is mapped to `VARCHAR(64)`. | ###### Optional row-count cache properties[​](#optional-row-count-cache-properties "Direct link to Optional row-count cache properties") StarRocks caches per-table row counts from JDBC sources to avoid blocking query planning. These properties let you tune the cache behavior per catalog. If not set, the global FE configuration values are used. | **Parameter** | **Default** | **Description** | | ------------------------------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | jdbc\_row\_count\_cache\_refresh\_sec | 600 | Background refresh interval (seconds). After this interval, the cached value is returned immediately while a reload runs asynchronously in the background. | | jdbc\_row\_count\_cache\_expire\_sec | 1200 | Hard eviction TTL (seconds). Cache entries not accessed within this window are evicted. Must be greater than `jdbc_row_count_cache_refresh_sec`. | | jdbc\_row\_count\_cache\_max\_size | 10000 | Maximum number of table entries in the row-count cache for this catalog. | > **NOTE** > > The FEs download the JDBC driver JAR package at the time of JDBC catalog creation, and the BEs or CNs download the JDBC driver JAR package at the time of the first query. The amount of time taken for the download varies depending on network conditions. ##### Examples[​](#examples "Direct link to Examples") The following example creates five different JDBC catalogs. ```sql -- PostgresSQL CREATE EXTERNAL CATALOG jdbc0 PROPERTIES ( "type"="jdbc", "user"="postgres", "password"="changeme", "jdbc_uri"="jdbc:postgresql://127.0.0.1:5432/jdbc_test", "driver_url"="https://repo1.maven.org/maven2/org/postgresql/postgresql/42.3.3/postgresql-42.3.3.jar", "driver_class"="org.postgresql.Driver" ); -- MySQL CREATE EXTERNAL CATALOG jdbc1 PROPERTIES ( "type"="jdbc", "user"="root", "password"="changeme", "jdbc_uri"="jdbc:mysql://127.0.0.1:3306", "driver_url"="https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.28/mysql-connector-java-8.0.28.jar", "driver_class"="com.mysql.cj.jdbc.Driver" ); -- Oracle CREATE EXTERNAL CATALOG jdbc2 PROPERTIES ( "type"="jdbc", "user"="root", "password"="changeme", "jdbc_uri"="jdbc:oracle:thin:@127.0.0.1:1521:ORCL", "driver_url"="https://repo1.maven.org/maven2/com/oracle/database/jdbc/ojdbc10/19.18.0.0/ojdbc10-19.18.0.0.jar", "driver_class"="oracle.jdbc.driver.OracleDriver" ); -- Oracle (with Oracle-specific optional properties) CREATE EXTERNAL CATALOG jdbc2_ext PROPERTIES ( "type"="jdbc", "user"="root", "password"="changeme", "jdbc_uri"="jdbc:oracle:thin:@127.0.0.1:1521/ORCLPDB1", "driver_url"="https://repo1.maven.org/maven2/com/oracle/database/jdbc/ojdbc10/19.18.0.0/ojdbc10-19.18.0.0.jar", "driver_class"="oracle.jdbc.driver.OracleDriver", "oracle.number.default-scale"="6", "oracle.temporal.to-datetime"="true", "oracle.timestamptz.to-datetime"="true" ); -- SQL Server CREATE EXTERNAL CATALOG jdbc3 PROPERTIES ( "type"="jdbc", "user"="root", "password"="changeme", "jdbc_uri"="jdbc:sqlserver://127.0.0.1:1433;databaseName=MyDatabase;", "driver_url"="https://repo1.maven.org/maven2/com/microsoft/sqlserver/mssql-jdbc/12.4.2.jre11/mssql-jdbc-12.4.2.jre11.jar", "driver_class"="com.microsoft.sqlserver.jdbc.SQLServerDriver" ); -- ClickHouse CREATE EXTERNAL CATALOG jdbc4 PROPERTIES ( "type"="jdbc", "user"="default", "jdbc_uri"="jdbc:clickhouse://127.0.0.1:8443", "driver_url"="https://repo1.maven.org/maven2/com/clickhouse/clickhouse-jdbc/0.4.6/clickhouse-jdbc-0.4.6.jar", "driver_class"="com.clickhouse.jdbc.ClickHouseDriver" ); ``` #### View JDBC catalogs[​](#view-jdbc-catalogs "Direct link to View JDBC catalogs") You can use [SHOW CATALOGS](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CATALOGS.md) to query all catalogs in the current StarRocks cluster: ```sql SHOW CATALOGS; ``` You can also use [SHOW CREATE CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CREATE_CATALOG.md) to query the creation statement of an external catalog. The following example queries the creation statement of a JDBC catalog named `jdbc0`: ```sql SHOW CREATE CATALOG jdbc0; ``` #### Drop a JDBC catalog[​](#drop-a-jdbc-catalog "Direct link to Drop a JDBC catalog") You can use [DROP CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/DROP_CATALOG.md) to drop a JDBC catalog. The following example drops a JDBC catalog named `jdbc0`: ```sql DROP Catalog jdbc0; ``` #### Query a table in a JDBC catalog[​](#query-a-table-in-a-jdbc-catalog "Direct link to Query a table in a JDBC catalog") 1. Use [SHOW DATABASES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATABASES.md) to view the databases in your JDBC-compatible cluster: ```sql SHOW DATABASES FROM ; ``` 2. Use [SET CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SET_CATALOG.md) to switch to the destination catalog in the current session: ```sql SET CATALOG ; ``` Then, use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to specify the active database in the current session: ```sql USE ; ``` Or, you can use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to directly specify the active database in the destination catalog: ```sql USE .; ``` 3. Use [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md) to query the destination table in the specified database: ```sql SELECT * FROM ; ``` #### FAQ[​](#faq "Direct link to FAQ") What do I do if an error suggesting "Malformed database URL, failed to parse the main URL sections" is thrown? If you encounter such an error, the URI that you passed in `jdbc_uri` is invalid. Check the URI that you pass and make sure it is valid. For more information, see the parameter descriptions in the "[PROPERTIES](#properties)" section of this topic. --- ### Kudu catalog Experimental feature [Advice on use of experimental features](https://docs.starrocks.io/docs/introduction/maturity.md) StarRocks supports Kudu catalogs from v3.3 onwards. A Kudu catalog is a kind of external catalog that enables you to query data from Apache Kudu without ingestion. Also, you can directly transform and load data from Kudu by using [INSERT INTO](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) based on Kudu catalogs. To ensure successful SQL workloads on your Kudu cluster, your StarRocks cluster needs to integrate with the following important components: * Metastore like your kudu file system or Hive metastore #### Usage notes[​](#usage-notes "Direct link to Usage notes") You can only use Kudu catalogs to query data. You cannot use Kudu catalogs to drop, delete, or insert data into your Kudu cluster. #### Integration preparations[​](#integration-preparations "Direct link to Integration preparations") Before you create a Kudu catalog, make sure your StarRocks cluster can integrate with the storage system and metastore of your Kudu cluster. > **NOTE** > > If an error indicating an unknown host is returned when you send a query, you must add the mapping between the host names and IP addresses of your KUDU cluster nodes to the **/etc/hosts** path. ##### Kerberos authentication[​](#kerberos-authentication "Direct link to Kerberos authentication") If Kerberos authentication is enabled for your KUDU cluster or Hive metastore, configure your StarRocks cluster as follows: * Run the `kinit -kt keytab_path principal` command on each FE and each BE to obtain Ticket Granting Ticket (TGT) from Key Distribution Center (KDC). To run this command, you must have the permissions to access your KUDU cluster and Hive metastore. Note that accessing KDC with this command is time-sensitive. Therefore, you need to use cron to run this command periodically. * Add `JAVA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf"` to the **$FE\_HOME/conf/fe.conf** file of each FE and to the **$BE\_HOME/conf/be.conf** file of each BE. In this example, `/etc/krb5.conf` is the save path of the **krb5.conf** file. You can modify the path based on your needs. #### Create a Kudu catalog[​](#create-a-kudu-catalog "Direct link to Create a Kudu catalog") ##### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE EXTERNAL CATALOG [COMMENT ] PROPERTIES ( "type" = "kudu", CatalogParams ) ``` ##### Parameters[​](#parameters "Direct link to Parameters") ###### catalog\_name[​](#catalog_name "Direct link to catalog_name") The name of the Kudu catalog. The naming conventions are as follows: * The name can contain letters, digits (0-9), and underscores (\_). It must start with a letter. * The name is case-sensitive and cannot exceed 1023 characters in length. ###### comment[​](#comment "Direct link to comment") The description of the Kudu catalog. This parameter is optional. ###### type[​](#type "Direct link to type") The type of your data source. Set the value to `kudu`. ###### CatalogParams[​](#catalogparams "Direct link to CatalogParams") A set of parameters about how StarRocks accesses the metadata of your Kudu cluster. The following table describes the parameter you need to configure in `CatalogParams`. | Parameter | Required | Description | | ----------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | kudu.catalog.type | Yes | The type of metastore that you use for your Kudu cluster. Set this parameter to `kudu` or `hive`. | | kudu.master | No | Specifies the Kudu Master address, which defaults to `localhost:7051`. | | hive.metastore.uris | No | The URI of your Hive metastore. Format: `thrift://:`. If high availability (HA) is enabled for your Hive metastore, you can specify multiple metastore URIs and separate them with commas (`,`), for example, `"thrift://:,thrift://:,thrift://:"`. | | kudu.schema-emulation.enabled | No | option to enable or disable the `schema` emulation. By default, it is turned off (false), which means that all tables belong to the `default` `schema`. | | kudu.schema-emulation.prefix | No | The prefix for `schema` emulation should only be set when `kudu.schema-emulation.enabled` = `true`. The default prefix used is empty string: \`\`. | > **NOTE** > > If you use Hive metastore, you must add the mapping between the host names and IP addresses of your Hive metastore nodes to the `/etc/hosts` path before you query Kudu data. Otherwise, StarRocks may fail to access your Hive metastore when you start a query. ##### Examples[​](#examples "Direct link to Examples") * The following examples create a Kudu catalog named `kudu_catalog` whose metastore type `kudu.catalog.type` is set to `kudu` to query data from your Kudu cluster. ```sql CREATE EXTERNAL CATALOG kudu_catalog PROPERTIES ( "type" = "kudu", "kudu.master" = "localhost:7051", "kudu.catalog.type" = "kudu", "kudu.schema-emulation.enabled" = "true", "kudu.schema-emulation.prefix" = "impala::" ); ``` * The following examples create a Kudu catalog named `kudu_catalog` whose metastore type `kudu.catalog.type` is set to `hive` to query data from your Kudu cluster. ```sql CREATE EXTERNAL CATALOG kudu_catalog PROPERTIES ( "type" = "kudu", "kudu.master" = "localhost:7051", "kudu.catalog.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "kudu.schema-emulation.enabled" = "true", "kudu.schema-emulation.prefix" = "impala::" ); ``` #### View Kudu catalogs[​](#view-kudu-catalogs "Direct link to View Kudu catalogs") You can use [SHOW CATALOGS](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CATALOGS.md) to query all catalogs in the current StarRocks cluster: ```sql SHOW CATALOGS; ``` You can also use [SHOW CREATE CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CREATE_CATALOG.md) to query the creation statement of an external catalog. The following example queries the creation statement of a Kudu catalog named `kudu_catalog`: ```sql SHOW CREATE CATALOG kudu_catalog; ``` #### Drop a Kudu catalog[​](#drop-a-kudu-catalog "Direct link to Drop a Kudu catalog") You can use [DROP CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/DROP_CATALOG.md) to drop an external catalog. The following example drops a Kudu catalog named `kudu_catalog`: ```sql DROP Catalog kudu_catalog; ``` #### View the schema of a Kudu table[​](#view-the-schema-of-a-kudu-table "Direct link to View the schema of a Kudu table") You can use one of the following syntaxes to view the schema of a Kudu table: * View schema ```sql DESC[RIBE] ..; ``` * View schema and location from the CREATE statement ```sql SHOW CREATE TABLE ..; ``` #### Query a Kudu table[​](#query-a-kudu-table "Direct link to Query a Kudu table") 1. Use [SHOW DATABASES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATABASES.md) to view the databases in your Kudu cluster: ```sql SHOW DATABASES FROM ; ``` 2. Use [SET CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SET_CATALOG.md) to switch to the destination catalog in the current session: ```sql SET CATALOG ; ``` Then, use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to specify the active database in the current session: ```sql USE ; ``` Or, you can use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to directly specify the active database in the destination catalog: ```sql USE .; ``` 3. Use [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md) to query the destination table in the specified database: ```sql SELECT count(*) FROM LIMIT 10; ``` #### Load data from Kudu[​](#load-data-from-kudu "Direct link to Load data from Kudu") Suppose you have an OLAP table named `olap_tbl`, you can transform and load data like below: ```sql INSERT INTO default_catalog.olap_db.olap_tbl SELECT * FROM kudu_table; ``` --- ### MaxCompute catalog Beta feature [Advice on use of Beta features](https://docs.starrocks.io/docs/introduction/maturity.md) StarRocks supports Alibaba Cloud MaxCompute (previously known as ODPS) catalogs from v3.3 onwards. A MaxCompute catalog is a kind of external catalog that enables you to query data from MaxCompute without ingestion. With MaxCompute catalogs, you also can directly transform and load the data from MaxCompute by using [INSERT INTO](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md). #### Usage notes[​](#usage-notes "Direct link to Usage notes") You can use MaxCompute catalogs only to query the data from MaxCompute. You cannot use MaxCompute catalogs to drop, delete, or insert data into your MaxCompute cluster. #### Integration preparations[​](#integration-preparations "Direct link to Integration preparations") Before creating a MaxCompute catalog, make sure that your StarRocks cluster can access your MaxCompute service properly. #### Create a MaxCompute catalog[​](#create-a-maxcompute-catalog "Direct link to Create a MaxCompute catalog") ##### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE EXTERNAL CATALOG [COMMENT ] PROPERTIES ( "type" = "odps", CatalogParams, ScanParams, CachingMetaParams ) ``` ##### Parameters[​](#parameters "Direct link to Parameters") ###### catalog\_name[​](#catalog_name "Direct link to catalog_name") The name of the MaxCompute catalog. The naming conventions are as follows: * The name can contain letters, digits (0-9), and underscores (\_). It must start with a letter. * The name is case-sensitive and cannot exceed 1023 characters in length. ###### comment[​](#comment "Direct link to comment") The description of the MaxCompute catalog. This parameter is optional. ###### type[​](#type "Direct link to type") The type of your data source. Set the value to `odps`. ###### CatalogParams[​](#catalogparams "Direct link to CatalogParams") A set of parameters about how StarRocks accesses the metadata of the MaxCompute cluster. The following table describes the parameter you need to configure in `CatalogParams`. | Parameter | Required | Description | | -------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | odps.endpoint | Yes | The connection address (namely, endpoint) for the MaxCompute service. You need to configure the endpoint according to the region selected when creating the MaxCompute project as well as the network connection mode. For details about the endpoints used in different regions and network connection modes, see [Endpoint](https://www.alibabacloud.com/help/en/maxcompute/user-guide/endpoints). Note that currently only two network connection modes of Alibaba Cloud are supported to provide the best experience: VPC and classic network. | | odps.project | Yes | The name of the MaxCompute project you want to access. If you have created a standard mode workspace, pay attention to the differences between the project names for the production environment and the development environment (\_dev) when configuring this parameter. You can log in to the [MaxCompute Console](https://account.alibabacloud.com/login/login.htm?spm=5176.12901015-2.0.0.593a525cwmiD7c), and obtain the MaxCompute project name on the **Workspace** > **Project Management** page. | | odps.access.id | Yes | The AccessKey ID of the Alibaba Cloud account or RAM user. You can enter the [AccessKey Management](https://ram.console.aliyun.com/manage/ak) page to obtain the AccessKey ID. | | odps.access.key | Yes | The AccessKey Secret matching the AccessKey ID. You can enter the [AccessKey Management](https://ram.console.aliyun.com/manage/ak) page to obtain the AccessKey Secret. | | odps.tunnel.endpoint | No | The public network access link for the Tunnel service. If you have not configured the Tunnel endpoint, Tunnel will automatically route to the Tunnel endpoint matching the network where the MaxCompute service is located. If you have configured the Tunnel endpoint, it will be used as configured and not automatically routed. | | odps.tunnel.quota | Yes | The name of the quota that is used to access MaxCompute. MaxCompute provides two types of resources for data transmission: exclusive resource group for MaxCompute Tunnel (subscription) and storage API (pay-as-you-go). You can perform the following operations to obtain the quota name based on the resource type.**Exclusive resource group for MaxCompute Tunnel**: Log on to the [MaxCompute console](https://maxcompute.console.aliyun.com/). In the top navigation bar, select a region. In the left-side navigation pane, choose Workspace > Quotas to view the available quotas. For more information, see [Manage quotas for computing resources in the MaxCompute console](https://help.aliyun.com/zh/maxcompute/user-guide/manage-quotas-in-the-maxcompute-console).**Storage API**: Log on to the [MaxCompute console](https://maxcompute.console.aliyun.com/). In the left-side navigation pane, choose Tenants > Tenant Property. On the Tenants page, turn on Storage API Switch. For more information, see [Use storage API (pay-as-you-go)](https://help.aliyun.com/zh/maxcompute/user-guide/overview-1). The default name of the storage API is **"pay-as-you-go"**. | ###### ScanParams[​](#scanparams "Direct link to ScanParams") A set of parameters about how StarRocks accesses the files stored in the MaxCompute cluster. This parameter set is optional. The following table describes the parameter you need to configure in `ScanParams`. | Parameter | Required | Description | | --------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | odps.split.policy | No | The shard policy used when for data scanning.
Valid values: `size` (shard by data size) and `row_offset` (shard by number of rows). Default value: `size`.
| | odps.split.row.count | No | The maximum number of rows per shard when `odps.split.policy` is set to `row_offset`.
Default value: `4 * 1024 * 1024 = 4194304`.
| ###### CachingMetaParams[​](#cachingmetaparams "Direct link to CachingMetaParams") A set of parameters about how StarRocks caches the metadata of Hive. This parameter set is optional. The following table describes the parameter you need to configure in `CachingMetaParams`. | Parameter | Required | Description | | ---------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | odps.cache.table.enable | No | Specifies whether StarRocks caches the metadata of MaxCompute tables. Valid values: `true` and `false`. Default value: `true`. The value `true` enables the cache, and the value `false` disables the cache. | | odps.cache.table.expire | No | The time interval, in seconds, at which StarRocks automatically evicts the cached metadata of MaxCompute tables or partitions. Default value: `86400` (24 hours). | | odps.cache.table.size | No | The number of MaxCompute table metadata entries that StarRocks caches. Default value: `1000`. | | odps.cache.partition.enable | No | Specifies whether StarRocks caches the metadata of all partitions for a MaxCompute table. Valid values: `true` and `false`. Default value: `true`. The value `true` enables the cache, and the value `false` disables the cache. | | odps.cache.partition.expire | No | The time interval, in seconds, at which StarRocks automatically evicts the cached metadata of all partitions for a MaxCompute table. Default value: `86400` (24 hours). | | odps.cache.partition.size | No | The number of MaxCompute tables for which StarRocks caches the metadata of all partitions. Default value: `1000`. | | odps.cache.table-name.enable | No | Specifies whether StarRocks caches the information of the tables from the MaxCompute project. Valid values: `true` and `false`. Default value: `false`. The value `true` enables the cache, and the value `false` disables the cache. | | odps.cache.table-name.expire | No | The time interval, in seconds, at which StarRocks automatically evicts the cached information of the tables from the MaxCompute project. Default value: `86400` (24 hours). | | odps.cache.table-name.size | No | The number of MaxCompute projects that StarRocks caches. Default value: `1000`. | ##### Examples[​](#examples "Direct link to Examples") The following example creates a MaxCompute catalog named `odps_catalog` which uses `odps_project` as the warehouse project. ```sql CREATE EXTERNAL CATALOG odps_catalog PROPERTIES ( "type"="odps", "odps.access.id"="", "odps.access.key"="", "odps.endpoint"="", "odps.project"="odps_project" ); ``` #### View MaxCompute catalogs[​](#view-maxcompute-catalogs "Direct link to View MaxCompute catalogs") You can use [SHOW CATALOGS](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CATALOGS.md) to query all catalogs in the current StarRocks cluster: ```sql SHOW CATALOGS; ``` You can also use [SHOW CREATE CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CREATE_CATALOG.md) to query the creation statement of an external catalog. The following example queries the creation statement of a MaxCompute catalog named `odps_catalog`: ```sql SHOW CREATE CATALOG odps_catalog; ``` #### Drop a MaxCompute catalog[​](#drop-a-maxcompute-catalog "Direct link to Drop a MaxCompute catalog") You can use [DROP CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/DROP_CATALOG.md) to drop an external catalog. The following example drops a MaxCompute catalog named `odps_catalog`: ```sql DROP CATALOG odps_catalog; ``` #### View the schema of a MaxCompute table[​](#view-the-schema-of-a-maxcompute-table "Direct link to View the schema of a MaxCompute table") You can use one of the following syntaxes to view the schema of a MaxCompute table: * View schema ```sql DESC[RIBE] ..; ``` * View schema and location from the CREATE statement ```sql SHOW CREATE TABLE ..; ``` #### Query a MaxCompute table[​](#query-a-maxcompute-table "Direct link to Query a MaxCompute table") 1. Use [SHOW DATABASES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATABASES.md) to view the databases in your MaxCompute cluster: ```sql SHOW DATABASES FROM ; ``` 2. Use [SET CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SET_CATALOG.md) to switch to the destination catalog in the current session: ```sql SET CATALOG ; ``` Then, use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to specify the active database in the current session: ```sql USE ; ``` Or, you can use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to directly specify the active database in the destination catalog: ```sql USE .; ``` 3. Use [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md) to query the destination table in the specified database: ```sql SELECT count(*) FROM LIMIT 10; ``` #### Load data from MaxCompute[​](#load-data-from-maxcompute "Direct link to Load data from MaxCompute") Suppose that there is an OLAP table named `olap_tbl` in your StarRock cluster and there is a table named `mc_table` in your MaxCompute cluster. You can transform and load the data from the MaxCompute table `mc_table` into the StarRocks table `olap_tbl` like below: ```sql INSERT INTO default_catalog.olap_db.olap_tbl SELECT * FROM mc_table; ``` #### Data type mapping[​](#data-type-mapping "Direct link to Data type mapping") MaxCompute catalogs map the MaxCompute data types to the StarRocks data types. The following table shows the mapping between the MaxCompute data types to the StarRocks data types. | MaxCompute data type | StarRocks data type | | -------------------- | ------------------- | | BOOLEAN | BOOLEAN | | TINYINT | TINYINT | | SMALLINT | SMALLINT | | INT | INT | | BIGINT | BIGINT | | FLOAT | FLOAT | | DOUBLE | DOUBLE | | DECIMAL(p, s) | DECIMAL(p, s) | | STRING | VARCHAR(1073741824) | | VARCHAR(n) | VARCHAR(n) | | CHAR(n) | CHAR(n) | | JSON | VARCHAR(1073741824) | | BINARY | VARBINARY | | DATE | DATE | | DATETIME | DATETIME | | TIMESTAMP | DATETIME | | ARRAY | ARRAY | | MAP | MAP | | STRUCT | STRUCT | note The TIMESTAMP type will lose precision due to type conversion in StarRocks. #### Collect CBO statistics[​](#collect-cbo-statistics "Direct link to Collect CBO statistics") In the current version, MaxCompute catalogs cannot automatically collect CBO statistics for MaxCompute tables, and consequently the optimizer may not be able to generate the optimal query plans. As such, manually scanning the CBO statistics for MaxCompute tables and importing them into StarRocks can effectively expedite queries. Suppose that there is a MaxCompute table named `mc_table` in your MaxCompute cluster. You can create a manual collection task for collecting CBO statistics by using [ANALYZE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cbo_stats/ANALYZE_TABLE.md): ```sql ANALYZE TABLE mc_table; ``` #### Manually update metadata cache[​](#manually-update-metadata-cache "Direct link to Manually update metadata cache") By default, StarRocks caches the metadata of MaxCompute to improve query performance. Therefore, after making schema changes or other updates to a MaxCompute table, you can use [REFRESH EXTERNAL TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/REFRESH_EXTERNAL_TABLE.md) to manually update the metadata of the table, thereby ensuring that StarRocks can obtain the most recent metadata promptly: ```sql REFRESH EXTERNAL TABLE [PARTITION ('partition_name', ...)] ``` --- ### Paimon catalog Beta feature [Advice on use of Beta features](https://docs.starrocks.io/docs/introduction/maturity.md) StarRocks supports Paimon catalogs from v3.1 onwards. A Paimon catalog is a kind of external catalog that enables you to query data from Apache Paimon without ingestion. Also, you can directly transform and load data from Paimon by using [INSERT INTO](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) based on Paimon catalogs. To ensure successful SQL workloads on your Paimon cluster, your StarRocks cluster must be able to access the storage system and metastore of your Paimon cluster. StarRocks supports the following storage systems and metastores: * Distributed file system (HDFS) or object storage like AWS S3, Microsoft Azure Storage, Google GCS, or or other S3-compatible storage system (for example, MinIO) * Metastore like your file system or Hive metastore #### Usage notes[​](#usage-notes "Direct link to Usage notes") You can only use Paimon catalogs to query data. You cannot use Paimon catalogs to drop, delete, or insert data into your Paimon cluster. #### Paimon to StarRocks data types[​](#paimon-to-starrocks-data-types "Direct link to Paimon to StarRocks data types") | Paimon Type | StarRocks Type | | --------------------- | --------------------------- | | `BINARY` | `VARBINARY` | | `VARBINARY` | `VARBINARY` | | `CHAR` | `CHAR(length)` | | `VARCHAR` | `VARCHAR` | | `BOOLEAN` | `BOOLEAN` | | `DECIMAL` | `DECIMAL(precision, scale)` | | `TINYINT` | `TINYINT` | | `SMALLINT` | `SMALLINT` | | `INT` | `INT` | | `BIGINT` | `BIGINT` | | `FLOAT` | `FLOAT` | | `DOUBLE` | `DOUBLE` | | `DATE` | `DATE` | | `TIME` | `TIME` | | `TIMESTAMP` | `DATETIME` | | `LocalZonedTimestamp` | `DATETIME` | | `ARRAY` | `ARRAY` | | `MAP` | `MAP` | | `ROW/STRUCT` | `STRUCT` | #### Integration preparations[​](#integration-preparations "Direct link to Integration preparations") Before you create a Paimon catalog, make sure your StarRocks cluster can integrate with the storage system and metastore of your Paimon cluster. ##### AWS IAM[​](#aws-iam "Direct link to AWS IAM") If your Paimon cluster uses AWS S3 as storage, choose your suitable authentication method and make the required preparations to ensure that your StarRocks cluster can access the related AWS cloud resources. The following authentication methods are recommended: * Instance profile (recommended) * Assumed role * IAM user Of the above-mentioned three authentication methods, instance profile is the most widely used. For more information, see [Preparation for authentication in AWS IAM](https://docs.starrocks.io/docs/integrations/authenticate_to_aws_resources.md#preparation-for-iam-user-based-authentication). ##### HDFS[​](#hdfs "Direct link to HDFS") If you choose HDFS as storage, configure your StarRocks cluster as follows: * (Optional) Set the username that is used to access your HDFS cluster and Hive metastore. By default, StarRocks uses the username of the FE and BE or CN processes to access your HDFS cluster and Hive metastore. You can also set the username by adding `export HADOOP_USER_NAME=""` at the beginning of the **fe/conf/hadoop\_env.sh** file of each FE and at the beginning of the **be/conf/hadoop\_env.sh** file of each BE or the **cn/conf/hadoop\_env.sh** file of each CN. After you set the username in these files, restart each FE and each BE or CN to make the parameter settings take effect. You can set only one username for each StarRocks cluster. * When you query Paimon data, the FEs and BEs or CNs of your StarRocks cluster use the HDFS client to access your HDFS cluster. In most cases, you do not need to configure your StarRocks cluster to achieve that purpose, and StarRocks starts the HDFS client using the default configurations. You need to configure your StarRocks cluster only in the following situations: * High availability (HA) is enabled for your HDFS cluster: Add the **hdfs-site.xml** file of your HDFS cluster to the **$FE\_HOME/conf** path of each FE and to the **$BE\_HOME/conf** path of each BE or the **$CN\_HOME/conf** path of each CN. * View File System (ViewFs) is enabled for your HDFS cluster: Add the **core-site.xml** file of your HDFS cluster to the **$FE\_HOME/conf** path of each FE and to the **$BE\_HOME/conf** path of each BE or the **$CN\_HOME/conf** path of each CN. > **NOTE** > > If an error indicating an unknown host is returned when you send a query, you must add the mapping between the host names and IP addresses of your HDFS cluster nodes to the **/etc/hosts** path. ##### Kerberos authentication[​](#kerberos-authentication "Direct link to Kerberos authentication") If Kerberos authentication is enabled for your HDFS cluster or Hive metastore, configure your StarRocks cluster as follows: * Run the `kinit -kt keytab_path principal` command on each FE and each BE or CN to obtain Ticket Granting Ticket (TGT) from Key Distribution Center (KDC). To run this command, you must have the permissions to access your HDFS cluster and Hive metastore. Note that accessing KDC with this command is time-sensitive. Therefore, you need to use cron to run this command periodically. * Add `JAVA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf"` to the **$FE\_HOME/conf/fe.conf** file of each FE and to the **$BE\_HOME/conf/be.conf** file of each BE or the **$CN\_HOME/conf/cn.conf** file of each CN. In this example, `/etc/krb5.conf` is the save path of the **krb5.conf** file. You can modify the path based on your needs. #### Create a Paimon catalog[​](#create-a-paimon-catalog "Direct link to Create a Paimon catalog") ##### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE EXTERNAL CATALOG [COMMENT ] PROPERTIES ( "type" = "paimon", CatalogParams, StorageCredentialParams, ) ``` ##### Parameters[​](#parameters "Direct link to Parameters") ###### catalog\_name[​](#catalog_name "Direct link to catalog_name") The name of the Paimon catalog. The naming conventions are as follows: * The name can contain letters, digits (0-9), and underscores (\_). It must start with a letter. * The name is case-sensitive and cannot exceed 1023 characters in length. ###### comment[​](#comment "Direct link to comment") The description of the Paimon catalog. This parameter is optional. ###### type[​](#type "Direct link to type") The type of your data source. Set the value to `paimon`. ###### CatalogParams[​](#catalogparams "Direct link to CatalogParams") A set of parameters about how StarRocks accesses the metadata of your Paimon cluster. The following table describes the parameter you need to configure in `CatalogParams`. | Parameter | Required | Description | | ------------------------ | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | paimon.catalog.type | Yes | The type of metastore that you use for your Paimon cluster. Set this parameter to `filesystem` or `hive`. | | paimon.catalog.warehouse | Yes | The warehouse storage path of your Paimon data. | | hive.metastore.uris | No | The URI of your Hive metastore. Format: `thrift://:`. If high availability (HA) is enabled for your Hive metastore, you can specify multiple metastore URIs and separate them with commas (`,`), for example, `"thrift://:,thrift://:,thrift://:"`. | > **NOTE** > > If you use Hive metastore, you must add the mapping between the host names and IP addresses of your Hive metastore nodes to the `/etc/hosts` path before you query Paimon data. Otherwise, StarRocks may fail to access your Hive metastore when you start a query. ###### StorageCredentialParams[​](#storagecredentialparams "Direct link to StorageCredentialParams") A set of parameters about how StarRocks integrates with your storage system. This parameter set is optional. If you use HDFS as storage, you do not need to configure `StorageCredentialParams`. If you use AWS S3, other S3-compatible storage system, Microsoft Azure Storage, or Google GCS as storage, you must configure `StorageCredentialParams`. ###### AWS S3[​](#aws-s3 "Direct link to AWS S3") If you choose AWS S3 as storage for your Paimon cluster, take one of the following actions: * To choose the instance profile-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "true", "aws.s3.endpoint" = "" ``` * To choose the assumed role-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "", "aws.s3.endpoint" = "" ``` * To choose the IAM user-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.endpoint" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ----------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | aws.s3.use\_instance\_profile | Yes | Specifies whether to enable the instance profile-based authentication method and the assumed role-based authentication method. Valid values: `true` and `false`. Default value: `false`. | | aws.s3.iam\_role\_arn | No | The ARN of the IAM role that has privileges on your AWS S3 bucket. If you use the assumed role-based authentication method to access AWS S3, you must specify this parameter. | | aws.s3.endpoint | Yes | The endpoint that is used to connect to your AWS S3 bucket. For example, `https://s3.us-west-2.amazonaws.com`. | | aws.s3.access\_key | No | The access key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter. | | aws.s3.secret\_key | No | The secret key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter. | For information about how to choose an authentication method for accessing AWS S3 and how to configure an access control policy in AWS IAM Console, see [Authentication parameters for accessing AWS S3](https://docs.starrocks.io/docs/integrations/authenticate_to_aws_resources.md#authentication-parameters-for-accessing-aws-s3). ###### S3-compatible storage system[​](#s3-compatible-storage-system "Direct link to S3-compatible storage system") If you choose an S3-compatible storage system, such as MinIO, as storage for your Paimon cluster, configure `StorageCredentialParams` as follows to ensure a successful integration: ```sql "aws.s3.enable_ssl" = "false", "aws.s3.enable_path_style_access" = "true", "aws.s3.endpoint" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ---------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | aws.s3.enable\_ssl | Yes | Specifies whether to enable SSL connection.
Valid values: `true` and `false`. Default value: `true`. | | aws.s3.enable\_path\_style\_access | Yes | Specifies whether to enable path-style access.
Valid values: `true` and `false`. Default value: `false`. For MinIO, you must set the value to `true`.
Path-style URLs use the following format: `https://s3..amazonaws.com//`. For example, if you create a bucket named `DOC-EXAMPLE-BUCKET1` in the US West (Oregon) Region, and you want to access the `alice.jpg` object in that bucket, you can use the following path-style URL: `https://s3.us-west-2.amazonaws.com/DOC-EXAMPLE-BUCKET1/alice.jpg`. | | aws.s3.endpoint | Yes | The endpoint that is used to connect to your S3-compatible storage system instead of AWS S3. | | aws.s3.access\_key | Yes | The access key of your IAM user. | | aws.s3.secret\_key | Yes | The secret key of your IAM user. | ###### Microsoft Azure Storage[​](#microsoft-azure-storage "Direct link to Microsoft Azure Storage") ###### Azure Blob Storage[​](#azure-blob-storage "Direct link to Azure Blob Storage") If you choose Blob Storage as storage for your Paimon cluster, take one of the following actions: * To choose the Shared Key authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.blob.storage_account" = "", "azure.blob.shared_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | --------------------------- | -------- | -------------------------------------------- | | azure.blob.storage\_account | Yes | The username of your Blob Storage account. | | azure.blob.shared\_key | Yes | The shared key of your Blob Storage account. | * To choose the SAS Token authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.blob.storage_account" = "", "azure.blob.container" = "", "azure.blob.sas_token" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | --------------------------- | -------- | --------------------------------------------------------------- | | azure.blob.storage\_account | Yes | The username of your Blob Storage account. | | azure.blob.container | Yes | The name of the blob container that stores your data. | | azure.blob.sas\_token | Yes | The SAS token that is used to access your Blob Storage account. | ###### Azure Data Lake Storage Gen2[​](#azure-data-lake-storage-gen2 "Direct link to Azure Data Lake Storage Gen2") If you choose Data Lake Storage Gen2 as storage for your Paimon cluster, take one of the following actions: * To choose the Managed Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ------------------------------------------ | -------- | ------------------------------------------------------------------------------------------------ | | azure.adls2.oauth2\_use\_managed\_identity | Yes | Specifies whether to enable the Managed Identity authentication method. Set the value to `true`. | | azure.adls2.oauth2\_tenant\_id | Yes | The ID of the tenant whose data you want to access. | | azure.adls2.oauth2\_client\_id | Yes | The client (application) ID of the managed identity. | * To choose the Shared Key authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.storage_account" = "", "azure.adls2.shared_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ---------------------------- | -------- | -------------------------------------------------------------- | | azure.adls2.storage\_account | Yes | The username of your Data Lake Storage Gen2 storage account. | | azure.adls2.shared\_key | Yes | The shared key of your Data Lake Storage Gen2 storage account. | * To choose the Service Principal authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_client_secret" = "", "azure.adls2.oauth2_client_endpoint" = "" ``` The following table describes the parameters you need to configure `in StorageCredentialParams`. | Parameter | Required | Description | | ------------------------------------ | -------- | -------------------------------------------------------------------------- | | azure.adls2.oauth2\_client\_id | Yes | The client (application) ID of the service principal. | | azure.adls2.oauth2\_client\_secret | Yes | The value of the new client (application) secret created. | | azure.adls2.oauth2\_client\_endpoint | Yes | The OAuth 2.0 token endpoint (v1) of the service principal or application. | * To choose the Workload Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_token_file" = "", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | azure.adls2.oauth2\_token\_file | Yes | The absolute file path to the OAuth2 token file projected into the pod by the Azure Workload Identity webhook. | | azure.adls2.oauth2\_tenant\_id | Yes | The ID of the tenant whose data you want to access. | | azure.adls2.oauth2\_client\_id | Yes | The client ID (application ID) of the Azure AD application (user-assigned managed identity or app registration) associated with the workload identity. | ###### Azure Data Lake Storage Gen1[​](#azure-data-lake-storage-gen1 "Direct link to Azure Data Lake Storage Gen1") If you choose Data Lake Storage Gen1 as storage for your Paimon cluster, take one of the following actions: * To choose the Managed Service Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls1.use_managed_service_identity" = "true" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ------------------------------------------- | -------- | -------------------------------------------------------------------------------------------------------- | | azure.adls1.use\_managed\_service\_identity | Yes | Specifies whether to enable the Managed Service Identity authentication method. Set the value to `true`. | * To choose the Service Principal authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls1.oauth2_client_id" = "", "azure.adls1.oauth2_credential" = "", "azure.adls1.oauth2_endpoint" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ------------------------------ | -------- | -------------------------------------------------------------------------- | | azure.adls1.oauth2\_client\_id | Yes | The client (application) ID of the service principal. | | azure.adls1.oauth2\_credential | Yes | The value of the new client (application) secret created. | | azure.adls1.oauth2\_endpoint | Yes | The OAuth 2.0 token endpoint (v1) of the service principal or application. | ###### Google GCS[​](#google-gcs "Direct link to Google GCS") If you choose Google GCS as storage for your Paimon cluster, take one of the following actions: * To choose the VM-based authentication method, configure `StorageCredentialParams` as follows: ```sql "gcp.gcs.use_compute_engine_service_account" = "true" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Default value | Value example | Description | | ---------------------------------------------- | ------------- | ------------- | ------------------------------------------------------------------------------------------- | | gcp.gcs.use\_compute\_engine\_service\_account | FALSE | TRUE | Specifies whether to directly use the service account that is bound to your Compute Engine. | * To choose the service account-based authentication method, configure `StorageCredentialParams` as follows: ```sql "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Default value | Value example | Description | | ------------------------------------------ | ------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------- | | gcp.gcs.service\_account\_email | "" | "" | The email address in the JSON file generated at the creation of the service account. | | gcp.gcs.service\_account\_private\_key\_id | "" | "61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea" | The private key ID in the JSON file generated at the creation of the service account. | | gcp.gcs.service\_account\_private\_key | "" | "-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n" | The private key in the JSON file generated at the creation of the service account. | * To choose the impersonation-based authentication method, configure `StorageCredentialParams` as follows: * Make a VM instance impersonate a service account: ```sql "gcp.gcs.use_compute_engine_service_account" = "true", "gcp.gcs.impersonation_service_account" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Default value | Value example | Description | | ---------------------------------------------- | ------------- | ------------- | ------------------------------------------------------------------------------------------- | | gcp.gcs.use\_compute\_engine\_service\_account | FALSE | TRUE | Specifies whether to directly use the service account that is bound to your Compute Engine. | | gcp.gcs.impersonation\_service\_account | "" | "hello" | The service account that you want to impersonate. | * Make a service account (temporarily named as meta service account) impersonate another service account (temporarily named as data service account): ```sql "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "", "gcp.gcs.impersonation_service_account" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Default value | Value example | Description | | ------------------------------------------ | ------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------ | | gcp.gcs.service\_account\_email | "" | "" | The email address in the JSON file generated at the creation of the meta service account. | | gcp.gcs.service\_account\_private\_key\_id | "" | "61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea" | The private key ID in the JSON file generated at the creation of the meta service account. | | gcp.gcs.service\_account\_private\_key | "" | "-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n" | The private key in the JSON file generated at the creation of the meta service account. | | gcp.gcs.impersonation\_service\_account | "" | "hello" | The data service account that you want to impersonate. | ##### Examples[​](#examples "Direct link to Examples") The following examples create a Paimon catalog named `paimon_catalog_fs` whose metastore type `paimon.catalog.type` is set to `filesystem` to query data from your Paimon cluster. ###### AWS S3[​](#aws-s3-1 "Direct link to AWS S3") * If you choose the instance profile-based authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG paimon_catalog_fs PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", "paimon.catalog.warehouse" = "", "aws.s3.use_instance_profile" = "true", "aws.s3.endpoint" = "" ); ``` * If you choose the assumed role-based authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG paimon_catalog_fs PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", "paimon.catalog.warehouse" = "", "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::081976408565:role/test_s3_role", "aws.s3.endpoint" = "" ); ``` * If you choose the IAM user-based authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG paimon_catalog_fs PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", "paimon.catalog.warehouse" = "", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.endpoint" = "" ); ``` ###### S3-compatible storage system[​](#s3-compatible-storage-system-1 "Direct link to S3-compatible storage system") Use MinIO as an example. Run a command like below: ```sql CREATE EXTERNAL CATALOG paimon_catalog_fs PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", "paimon.catalog.warehouse" = "", "aws.s3.enable_ssl" = "true", "aws.s3.enable_path_style_access" = "true", "aws.s3.endpoint" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ); ``` ###### Microsoft Azure Storage[​](#microsoft-azure-storage-1 "Direct link to Microsoft Azure Storage") ###### Azure Blob Storage[​](#azure-blob-storage-1 "Direct link to Azure Blob Storage") * If you choose the Shared Key authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG paimon_catalog_fs PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", "paimon.catalog.warehouse" = "", "azure.blob.storage_account" = "", "azure.blob.shared_key" = "" ); ``` * If you choose the SAS Token authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG paimon_catalog_fs PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", "paimon.catalog.warehouse" = "", "azure.blob.storage_account" = "", "azure.blob.container" = "", "azure.blob.sas_token" = "" ); ``` ###### Azure Data Lake Storage Gen1[​](#azure-data-lake-storage-gen1-1 "Direct link to Azure Data Lake Storage Gen1") * If you choose the Managed Service Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG paimon_catalog_fs PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", "paimon.catalog.warehouse" = "", "azure.adls1.use_managed_service_identity" = "true" ); ``` * If you choose the Service Principal authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG paimon_catalog_fs PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", "paimon.catalog.warehouse" = "", "azure.adls1.oauth2_client_id" = "", "azure.adls1.oauth2_credential" = "", "azure.adls1.oauth2_endpoint" = "" ); ``` ###### Azure Data Lake Storage Gen2[​](#azure-data-lake-storage-gen2-1 "Direct link to Azure Data Lake Storage Gen2") * If you choose the Managed Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG paimon_catalog_fs PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", "paimon.catalog.warehouse" = "", "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` * If you choose the Shared Key authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG paimon_catalog_fs PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", "paimon.catalog.warehouse" = "", "azure.adls2.storage_account" = "", "azure.adls2.shared_key" = "" ); ``` * If you choose the Service Principal authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG paimon_catalog_fs PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", "paimon.catalog.warehouse" = "", "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_client_secret" = "", "azure.adls2.oauth2_client_endpoint" = "" ); ``` * If you choose the Workload Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG paimon_catalog_fs PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", "paimon.catalog.warehouse" = "", "azure.adls2.oauth2_token_file" = "/var/run/secrets/azure/tokens/azure-identity-token", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` ###### Google GCS[​](#google-gcs-1 "Direct link to Google GCS") * If you choose the VM-based authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG paimon_catalog_fs PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", "paimon.catalog.warehouse" = "", "gcp.gcs.use_compute_engine_service_account" = "true" ); ``` * If you choose the service account-based authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG paimon_catalog_fs PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", "paimon.catalog.warehouse" = "", "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "" ); ``` * If you choose the impersonation-based authentication method: * If you make a VM instance impersonate a service account, run a command like below: ```sql CREATE EXTERNAL CATALOG paimon_catalog_fs PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", "paimon.catalog.warehouse" = "", "gcp.gcs.use_compute_engine_service_account" = "true", "gcp.gcs.impersonation_service_account" = "" ); ``` * If you make a service account impersonate another service account, run a command like below: ```sql CREATE EXTERNAL CATALOG paimon_catalog_fs PROPERTIES ( "type" = "paimon", "paimon.catalog.type" = "filesystem", "paimon.catalog.warehouse" = "", "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "", "gcp.gcs.impersonation_service_account" = "" ); ``` #### View Paimon catalogs[​](#view-paimon-catalogs "Direct link to View Paimon catalogs") You can use [SHOW CATALOGS](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CATALOGS.md) to query all catalogs in the current StarRocks cluster: ```sql SHOW CATALOGS; ``` You can also use [SHOW CREATE CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CREATE_CATALOG.md) to query the creation statement of an external catalog. The following example queries the creation statement of a Paimon catalog named `paimon_catalog_fs`: ```sql SHOW CREATE CATALOG paimon_catalog_fs; ``` #### Drop a Paimon catalog[​](#drop-a-paimon-catalog "Direct link to Drop a Paimon catalog") You can use [DROP CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/DROP_CATALOG.md) to drop an external catalog. The following example drops a Paimon catalog named `paimon_catalog_fs`: ```sql DROP Catalog paimon_catalog_fs; ``` #### View the schema of a Paimon table[​](#view-the-schema-of-a-paimon-table "Direct link to View the schema of a Paimon table") You can use one of the following syntaxes to view the schema of a Paimon table: * View schema ```sql DESC[RIBE] ..; ``` * View schema and location from the CREATE statement ```sql SHOW CREATE TABLE ..; ``` #### Query a Paimon table[​](#query-a-paimon-table "Direct link to Query a Paimon table") 1. Use [SHOW DATABASES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATABASES.md) to view the databases in your Paimon cluster: ```sql SHOW DATABASES FROM ; ``` 2. Use [SET CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SET_CATALOG.md) to switch to the destination catalog in the current session: ```sql SET CATALOG ; ``` Then, use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to specify the active database in the current session: ```sql USE ; ``` Or, you can use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to directly specify the active database in the destination catalog: ```sql USE .; ``` 3. Use [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md) to query the destination table in the specified database: ```sql SELECT count(*) FROM LIMIT 10; ``` #### Load data from Paimon[​](#load-data-from-paimon "Direct link to Load data from Paimon") Suppose you have an OLAP table named `olap_tbl`, you can transform and load data like below: ```sql INSERT INTO default_catalog.olap_db.olap_tbl SELECT * FROM paimon_table; ``` --- ### Query external data This topic guides you through querying data from external data sources by using external catalogs. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") External catalogs are created based on external data sources. For information about supported types of external catalogs, see [Catalog](https://docs.starrocks.io/docs/data_source/catalog/catalog_overview.md#catalog). #### Procedure[​](#procedure "Direct link to Procedure") 1. Connect your StarRocks cluster. * If you use the MySQL client to connect the StarRocks cluster, you go to `default_catalog` by default after connecting. * If you use JDBC to connect the StarRocks cluster, you can go directly to the destination database in the default catalog by specifying `default_catalog.db_name` when connecting. 2. (Optional) Execute the following statement to view all the catalogs and find the external catalog you have created. See [SHOW CATALOGS](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CATALOGS.md) to check the output of this statement. ```sql SHOW CATALOGS; ``` 3. (Optional) Execute the following statement to view all the databases in the external catalog. See [SHOW DATABASES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATABASES.md) to check the output of this statement. ```sql SHOW DATABASES FROM catalog_name; ``` 4. (Optional) Execute the following statement to go to the destination database in the external catalog. ```sql USE catalog_name.db_name; ``` 5. Query external data. For more usages of the SELECT statement, see [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md). ```sql SELECT * FROM table_name; ``` If you do not specify the external catalog and database in the preceding steps, you can directly specify them in a select query. ```sql SELECT * FROM catalog_name.db_name.table_name; ``` #### Examples[​](#examples "Direct link to Examples") If you already created a Hive catalog named `hive1` and want to use `hive1` to query data from `hive_db.hive_table` of an Apache Hive™ cluster, you can perform one of the following operations: ```sql USE hive1.hive_db; SELECT * FROM hive_table limit 1; ``` Or ```sql SELECT * FROM hive1.hive_db.hive_table limit 1; ``` #### References[​](#references "Direct link to References") To query data from your StarRocks cluster, see [Default catalog](https://docs.starrocks.io/docs/data_source/catalog/default_catalog.md). --- ### Unified catalog Beta feature [Advice on use of Beta features](https://docs.starrocks.io/docs/introduction/maturity.md) A unified catalog is a type of external catalog that is provided by StarRocks from v3.2 onwards to handle tables from Apache Hive™, Apache Iceberg, Apache Hudi, Delta Lake, and Apache Kudu data sources as a unified data source without ingestion. With unified catalogs, you can: * Directly query data stored in Hive, Iceberg, Hudi, Delta Lake, Paimon, and Kudu without the need to manually create tables. * Use [INSERT INTO](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) or asynchronous materialized views (which are supported from v2.5 onwards) to process data stored in Hive, Iceberg, Hudi, Delta Lake, Paimon, and Kudu and load the data into StarRocks. * Perform operations on StarRocks to create or drop Hive and Iceberg databases and tables. To ensure successful SQL workloads on your unified data source, your StarRocks cluster must be able to access the storage system and metastore of your unified data source. StarRocks supports the following storage systems and metastores: * Distributed file system (HDFS) or object storage like AWS S3, Microsoft Azure Storage, Google GCS, or other S3-compatible storage system (for example, MinIO) * Metastore like Hive metastore or AWS Glue > **NOTE** > > If you choose AWS S3 as storage, you can use HMS or AWS Glue as metastore. If you choose any other storage system, you can only use HMS as metastore. #### Limits[​](#limits "Direct link to Limits") One unified catalog supports integrations with only a single storage system and a single metastore service. Therefore, make sure all the data sources you want to integrate as a unified data source with StarRocks use the same storage system and metastore service. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * See the "Usage notes" section in [Hive catalog](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md), [Iceberg catalog](https://docs.starrocks.io/docs/data_source/catalog/iceberg/iceberg_catalog.md), [Hudi catalog](https://docs.starrocks.io/docs/data_source/catalog/hudi_catalog.md), [Delta Lake catalog](https://docs.starrocks.io/docs/data_source/catalog/deltalake_catalog.md), [Paimon catalog](https://docs.starrocks.io/docs/data_source/catalog/paimon_catalog.md), and [Kudu catalog](https://docs.starrocks.io/docs/data_source/catalog/kudu_catalog.md) to understand the file formats and data types supported. * Format-specific operations are supported only for specific table formats. For example, [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) and [DROP TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DROP_TABLE.md) are supported only for Hive and Iceberg, and [REFRESH EXTERNAL TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/REFRESH_EXTERNAL_TABLE.md) is supported only for Hive and Hudi. When you create a table within a unified catalog by using the [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) statement, use the `ENGINE` parameter to specify the table format (Hive or Iceberg). #### Integration preparations[​](#integration-preparations "Direct link to Integration preparations") Before you create a unified catalog, make sure your StarRocks cluster can integrate with the storage system and metastore of your unified data source. ##### AWS IAM[​](#aws-iam "Direct link to AWS IAM") If you use AWS S3 as storage or AWS Glue as metastore, choose your suitable authentication method and make the required preparations to ensure that your StarRocks cluster can access the related AWS cloud resources. For more information, see [Authenticate to AWS resources - Preparations](https://docs.starrocks.io/docs/integrations/authenticate_to_aws_resources.md#preparations). ##### HDFS[​](#hdfs "Direct link to HDFS") If you choose HDFS as storage, configure your StarRocks cluster as follows: * (Optional) Set the username that is used to access your HDFS cluster and Hive metastore. By default, StarRocks uses the username of the FE and BE or CN processes to access your HDFS cluster and Hive metastore. You can also set the username by adding `export HADOOP_USER_NAME=""` at the beginning of the **fe/conf/hadoop\_env.sh** file of each FE and at the beginning of the **be/conf/hadoop\_env.sh** file of each BE or the **cn/conf/hadoop\_env.sh** file of each CN. After you set the username in these files, restart each FE and each BE or CN to make the parameter settings take effect. You can set only one username for each StarRocks cluster. * When you query data, the FEs and BEs or CNs of your StarRocks cluster use the HDFS client to access your HDFS cluster. In most cases, you do not need to configure your StarRocks cluster to achieve that purpose, and StarRocks starts the HDFS client using the default configurations. You need to configure your StarRocks cluster only in the following situations: * High availability (HA) is enabled for your HDFS cluster: Add the **hdfs-site.xml** file of your HDFS cluster to the **$FE\_HOME/conf** path of each FE and to the **$BE\_HOME/conf** path of each BE or the **$CN\_HOME/conf** path of each CN. * View File System (ViewFs) is enabled for your HDFS cluster: Add the **core-site.xml** file of your HDFS cluster to the **$FE\_HOME/conf** path of each FE and to the **$BE\_HOME/conf** path of each BE or the **$CN\_HOME/conf** path of each CN. > **NOTE** > > If an error indicating an unknown host is returned when you send a query, you must add the mapping between the host names and IP addresses of your HDFS cluster nodes to the **/etc/hosts** path. ##### Kerberos authentication[​](#kerberos-authentication "Direct link to Kerberos authentication") If Kerberos authentication is enabled for your HDFS cluster or Hive metastore, configure your StarRocks cluster as follows: * Run the `kinit -kt keytab_path principal` command on each FE and each BE or CN to obtain Ticket Granting Ticket (TGT) from Key Distribution Center (KDC). To run this command, you must have the permissions to access your HDFS cluster and Hive metastore. Note that accessing KDC with this command is time-sensitive. Therefore, you need to use cron to run this command periodically. * Add `JAVA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf"` to the **$FE\_HOME/conf/fe.conf** file of each FE and to the **$BE\_HOME/conf/be.conf** file of each BE or the **$CN\_HOME/conf/cn.conf** file of each CN. In this example, `/etc/krb5.conf` is the save path of the krb5.conf file. You can modify the path based on your needs. #### Create a unified catalog[​](#create-a-unified-catalog "Direct link to Create a unified catalog") ##### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE EXTERNAL CATALOG [COMMENT ] PROPERTIES ( "type" = "unified", MetastoreParams, StorageCredentialParams, MetadataUpdateParams, PaimonCatalogParams, KuduCatalogParams ) ``` ##### Parameters[​](#parameters "Direct link to Parameters") ###### catalog\_name[​](#catalog_name "Direct link to catalog_name") The name of the unified catalog. The naming conventions are as follows: * The name can contain letters, digits (0-9), and underscores (\_). It must start with a letter. * The name is case-sensitive and cannot exceed 1023 characters in length. ###### comment[​](#comment "Direct link to comment") The description of the unified catalog. This parameter is optional. ###### type[​](#type "Direct link to type") The type of your data source. Set the value to `unified`. ###### MetastoreParams[​](#metastoreparams "Direct link to MetastoreParams") A set of parameters about how StarRocks integrates with your metastore. ###### Hive metastore[​](#hive-metastore "Direct link to Hive metastore") If you choose Hive metastore as the metastore of your unified data source, configure `MetastoreParams` as follows: ```sql "unified.metastore.type" = "hive", "hive.metastore.uris" = "" ``` > **NOTE** > > Before querying data, you must add the mapping between the host names and IP addresses of your Hive metastore nodes to the **/etc/hosts** path. Otherwise, StarRocks may fail to access your Hive metastore when you start a query. The following table describes the parameters you need to configure in `MetastoreParams`. | Parameter | Required | Description | | ---------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | unified.metastore.type | Yes | The type of metastore that you use for your unified data source. Set the value to `hive`. | | hive.metastore.uris | Yes | The URI of your Hive metastore. Format: `thrift://:`. If high availability (HA) is enabled for your Hive metastore, you can specify multiple metastore URIs and separate them with commas (`,`), for example, `"thrift://:,thrift://:,thrift://:"`. | ###### AWS Glue[​](#aws-glue "Direct link to AWS Glue") If you choose AWS Glue as the metastore of your data source, which is supported only when you choose AWS S3 as storage, take one of the following actions: * To choose the instance profile-based authentication method, configure `MetastoreParams` as follows: ```sql "unified.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.region" = "" ``` * To choose the assumed role-based authentication method, configure `MetastoreParams` as follows: ```sql "unified.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.iam_role_arn" = "", "aws.glue.region" = "" ``` * To choose the IAM user-based authentication method, configure `MetastoreParams` as follows: ```sql "unified.metastore.type" = "glue", "aws.glue.use_instance_profile" = "false", "aws.glue.access_key" = "", "aws.glue.secret_key" = "", "aws.glue.region" = "" ``` The following table describes the parameters you need to configure in `MetastoreParams`. | Parameter | Required | Description | | ------------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | unified.metastore.type | Yes | The type of metastore that you use for your unified data source. Set the value to `glue`. | | aws.glue.use\_instance\_profile | Yes | Specifies whether to enable the instance profile-based authentication method and the assumed role-based authentication. Valid values: `true` and `false`. Default value: `false`. | | aws.glue.iam\_role\_arn | No | The ARN of the IAM role that has privileges on your AWS Glue Data Catalog. If you use the assumed role-based authentication method to access AWS Glue, you must specify this parameter. | | aws.glue.region | Yes | The region in which your AWS Glue Data Catalog resides. Example: `us-west-1`. | | aws.glue.access\_key | No | The access key of your AWS IAM user. If you use the IAM user-based authentication method to access AWS Glue, you must specify this parameter. | | aws.glue.secret\_key | No | The secret key of your AWS IAM user. If you use the IAM user-based authentication method to access AWS Glue, you must specify this parameter. | For information about how to choose an authentication method for accessing AWS Glue and how to configure an access control policy in the AWS IAM Console, see [Authentication parameters for accessing AWS Glue](https://docs.starrocks.io/docs/integrations/authenticate_to_aws_resources.md#authentication-parameters-for-accessing-aws-glue). ###### StorageCredentialParams[​](#storagecredentialparams "Direct link to StorageCredentialParams") A set of parameters about how StarRocks integrates with your storage system. This parameter set is optional. If you use HDFS as storage, you do not need to configure `StorageCredentialParams`. If you use AWS S3, other S3-compatible storage system, Microsoft Azure Storage, or Google GCS as storage, you must configure `StorageCredentialParams`. ###### AWS S3[​](#aws-s3 "Direct link to AWS S3") If you choose AWS S3 as storage, take one of the following actions: * To choose the instance profile-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "" ``` * To choose the assumed role-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "", "aws.s3.region" = "" ``` * To choose the IAM user-based authentication method, configure `StorageCredentialParams` as follows: ```sql "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ----------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | aws.s3.use\_instance\_profile | Yes | Specifies whether to enable the instance profile-based authentication method and the assumed role-based authentication method. Valid values: `true` and `false`. Default value: `false`. | | aws.s3.iam\_role\_arn | No | The ARN of the IAM role that has privileges on your AWS S3 bucket. If you use the assumed role-based authentication method to access AWS S3, you must specify this parameter. | | aws.s3.region | Yes | The region in which your AWS S3 bucket resides. Example: `us-west-1`. | | aws.s3.access\_key | No | The access key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter. | | aws.s3.secret\_key | No | The secret key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter. | For information about how to choose an authentication method for accessing AWS S3 and how to configure an access control policy in AWS IAM Console, see [Authentication parameters for accessing AWS S3](https://docs.starrocks.io/docs/integrations/authenticate_to_aws_resources.md#authentication-parameters-for-accessing-aws-s3). ###### S3-compatible storage system[​](#s3-compatible-storage-system "Direct link to S3-compatible storage system") If you choose an S3-compatible storage system, such as MinIO, as storage, configure `StorageCredentialParams` as follows to ensure a successful integration: ```sql "aws.s3.enable_ssl" = "false", "aws.s3.enable_path_style_access" = "true", "aws.s3.endpoint" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ---------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | aws.s3.enable\_ssl | Yes | Specifies whether to enable SSL connection.
Valid values: `true` and `false`. Default value: `true`. | | aws.s3.enable\_path\_style\_access | Yes | Specifies whether to enable path-style access.
Valid values: `true` and `false`. Default value: `false`. For MinIO, you must set the value to `true`.
Path-style URLs use the following format: `https://s3..amazonaws.com//`. For example, if you create a bucket named `DOC-EXAMPLE-BUCKET1` in the US West (Oregon) Region, and you want to access the `alice.jpg` object in that bucket, you can use the following path-style URL: `https://s3.us-west-2.amazonaws.com/DOC-EXAMPLE-BUCKET1/alice.jpg`. | | aws.s3.endpoint | Yes | The endpoint that is used to connect to your S3-compatible storage system instead of AWS S3. | | aws.s3.access\_key | Yes | The access key of your IAM user. | | aws.s3.secret\_key | Yes | The secret key of your IAM user. | ###### Microsoft Azure Storage[​](#microsoft-azure-storage "Direct link to Microsoft Azure Storage") ###### Azure Blob Storage[​](#azure-blob-storage "Direct link to Azure Blob Storage") If you choose Blob Storage as storage, take one of the following actions: * To choose the Shared Key authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.blob.storage_account" = "", "azure.blob.shared_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | --------------------------- | ------------ | -------------------------------------------- | | azure.blob.storage\_account | Yes | The username of your Blob Storage account. | | azure.blob.shared\_key | Yes | The shared key of your Blob Storage account. | * To choose the SAS Token authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.blob.storage_account" = "", "azure.blob.container" = "", "azure.blob.sas_token" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | --------------------------- | ------------ | --------------------------------------------------------------- | | azure.blob.storage\_account | Yes | The username of your Blob Storage account. | | azure.blob.container | Yes | The name of the blob container that stores your data. | | azure.blob.sas\_token | Yes | The SAS token that is used to access your Blob Storage account. | ###### Azure Data Lake Storage Gen2[​](#azure-data-lake-storage-gen2 "Direct link to Azure Data Lake Storage Gen2") If you choose Data Lake Storage Gen2 as storage, take one of the following actions: * To choose the Managed Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------------------ | ------------ | ------------------------------------------------------------------------------------------------ | | azure.adls2.oauth2\_use\_managed\_identity | Yes | Specifies whether to enable the Managed Identity authentication method. Set the value to `true`. | | azure.adls2.oauth2\_tenant\_id | Yes | The ID of the tenant whose data you want to access. | | azure.adls2.oauth2\_client\_id | Yes | The client (application) ID of the managed identity. | * To choose the Shared Key authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.storage_account" = "", "azure.adls2.shared_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ---------------------------- | ------------ | -------------------------------------------------------------- | | azure.adls2.storage\_account | Yes | The username of your Data Lake Storage Gen2 storage account. | | azure.adls2.shared\_key | Yes | The shared key of your Data Lake Storage Gen2 storage account. | * To choose the Service Principal authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_client_secret" = "", "azure.adls2.oauth2_client_endpoint" = "" ``` The following table describes the parameters you need to configure `in StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------------ | ------------ | -------------------------------------------------------------------------- | | azure.adls2.oauth2\_client\_id | Yes | The client (application) ID of the service principal. | | azure.adls2.oauth2\_client\_secret | Yes | The value of the new client (application) secret created. | | azure.adls2.oauth2\_client\_endpoint | Yes | The OAuth 2.0 token endpoint (v1) of the service principal or application. | * To choose the Workload Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls2.oauth2_token_file" = "", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | azure.adls2.oauth2\_token\_file | Yes | The absolute file path to the OAuth2 token file projected into the pod by the Azure Workload Identity webhook. | | azure.adls2.oauth2\_tenant\_id | Yes | The ID of the tenant whose data you want to access. | | azure.adls2.oauth2\_client\_id | Yes | The client ID (application ID) of the Azure AD application (user-assigned managed identity or app registration) associated with the workload identity. | ###### Azure Data Lake Storage Gen1[​](#azure-data-lake-storage-gen1 "Direct link to Azure Data Lake Storage Gen1") If you choose Data Lake Storage Gen1 as storage, take one of the following actions: * To choose the Managed Service Identity authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls1.use_managed_service_identity" = "true" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------------------- | ------------ | -------------------------------------------------------------------------------------------------------- | | azure.adls1.use\_managed\_service\_identity | Yes | Specifies whether to enable the Managed Service Identity authentication method. Set the value to `true`. | * To choose the Service Principal authentication method, configure `StorageCredentialParams` as follows: ```sql "azure.adls1.oauth2_client_id" = "", "azure.adls1.oauth2_credential" = "", "azure.adls1.oauth2_endpoint" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Required** | **Description** | | ------------------------------ | ------------ | -------------------------------------------------------------------------- | | azure.adls1.oauth2\_client\_id | Yes | The client (application) ID of the service principal. | | azure.adls1.oauth2\_credential | Yes | The value of the new client (application) secret created. | | azure.adls1.oauth2\_endpoint | Yes | The OAuth 2.0 token endpoint (v1) of the service principal or application. | ###### Google GCS[​](#google-gcs "Direct link to Google GCS") If you choose Google GCS as storage, take one of the following actions: * To choose the VM-based authentication method, configure `StorageCredentialParams` as follows: ```sql "gcp.gcs.use_compute_engine_service_account" = "true" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ---------------------------------------------- | ----------------- | --------------------- | ------------------------------------------------------------------------------------------- | | gcp.gcs.use\_compute\_engine\_service\_account | false | true | Specifies whether to directly use the service account that is bound to your Compute Engine. | * To choose the service account-based authentication method, configure `StorageCredentialParams` as follows: ```sql "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ------------------------------------------ | ----------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------- | | gcp.gcs.service\_account\_email | "" | "" | The email address in the JSON file generated at the creation of the service account. | | gcp.gcs.service\_account\_private\_key\_id | "" | "61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea" | The private key ID in the JSON file generated at the creation of the service account. | | gcp.gcs.service\_account\_private\_key | "" | "-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n" | The private key in the JSON file generated at the creation of the service account. | * To choose the impersonation-based authentication method, configure `StorageCredentialParams` as follows: * Make a VM instance impersonate a service account: ```sql "gcp.gcs.use_compute_engine_service_account" = "true", "gcp.gcs.impersonation_service_account" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ---------------------------------------------- | ----------------- | --------------------- | ------------------------------------------------------------------------------------------- | | gcp.gcs.use\_compute\_engine\_service\_account | false | true | Specifies whether to directly use the service account that is bound to your Compute Engine. | | gcp.gcs.impersonation\_service\_account | "" | "hello" | The service account that you want to impersonate. | * Make a service account (temporarily named as meta service account) impersonate another service account (temporarily named as data service account): ```sql "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "", "gcp.gcs.impersonation_service_account" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ------------------------------------------ | ----------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------ | | gcp.gcs.service\_account\_email | "" | "" | The email address in the JSON file generated at the creation of the meta service account. | | gcp.gcs.service\_account\_private\_key\_id | "" | "61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea" | The private key ID in the JSON file generated at the creation of the meta service account. | | gcp.gcs.service\_account\_private\_key | "" | "-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n" | The private key in the JSON file generated at the creation of the meta service account. | | gcp.gcs.impersonation\_service\_account | "" | "hello" | The data service account that you want to impersonate. | ###### MetadataUpdateParams[​](#metadataupdateparams "Direct link to MetadataUpdateParams") A set of parameters about how StarRocks updates the cached metadata of Hive, Hudi, and Delta Lake. This parameter set is optional. For more information about the policies for updating cached metadata from Hive, Hudi, and Delta Lake, see [Hive catalog](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md), [Hudi catalog](https://docs.starrocks.io/docs/data_source/catalog/hudi_catalog.md), and [Delta Lake catalog](https://docs.starrocks.io/docs/data_source/catalog/deltalake_catalog.md). In most cases, you can ignore `MetadataUpdateParams` and do not need to tune the policy parameters in it, because the default values of these parameters already provide you with an out-of-the-box performance. However, if the frequency of data updates in Hive, Hudi, or Delta Lake is high, you can tune these parameters to further optimize the performance of automatic asynchronous updates. | Parameter | Required | Description | | ------------------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | enable\_metastore\_cache | No | Specifies whether StarRocks caches the metadata of Hive, Hudi, or Delta Lake tables. Valid values: `true` and `false`. Default value: `true`. The value `true` enables the cache, and the value `false` disables the cache. | | enable\_remote\_file\_cache | No | Specifies whether StarRocks caches the metadata of the underlying data files of Hive, Hudi, or Delta Lake tables or partitions. Valid values: `true` and `false`. Default value: `true`. The value true enables the cache, and the value `false` disables the cache. | | metastore\_cache\_refresh\_interval\_sec | No | The time interval at which StarRocks asynchronously updates the metadata of Hive, Hudi, or Delta Lake tables or partitions cached in itself. Unit: seconds. Default value: `7200`, which is 2 hours. | | remote\_file\_cache\_refresh\_interval\_sec | No | The time interval at which StarRocks asynchronously updates the metadata of the underlying data files of Hive, Hudi, or Delta Lake tables or partitions cached in itself. Unit: seconds. Default value: `60`. | | metastore\_cache\_ttl\_sec | No | The time interval at which StarRocks automatically discards the metadata of Hive, Hudi, or Delta Lake tables or partitions cached in itself. Unit: seconds. Default value: `86400`, which is 24 hours. | | remote\_file\_cache\_ttl\_sec | No | The time interval at which StarRocks automatically discards the metadata of the underlying data files of Hive, Hudi, or Delta Lake tables or partitions cached in itself. Unit: seconds. Default value: `129600`, which is 36 hours. | ###### PaimonCatalogParams[​](#paimoncatalogparams "Direct link to PaimonCatalogParams") A set of parameters about how to connect Paimon Catalog. This parameter set is optional. | Parameter | Required | Description | | ------------------------ | -------- | ----------------------------------------------- | | paimon.catalog.warehouse | No | The warehouse storage path of your Paimon data. | ###### KuduCatalogParams[​](#kuducatalogparams "Direct link to KuduCatalogParams") A set of parameters about how to connect Kudu Catalog. This parameter set is optional. | Parameter | Required | Description | | ----------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | kudu.master | No | Specifies the Kudu Master address, which defaults to `localhost:7051`. | | kudu.schema-emulation.enabled | No | option to enable or disable the `schema` emulation. By default, it is turned off (false), which means that all tables belong to the `default` `schema`. | | kudu.schema-emulation.prefix | No | The prefix for `schema` emulation should only be set when `kudu.schema-emulation.enabled` = `true`. The default prefix used is empty string: \`\`. | ##### Examples[​](#examples "Direct link to Examples") The following examples create a unified catalog named `unified_catalog_hms` or `unified_catalog_glue`, depending on the type of metastore you use, to query data from your unified data source. ###### HDFS[​](#hdfs-1 "Direct link to HDFS") If you use HDFS as storage, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083" ); ``` ###### AWS S3[​](#aws-s3-1 "Direct link to AWS S3") ###### Instance profile-based authentication[​](#instance-profile-based-authentication "Direct link to Instance profile-based authentication") * If you use Hive metastore, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "us-west-2" ); ``` * If you use AWS Glue with Amazon EMR, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_glue PROPERTIES ( "type" = "unified", "unified.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.region" = "us-west-2", "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "us-west-2" ); ``` ###### Assumed role-based authentication[​](#assumed-role-based-authentication "Direct link to Assumed role-based authentication") * If you use Hive metastore, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::081976408565:role/test_s3_role", "aws.s3.region" = "us-west-2" ); ``` * If you use AWS Glue with Amazon EMR, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_glue PROPERTIES ( "type" = "unified", "unified.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.iam_role_arn" = "arn:aws:iam::081976408565:role/test_glue_role", "aws.glue.region" = "us-west-2", "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::081976408565:role/test_s3_role", "aws.s3.region" = "us-west-2" ); ``` ###### IAM user-based authentication[​](#iam-user-based-authentication "Direct link to IAM user-based authentication") * If you use Hive metastore, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "us-west-2" ); ``` * If you use AWS Glue with Amazon EMR, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_glue PROPERTIES ( "type" = "unified", "unified.metastore.type" = "glue", "aws.glue.use_instance_profile" = "false", "aws.glue.access_key" = "", "aws.glue.secret_key" = "", "aws.glue.region" = "us-west-2", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "us-west-2" ); ``` ###### S3-compatible storage system[​](#s3-compatible-storage-system-1 "Direct link to S3-compatible storage system") Use MinIO as an example. Run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "aws.s3.enable_ssl" = "true", "aws.s3.enable_path_style_access" = "true", "aws.s3.endpoint" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ); ``` ###### Microsoft Azure Storage[​](#microsoft-azure-storage-1 "Direct link to Microsoft Azure Storage") ###### Azure Blob Storage[​](#azure-blob-storage-1 "Direct link to Azure Blob Storage") * If you choose the Shared Key authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.blob.storage_account" = "", "azure.blob.shared_key" = "" ); ``` * If you choose the SAS Token authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.blob.storage_account" = "", "azure.blob.container" = "", "azure.blob.sas_token" = "" ); ``` ###### Azure Data Lake Storage Gen1[​](#azure-data-lake-storage-gen1-1 "Direct link to Azure Data Lake Storage Gen1") * If you choose the Managed Service Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls1.use_managed_service_identity" = "true" ); ``` * If you choose the Service Principal authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls1.oauth2_client_id" = "", "azure.adls1.oauth2_credential" = "", "azure.adls1.oauth2_endpoint" = "" ); ``` ###### Azure Data Lake Storage Gen2[​](#azure-data-lake-storage-gen2-1 "Direct link to Azure Data Lake Storage Gen2") * If you choose the Managed Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` * If you choose the Shared Key authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.storage_account" = "", "azure.adls2.shared_key" = "" ); ``` * If you choose the Service Principal authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_client_secret" = "", "azure.adls2.oauth2_client_endpoint" = "" ); ``` * If you choose the Workload Identity authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_token_file" = "/var/run/secrets/azure/tokens/azure-identity-token", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` ###### Google GCS[​](#google-gcs-1 "Direct link to Google GCS") * If you choose the VM-based authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.use_compute_engine_service_account" = "true" ); ``` * If you choose the service account-based authentication method, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "" ); ``` * If you choose the impersonation-based authentication method: * If you make a VM instance impersonate a service account, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.use_compute_engine_service_account" = "true", "gcp.gcs.impersonation_service_account" = "" ); ``` * If you make a service account impersonate another service account, run a command like below: ```sql CREATE EXTERNAL CATALOG unified_catalog_hms PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "", "gcp.gcs.impersonation_service_account" = "" ); ``` #### View unified catalogs[​](#view-unified-catalogs "Direct link to View unified catalogs") You can use [SHOW CATALOGS](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CATALOGS.md) to query all catalogs in the current StarRocks cluster: ```sql SHOW CATALOGS; ``` You can also use [SHOW CREATE CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CREATE_CATALOG.md) to query the creation statement of an external catalog. The following example queries the creation statement of a unified catalog named `unified_catalog_glue`: ```sql SHOW CREATE CATALOG unified_catalog_glue; ``` #### Switch to a Unified Catalog and a database in it[​](#switch-to-a-unified-catalog-and-a-database-in-it "Direct link to Switch to a Unified Catalog and a database in it") You can use one of the following methods to switch to a unified catalog and a database in it: * Use [SET CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SET_CATALOG.md) to specify a unified catalog in the current session, and then use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to specify an active database: ```sql -- Switch to a specified catalog in the current session: SET CATALOG -- Specify the active database in the current session: USE ``` * Directly use [USE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/USE.md) to switch to a unified catalog and a database in it: ```sql USE . ``` #### Drop a unified catalog[​](#drop-a-unified-catalog "Direct link to Drop a unified catalog") You can use [DROP CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/DROP_CATALOG.md) to drop an external catalog. The following example drops a unified catalog named `unified_catalog_glue`: ```sql DROP CATALOG unified_catalog_glue; ``` #### View the schema of a table from a unified catalog[​](#view-the-schema-of-a-table-from-a-unified-catalog "Direct link to View the schema of a table from a unified catalog") You can use one of the following syntaxes to view the schema of a table from a unified catalog: * View schema ```sql DESC[RIBE] .. ``` * View schema and location from the CREATE statement ```sql SHOW CREATE TABLE .. ``` #### Query data from a unified catalog[​](#query-data-from-a-unified-catalog "Direct link to Query data from a unified catalog") To query data from a unified catalog, follow these steps: 1. Use [SHOW DATABASES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATABASES.md) to view the databases in your unified data source with which the unified catalog is associated: ```sql SHOW DATABASES FROM ``` 2. [Switch to a Hive Catalog and a database in it](#switch-to-a-unified-catalog-and-a-database-in-it). 3. Use [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md) to query the destination table in the specified database: ```sql SELECT count(*) FROM LIMIT 10 ``` #### Load data from Hive, Iceberg, Hudi, Delta Lake, or Kudu[​](#load-data-from-hive-iceberg-hudi-delta-lake-or-kudu "Direct link to Load data from Hive, Iceberg, Hudi, Delta Lake, or Kudu") You can use [INSERT INTO](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) to load the data of a Hive, Iceberg, Hudi, Delta Lake, or Kudu table into a StarRocks table created within a unified catalog. The following example loads the data of the Hive table `hive_table` into the StarRocks table `test_tbl` created in the database `test_database` that belongs to the unified catalog `unified_catalog`: ```sql INSERT INTO unified_catalog.test_database.test_table SELECT * FROM hive_table ``` #### Create a database in a unified catalog[​](#create-a-database-in-a-unified-catalog "Direct link to Create a database in a unified catalog") Similar to the internal catalog of StarRocks, if you have the CREATE DATABASE privilege on a unified catalog, you can use the CREATE DATABASE statement to create a database in that catalog. > **NOTE** > > You can grant and revoke privileges by using [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) and [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE.md). StarRocks supports creating only Hive and Iceberg databases in unified catalogs. [Switch to a unified catalog](#switch-to-a-unified-catalog-and-a-database-in-it), and then use the following statement to create a database in that catalog: ```sql CREATE DATABASE [properties ("location" = ":///")] ``` The `location` parameter specifies the file path in which you want to create the database, which can be in either HDFS or cloud storage. * When you use Hive metastore as the metastore of your data source, the `location` parameter defaults to `/`, which is supported by Hive metastore if you do not specify that parameter at database creation. * When you use AWS Glue as the metastore of your data source, the `location` parameter does not have a default value, and therefore you must specify that parameter at database creation. The `prefix` varies based on the storage system you use: | **Storage system** | **`Prefix`** **value** | | ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | HDFS | `hdfs` | | Google GCS | `gs` | | Azure Blob Storage | - If your storage account allows access over HTTP, the `prefix` is `wasb`.
- If your storage account allows access over HTTPS, the `prefix` is `wasbs`. | | Azure Data Lake Storage Gen1 | `adl` | | Azure Data Lake Storage Gen2 | - If your storage account allows access over HTTP, the`prefix` is `abfs`.
- If your storage account allows access over HTTPS, the `prefix` is `abfss`. | | AWS S3 or other S3-compatible storage (for example, MinIO) | `s3` | #### Drop a database from a unified catalog[​](#drop-a-database-from-a-unified-catalog "Direct link to Drop a database from a unified catalog") Similar to the internal databases of StarRocks, if you have the [DROP](https://docs.starrocks.io/docs/administration/user_privs/authorization/user_privs.md#database) privilege on a database created within a unified catalog, you can use the [DROP DATABASE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/DROP_DATABASE.md) statement to drop that database. You can only drop empty databases. > **NOTE** > > You can grant and revoke privileges by using [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) and [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE.md). StarRocks supports dropping only Hive and Iceberg databases from unified catalogs. When you drop a database from a unified catalog, the database's file path on your HDFS cluster or cloud storage will not be dropped along with the database. [Switch to a unified catalog](#switch-to-a-unified-catalog-and-a-database-in-it), and then use the following statement to drop a database in that catalog: ```sql DROP DATABASE ``` #### Create a table in a unified catalog[​](#create-a-table-in-a-unified-catalog "Direct link to Create a table in a unified catalog") Similar to the internal databases of StarRocks, if you have the [CREATE TABLE](https://docs.starrocks.io/docs/administration/user_privs/authorization/user_privs.md#database) privilege on a database created within a unified catalog, you can use the [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) or \[CREATE TABLE AS SELECT ../../sql-reference/sql-statements/table\_bucket\_part\_index/CREATE\_TABLE\_AS\_SELECT.mdELECT.md) statement to create a table in that database. > **NOTE** > > You can grant and revoke privileges by using [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) and [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE.md). StarRocks supports creating only Hive and Iceberg tables in unified catalogs. [Switch to a Hive Catalog and a database in it](#switch-to-a-unified-catalog-and-a-database-in-it). Then, use [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) to create a Hive or Iceberg table in that database: ```sql CREATE TABLE (column_definition1[, column_definition2, ...] ENGINE = {|hive|iceberg} [partition_desc] ``` For more information, see [Create a Hive table](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md#create-a-hive-table) and [Create an Iceberg table](https://docs.starrocks.io/docs/data_source/catalog/iceberg/iceberg_catalog.md#create-an-iceberg-table). The following example creates a Hive table named `hive_table`. The table consists of three columns `action`, `id`, and `dt`, of which `id` and `dt`are partition columns. ```sql CREATE TABLE hive_table ( action varchar(65533), id int, dt date ) ENGINE = hive PARTITION BY (id,dt); ``` #### Sink data to a table in a unified catalog[​](#sink-data-to-a-table-in-a-unified-catalog "Direct link to Sink data to a table in a unified catalog") Similar to the internal tables of StarRocks, if you have the [INSERT](https://docs.starrocks.io/docs/administration/user_privs/authorization/user_privs.md#table) privilege on a table created within a unified catalog, you can use the [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) statement to sink the data of a StarRocks table to that Unified Catalog table (currently only Parquet-formatted Unified Catalog tables are supported). > **NOTE** > > You can grant and revoke privileges by using [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) and [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE.md). StarRocks supports sinking data only to Hive and Iceberg tables in unified catalogs. [Switch to a Hive Catalog and a database in it](#switch-to-a-unified-catalog-and-a-database-in-it). Then, use [INSERT INTO](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) to insert data into a Hive or Iceberg table in that database: ```sql INSERT {INTO | OVERWRITE} [ (column_name [, ...]) ] { VALUES ( { expression | DEFAULT } [, ...] ) [, ...] | query } -- If you want to sink data to specified partitions, use the following syntax: INSERT {INTO | OVERWRITE} PARTITION (par_col1= [, par_col2=...]) { VALUES ( { expression | DEFAULT } [, ...] ) [, ...] | query } ``` For more information, see [Sink data to a Hive table ](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md#sink-data-to-a-hive-table)and [Sink data to an Iceberg table](https://docs.starrocks.io/docs/data_source/catalog/iceberg/iceberg_catalog.md#sink-data-to-an-iceberg-table). The following example inserts three data rows to a Hive table named `hive_table`: ```sql INSERT INTO hive_table VALUES ("buy", 1, "2023-09-01"), ("sell", 2, "2023-09-02"), ("buy", 3, "2023-09-03"); ``` #### Drop a table from a unified catalog[​](#drop-a-table-from-a-unified-catalog "Direct link to Drop a table from a unified catalog") Similar to the internal tables of StarRocks, if you have the [DROP](https://docs.starrocks.io/docs/administration/user_privs/authorization/user_privs.md#table) privilege on a table created within a unified catalog, you can use the [DROP TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DROP_TABLE.md) statement to drop that table. > **NOTE** > > You can grant and revoke privileges by using [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) and [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE.md). StarRocks supports dropping only Hive and Iceberg tables from unified catalogs. [Switch to a Hive Catalog and a database in it](#switch-to-a-unified-catalog-and-a-database-in-it). Then, use [DROP TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DROP_TABLE.md) to drop a Hive or Iceberg table in that database: ```sql DROP TABLE ``` For more information, see [Drop a Hive table](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md#drop-a-hive-table) and [Drop an Iceberg table](https://docs.starrocks.io/docs/data_source/catalog/iceberg/iceberg_catalog.md#drop-an-iceberg-table). The following example drops a Hive table named `hive_table`: ```sql DROP TABLE hive_table FORCE ``` --- ### Data Cache Understand the core principles of Data Cache and how to accelerate data queries with Data Cache. Data Cache is used to cache the data of native tables and external tables. This feature is enabled by default from v3.3.0 onwards. And from v4.0 onwards, the in-memory cache and disk cache have been uniformly integrated into the Data Cache system to facilitate management. Data Cache consists of two components: Page Cache (in-memory cache) and Block Cache (disk cache). #### Principles of Page Cache[​](#principles-of-page-cache "Direct link to Principles of Page Cache") As an in-memory cache, Page Cache is responsible for storing data pages of native and external tables after decompression. The size of these pages is not fixed. Currently, Page Cache supports caching the following types of data: * Data pages and index pages of native tables * Footer information of external table data files * Partial decompressed data pages of external tables Page Cache currently uses the LRU (Least Recently Used) strategy for data eviction. #### Principles of Block Cache[​](#principles-of-block-cache "Direct link to Principles of Block Cache") Block Cache is a disk-based cache whose primary function is to cache data files (from external tables, and also from cloud-native tables in shared-data cluster) to local disks. This reduces remote data access latency and improves query efficiency. The size of each data block is fixed. ##### Background and values[​](#background-and-values "Direct link to Background and values") In data lake analytics and cloud-native table scenarios, StarRocks, acting as an OLAP query engine, needs to scan data files stored in HDFS or object storage (hereinafter referred to as the "remote storage system"). This process faces two major performance bottlenecks: * The more files a query needs to read, the greater the remote I/O overhead. * In ad-hoc query scenarios, frequent access to the same data leads to redundant remote I/O consumption. To address these issues, the Block Cache feature has been introduced in v2.5. It splits raw data from the remote storage system into multiple blocks based on a specific strategy and caches these blocks in the local disks of BE or CN nodes. By avoiding repeated retrieval of remote data, the query performance against hot data is significantly improved. ##### Scenarios[​](#scenarios "Direct link to Scenarios") * Query against data from remote storage systems using external catalogs (except JDBC Catalogs). * Query against cloud-native tables in the shared-data clusters. ##### Core mechanism[​](#core-mechanism "Direct link to Core mechanism") ###### Data splitting and cache unit[​](#data-splitting-and-cache-unit "Direct link to Data splitting and cache unit") When the system caches remote files, it splits the original files into blocks of equal size according to the configured strategy. A block is the minimum cache unit, and its size is customizable. Example: If the block size is configured as 1 MB, when querying a 128 MB Parquet file on Amazon S3, the file will be split into 128 consecutive blocks (that is, `[0, 1 MB)`, `[1 MB, 2 MB)`, ..., `[127 MB, 128 MB)`). Each block is assigned a globally unique cache identifier (cache key), which consists of the following three parts: ```plain hash(filename) + fileModificationTime + blockId ``` | **Component** | **Description** | | -------------------- | ------------------------------------------------------------------------------------------------------------------------- | | filename | The name of the data file. | | fileModificationTime | The last time when the data file was modified. | | blockId | The ID assigned to each block when a data file was split. This ID is unique within a single file but not globally unique. | ###### Cache Hit and Read Process[​](#cache-hit-and-read-process "Direct link to Cache Hit and Read Process") Assuming a query hits the block within the range `[1 MB, 2 MB)`, Block Cache proceeds as follows: 1. The system first checks whether the block exists in the local BE node's Block Cache (by matching the cache key). 2. If found (cache hit), the block is read directly from the local disk. 3. If not found (cache miss), the block is fetched from remote storage and synchronized to the local BE node's Block Cache for reuse in subsequent queries. ##### Cache medium[​](#cache-medium "Direct link to Cache medium") Block Cache uses the local disks of BE or CN nodes as its storage medium, and the cache acceleration effect is directly related to disk performance: * It is recommended to use high-performance local disks (for example, NVMe disks) to minimize cache read/write latency. * If disk performance is not optimal, you can increase the number of disks to achieve load balancing and reduce I/O pressure on individual disks. ##### Cache replacement policies[​](#cache-replacement-policies "Direct link to Cache replacement policies") Block Cache supports two data caching and eviction strategies: LRU and SLRU (Segmented LRU). ###### LRU[​](#lru "Direct link to LRU") The LRU strategy is based on the "Least Recently Used" principle - evicting the blocks that have not been accessed for the longest time. It is simple to implement, and suitable for scenarios with stable access patterns. ###### SLRU[​](#slru "Direct link to SLRU") The SLRU strategy divides the cache space into an eviction segment and a protection segment, both following the LRU rules: 1. Data enters the eviction segment on first access. 2. Data in the eviction segment is promoted to the protection segment when accessed again. 3. Data evicted from the protection segment falls back to the eviction segment, while data evicted from the eviction segment is directly removed from the cache. The SLRU strategy can effectively resist sudden sparse traffic, preventing "temporary data accessed only once" from directly evicting hot data in the protection segment. It offers better stability than LRU. #### Enable and configure Data Cache[​](#enable-and-configure-data-cache "Direct link to Enable and configure Data Cache") Data Cache is enabled by default, controlled by the BE configuration item `datacache_enable` (Default: `true`). Page Cache and Block Cache, as two independent components, are also enabled by default. Setting `datacache_enable` to `false` will disable Data Cache overall, that is, both Page Cache and Block Cache. You can also activate or deactivate Page Cache and Block Cache separately using different BE configuration items. * Page Cache (enabled by default) is controlled by `disable_storage_page_cache` (Default: `false`). * Block Cache (enabled by default) is controlled by `block_cache_enable` (Default: `true`). You can further use the following BE configurations to set the maximum memory and disk usage limits for Data Cache, preventing excessive resource occupation: * `datacache_mem_size`: Sets the maximum memory usage limit for Data Cache (used for storing data in Page Cache). * `datacache_disk_size`: Sets the maximum disk usage limit for Data Cache (used for storing data in Block Cache). #### Populate Block Cache[​](#populate-block-cache "Direct link to Populate Block Cache") ##### Population rules[​](#population-rules "Direct link to Population rules") From v3.3.2 onwards, in order to improve the cache hit rate of Block Cache, the system populates Block Cache according to the following rules: * The cache will not be populated for statements that are not `SELECT`, for example, `ANALYZE TABLE` and `INSERT INTO SELECT`. * Queries that scan all partitions of a table will not populate the cache. However, if the table has only one partition, population is performed by default. * Queries that scan all columns of a table will not populate the cache. However, if the table has only one column, population is performed by default. * The cache will not be populated for tables that are not Hive, Paimon, Delta Lake, Hudi, or Iceberg. ##### View Cache Population Behavior[​](#view-cache-population-behavior "Direct link to View Cache Population Behavior") You can view the population behavior for a specific query with the `EXPLAIN VERBOSE` command. Example: ```sql mysql> EXPLAIN VERBOSE SELECT col1 FROM hudi_table; ... | 0:HudiScanNode | | TABLE: hudi_table | | partitions=3/3 | | cardinality=9084 | | avgRowSize=2.0 | | dataCacheOptions={populate: false} | | cardinality: 9084 | +-----------------------------------------+ ``` `dataCacheOptions={populate: false}` indicates that the cache will not be populated because the query will scan all partitions. You can also fine-tune the population behavior of Block Cache via the session variable [`populate_datacache_mode`](https://docs.starrocks.io/docs/sql-reference/System_variable.md#populate_datacache_mode). ##### Population mode[​](#population-mode "Direct link to Population mode") Block Cache supports two modes, that is, synchronous population and asynchronous population. You can choose between them based on your business requirements for "first query performance" and "cache efficiency". **Synchronous Population** * Core Logic: When remote data is read for the first query, the data is immediately cached locally. Subsequent queries can directly reuse the cache. * Pros: High cache efficiency because data caching is completed with a single query. * Cons: Cache operations are executed synchronously with read operations, which may increase latency for the first query. **Asynchronous Population** * Core Logic: For the first query, data reading is prioritized and completed first. Cache writing is executed asynchronously in the background, without blocking the current query process. * Pros: It will not affect the performance of first query and prevent read operations from being delayed due to caching. * Cons: Lower cache efficiency because a single query may not fully cache all accessed data. It requires multiple queries to gradually improve cache coverage. From v3.3.0, asynchronous cache population is enabled by default. You can change the population mode by setting the session variable [`enable_datacache_async_populate_mode`](https://docs.starrocks.io/docs/sql-reference/System_variable.md). ##### Persistence[​](#persistence "Direct link to Persistence") The cached data in disks can be persistent by default, and these data can be reused after BE or CN restarts. #### Check whether a query hits data cache[​](#check-whether-a-query-hits-data-cache "Direct link to Check whether a query hits data cache") You can check whether a query hits Data Cache by analyzing the following metrics in the query profile: * `DataCacheReadBytes`: the size of data that the system reads directly from the memory and disks. * `DataCacheWriteBytes`: the size of data loaded from the remote storage system to the memory and disks. * `BytesRead`: the total amount of data that is read, including data that the system reads from the remote storage system, and its memory and disks. Example 1: In this example, the system reads a large amount of data (7.65 GB) from the remote storage system and only a few data (518.73 MB) from the disks. This means that few Block Caches were hit. ```plain - Table: lineorder - DataCacheReadBytes: 518.73 MB - __MAX_OF_DataCacheReadBytes: 4.73 MB - __MIN_OF_DataCacheReadBytes: 16.00 KB - DataCacheReadCounter: 684 - __MAX_OF_DataCacheReadCounter: 4 - __MIN_OF_DataCacheReadCounter: 0 - DataCacheReadTimer: 737.357us - DataCacheWriteBytes: 7.65 GB - __MAX_OF_DataCacheWriteBytes: 64.39 MB - __MIN_OF_DataCacheWriteBytes: 0.00 - DataCacheWriteCounter: 7.887K (7887) - __MAX_OF_DataCacheWriteCounter: 65 - __MIN_OF_DataCacheWriteCounter: 0 - DataCacheWriteTimer: 23.467ms - __MAX_OF_DataCacheWriteTimer: 62.280ms - __MIN_OF_DataCacheWriteTimer: 0ns - BufferUnplugCount: 15 - __MAX_OF_BufferUnplugCount: 2 - __MIN_OF_BufferUnplugCount: 0 - BytesRead: 7.65 GB - __MAX_OF_BytesRead: 64.39 MB - __MIN_OF_BytesRead: 0.00 ``` Example 2: In this example, the system reads a large amount of data (46.08 GB) from data cache and no data from the remote storage system, which means it reads data only from Block Cache. ```plain Table: lineitem - DataCacheReadBytes: 46.08 GB - __MAX_OF_DataCacheReadBytes: 194.99 MB - __MIN_OF_DataCacheReadBytes: 81.25 MB - DataCacheReadCounter: 72.237K (72237) - __MAX_OF_DataCacheReadCounter: 299 - __MIN_OF_DataCacheReadCounter: 118 - DataCacheReadTimer: 856.481ms - __MAX_OF_DataCacheReadTimer: 1s547ms - __MIN_OF_DataCacheReadTimer: 261.824ms - DataCacheWriteBytes: 0.00 - DataCacheWriteCounter: 0 - DataCacheWriteTimer: 0ns - BufferUnplugCount: 1.231K (1231) - __MAX_OF_BufferUnplugCount: 81 - __MIN_OF_BufferUnplugCount: 35 - BytesRead: 46.08 GB - __MAX_OF_BytesRead: 194.99 MB - __MIN_OF_BytesRead: 81.25 MB ``` #### I/O Adaptor[​](#io-adaptor "Direct link to I/O Adaptor") To prevent significant tail latency in disk access due to high cache disk I/O load, which can lead to negative optimization of the cache system, Data Cache provides the I/O Adaptor feature. This feature routes some cache requests to remote storage when disk load is high, utilizing both local cache and remote storage to improve I/O throughput. This feature is enabled by default. You can enable I/O Adaptor by setting the following system variable: ```sql SET GLOBAL enable_datacache_io_adaptor=true; ``` #### Dynamic scaling[​](#dynamic-scaling "Direct link to Dynamic scaling") Data Cache supports manual adjustment of cache capacity without restarting the BE process, and also supports automatic adjustment of cache capacity. ##### Manual scaling[​](#manual-scaling "Direct link to Manual scaling") You can modify Data Cache's memory limit or disk capacity by dynamically adjusting BE configuration items. Examples: ```sql -- Adjust the Data Cache memory limit for a specific BE instance. UPDATE be_configs SET VALUE="10G" WHERE NAME="datacache_mem_size" and BE_ID=10005; -- Adjust the Data Cache memory ratio limit for all BE instances. UPDATE be_configs SET VALUE="10%" WHERE NAME="datacache_mem_size"; -- Adjust the Data Cache disk limit for all BE instances. UPDATE be_configs SET VALUE="2T" WHERE NAME="datacache_disk_size"; ``` note * Be cautious when adjusting capacities in this way. Make sure not to omit the WHERE clause to avoid modifying irrelevant configuration items. * Cache capacity adjustments made this way will not be persisted and will be lost after the BE or CN process restarts. Therefore, you can first adjust the parameters dynamically as described above, and then manually modify the BE or CN configuration file to ensure that the changes take effect after the next restart. ##### Automatic Scaling[​](#automatic-scaling "Direct link to Automatic Scaling") StarRocks currently supports automatic scaling of disk capacity. If you do not specify the cache disk path and capacity limit in the BE configuration, automatic scaling is enabled by default. You can also enable automatic scaling by adding the following configuration item to the configuration file and restarting the BE or CN process: ```plain datacache_auto_adjust_enable=true ``` After automatic scaling is enabled: * When the disk usage exceeds the threshold specified by the BE configuration `disk_high_level` (default value is `90`, that is, 90% of disk space), the system will automatically evict cache data to free up disk space. * When the disk usage is consistently below the threshold specified by the BE configuration `disk_low_level` (default value is `60`, that is, 60% of disk space), and the current disk space used by Data Cache is full, the system will automatically expand the cache capacity. * When automatically scaling the cache capacity, the system will aim to adjust the cache capacity to the level specified by the BE configuration `disk_safe_level` (default value is `80`, that is, 80% of disk space). #### Cache Sharing[​](#cache-sharing "Direct link to Cache Sharing") Because Data Cache depends on the BE node's local disk, when the cluster are being scaled, changes in data routing can cause cache misses, which can easily lead to significant performance degradation during the elastic scaling. Cache Sharing is used to support accessing cache data between nodes through network. During cluster scaling, if a local cache miss occurs, the system first attempts to fetch cache data from other nodes within the same cluster. Only if all caches miss will the system re-fetch data from the remote storage. This feature effectively reduces the performance jitter caused by cache invalidation during elastic scaling and ensures stable query performance. ![cache sharing workflow](/assets/images/cache_sharing_workflow-4bc1221ee31928095ec36486bf6b03e1.png) You can enable the Cache Sharing feature by configuring the following two items: * Set the FE configuration item `enable_trace_historical_node` to `true`. * Set the system variable `enable_datacache_sharing` to `true`. In addition, you can check the following metrics in query profile to monitor Cache Sharing: * `DataCacheReadPeerCounter`: The read count from other nodes. * `DataCacheReadPeerBytes`: The bytes read from other nodes. * `DataCacheReadPeerTimer`: The time used for accessing cache data from other nodes. #### Configurations and variables[​](#configurations-and-variables "Direct link to Configurations and variables") You can configure Data Cache using the following system variables and parameters. ##### System variables[​](#system-variables "Direct link to System variables") * [populate\_datacache\_mode](https://docs.starrocks.io/docs/sql-reference/System_variable.md#populate_datacache_mode) * [enable\_datacache\_io\_adaptor](https://docs.starrocks.io/docs/sql-reference/System_variable.md#enable_datacache_io_adaptor) * [enable\_file\_metacache](https://docs.starrocks.io/docs/sql-reference/System_variable.md#enable_file_metacache) * [enable\_datacache\_async\_populate\_mode](https://docs.starrocks.io/docs/sql-reference/System_variable.md) * [enable\_datacache\_sharing](https://docs.starrocks.io/docs/sql-reference/System_variable.md#enable_datacache_sharing) ##### FE configurations[​](#fe-configurations "Direct link to FE configurations") * [enable\_trace\_historical\_node](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#enable_trace_historical_node) ##### BE configurations[​](#be-configurations "Direct link to BE configurations") * [datacache\_enable](https://docs.starrocks.io/docs/administration/management/BE_configuration.md#datacache_enable) * [datacache\_mem\_size](https://docs.starrocks.io/docs/administration/management/BE_configuration.md#datacache_mem_size) * [datacache\_disk\_size](https://docs.starrocks.io/docs/administration/management/BE_configuration.md#datacache_disk_size) * [datacache\_auto\_adjust\_enable](https://docs.starrocks.io/docs/administration/management/BE_configuration.md#datacache_auto_adjust_enable) * [datacache\_disk\_high\_level](https://docs.starrocks.io/docs/administration/management/BE_configuration.md#datacache_disk_high_level) * [datacache\_disk\_safe\_level](https://docs.starrocks.io/docs/administration/management/BE_configuration.md#datacache_disk_safe_level) * [datacache\_disk\_low\_level](https://docs.starrocks.io/docs/administration/management/BE_configuration.md#datacache_disk_low_level) * [datacache\_disk\_adjust\_interval\_seconds](https://docs.starrocks.io/docs/administration/management/BE_configuration.md#datacache_disk_adjust_interval_seconds) * [datacache\_disk\_idle\_seconds\_for\_expansion](https://docs.starrocks.io/docs/administration/management/BE_configuration.md#datacache_disk_idle_seconds_for_expansion) * [datacache\_min\_disk\_quota\_for\_adjustment](https://docs.starrocks.io/docs/administration/management/BE_configuration.md#datacache_min_disk_quota_for_adjustment) * [datacache\_eviction\_policy](https://docs.starrocks.io/docs/administration/management/BE_configuration.md#datacache_eviction_policy) * [datacache\_inline\_item\_count\_limit](https://docs.starrocks.io/docs/administration/management/BE_configuration.md#datacache_inline_item_count_limit) --- ### Data Cache observability In earlier versions, there are no rich metrics or efficient methods to monitor the performance, usage, and health of [Data Cache](https://docs.starrocks.io/docs/data_source/data_cache.md). In v3.3, StarRocks improves the observability of Data Cache by offering efficient monitoring methods and more metrics. Users can now check the overall disk and memory usage of the data cache, and related metrics, enhancing monitoring of cache usage. > **NOTE** > > From v3.4.0 onwards, queries against external catalogs and cloud-native tables (in shared-data clusters) use a unified Data Cache instance. Therefore, unless otherwise specified, the following methods default to displaying the metrics of the Data Cache instance itself, which includes the cache usage of queries against both external catalogs and cloud-native tables. #### SQL commands[​](#sql-commands "Direct link to SQL commands") You can run SQL commands to view the capacity and usage of Data Cache on each BE node. ##### SHOW BACKENDS[​](#show-backends "Direct link to SHOW BACKENDS") The `DataCacheMetrics` field records the used disk and memory space of Data Cache on a specific BE. ```sql mysql> show backends\G *************************** 1. row *************************** BackendId: 10004 IP: XXX.XX.XX.XXX HeartbeatPort: 4450 BePort: 4448 HttpPort: 4449 BrpcPort: 4451 LastStartTime: 2023-12-13 20:09:30 LastHeartbeat: 2023-12-13 20:10:43 Alive: true SystemDecommissioned: false ClusterDecommissioned: false TabletNum: 48 DataUsedCapacity: 0.000 B AvailCapacity: 280.103 GB TotalCapacity: 1.968 TB UsedPct: 86.10 % MaxDiskUsedPct: 86.10 % ErrMsg: Version: datacache-heartbeat-c68caf7 Status: {"lastSuccessReportTabletsTime":"2023-12-13 20:10:38"} DataTotalCapacity: 280.103 GB DataUsedPct: 0.00 % CpuCores: 104 NumRunningQueries: 0 MemUsedPct: 0.00 % CpuUsedPct: 0.0 % DataCacheMetrics: Status: Normal, DiskUsage: 0.00GB/2.00GB, MemUsage: 0.00GB/30.46GB 1 row in set (1.90 sec) ``` ##### information\_schema[​](#information_schema "Direct link to information_schema") The `be_datacache_metrics` view in `information_schema` records the following Data Cache-related information. ```bash mysql> select * from information_schema.be_datacache_metrics; +-------+--------+------------------+-----------------+-----------------+----------------+-----------------+----------------------------------------------------------------------------------------------+ | BE_ID | STATUS | DISK_QUOTA_BYTES | DISK_USED_BYTES | MEM_QUOTA_BYTES | MEM_USED_BYTES | META_USED_BYTES | DIR_SPACES | +-------+--------+------------------+-----------------+-----------------+----------------+-----------------+----------------------------------------------------------------------------------------------+ | 10004 | Normal | 2147483648 | 0 | 32706263420 | 0 | 0 | [{"Path":"/home/disk1/datacache","QuotaBytes":2147483648}] | +-------+--------+------------------+-----------------+-----------------+----------------+-----------------+----------------------------------------------------------------------------------------------+ 1 row in set (5.41 sec) ``` * `BE_ID`: the BE ID * `STATUS`: the BE status * `DISK_QUOTA_BYTES`: the disk cache capacity configured by users, in bytes * `DISK_USED_BYTES`: the disk cache space that has been used, in bytes * `MEM_QUOTA_BYTES`: the memory cache capacity configured by users, in bytes * `MEM_USED_BYTES`: the memory cache space that has been used, in bytes * `META_USED_BYTES`: the space used to cache metadata * `DIR_SPACES`: the cache path and its cache size #### API call[​](#api-call "Direct link to API call") Since v3.3.2, StarRocks provides two APIs to get cache metrics, which reflect the cache state at different levels: * `/api/datacache/app_stat`: Query the Block Cache and Page Cache hit rates. * `/api/datacache/stat`: the underlying execution state of Data Cache. This interface is mainly used for maintenance and bottleneck identification of Data Cache. It does not reflect the actual hit rate of the query. Common users do not need to pay attention to this interface. ##### View cache hit metrics[​](#view-cache-hit-metrics "Direct link to View cache hit metrics") View the cache hit metrics by accessing the following API interface: ```bash http://${BE_HOST}:${BE_HTTP_PORT}/api/datacache/app_stat ``` Return: ```bash { "block_cache_hit_bytes": 1642106883, "block_cache_miss_bytes": 8531219739, "block_cache_hit_rate": 0.16, "block_cache_hit_bytes_last_minute": 899037056, "block_cache_miss_bytes_last_minute": 4163253265, "block_cache_hit_rate_last_minute": 0.18, "page_cache_hit_count": 15048, "page_cache_miss_count": 10032, "page_cache_hit_rate": 0.6, "page_cache_hit_count_last_minute": 10032, "page_cache_miss_count_last_minute": 5016, "page_cache_hit_rate_last_minute": 0.67 } ``` | **Metric** | **Description** | | --------------------------------------- | --------------------------------------------------------------------------------------------------- | | block\_cache\_hit\_bytes | Bytes read from the block cache. | | block\_cache\_miss\_bytes | Bytes read from remote storage (Block Cache misses). | | block\_cache\_hit\_rate | Block Cache hit rate, `(block_cache_hit_bytes / (block_cache_hit_bytes + block_cache_miss_bytes))`. | | block\_cache\_hit\_bytes\_last\_minute | Bytes read from the Block Cache in the last minute. | | block\_cache\_miss\_bytes\_last\_minute | Bytes read from the remote storage in the last minute. | | block\_cache\_hit\_rate\_last\_minute | Block Cache hit rate in the last minute. | | page\_cache\_hit\_count | Number of pages read from the Page Cache. | | page\_cache\_miss\_count | Number of pages missed in the Page Cache. | | block\_cache\_hit\_rate | Page Cache hit rate: `(page_cache_hit_count / (page_cache_hit_count + page_cache_miss_count))`. | | page\_cache\_hit\_count\_last\_minute | Number of pages read from the Page Cache in the last minute. | | page\_cache\_miss\_count\_last\_minute | Number of pages missed in the Page Cache in the last minute. | | page\_cache\_hit\_rate\_last\_minute | Page Cache hit rate in the last minute. | ##### View the underlying execution state of Data Cache[​](#view-the-underlying-execution-state-of-data-cache "Direct link to View the underlying execution state of Data Cache") You can get more detailed metrics on the Data Cache by accessing the following API interfaces. ```bash http://${BE_HOST}:${BE_HTTP_PORT}/api/datacache/stat ``` The results are as follows: ```json { "page_cache_mem_quota_bytes": 10679976935, "page_cache_mem_used_bytes": 10663052377, "page_cache_mem_used_rate": 1.0, "page_cache_hit_count": 276890, "page_cache_miss_count": 153126, "page_cache_hit_rate": 0.64, "page_cache_hit_count_last_minute": 11196, "page_cache_miss_count_last_minute": 9982, "page_cache_hit_rate_last_minute": 0.53, "block_cache_status": "NORMAL", "block_cache_disk_quota_bytes": 214748364800, "block_cache_disk_used_bytes": 11371020288, "block_cache_disk_used_rate": 0.05, "block_cache_disk_spaces": "/disk1/sr/be/storage/datacache:107374182400;/disk2/sr/be/storage/datacache:107374182400", "block_cache_meta_used_bytes": 11756727, "block_cache_hit_count": 57707, "block_cache_miss_count": 2556, "block_cache_hit_rate": 0.96, "block_cache_hit_bytes": 15126253744, "block_cache_miss_bytes": 620687633, "block_cache_hit_count_last_minute": 18108, "block_cache_miss_count_last_minute": 2449, "block_cache_hit_bytes_last_minute": 4745613488, "block_cache_miss_bytes_last_minute": 607536783, "block_cache_read_disk_bytes": 15126253744, "block_cache_write_bytes": 11338218093, "block_cache_write_success_count": 43377, "block_cache_write_fail_count": 36394, "block_cache_remove_bytes": 0, "block_cache_remove_success_count": 0, "block_cache_remove_fail_count": 0, "block_cache_current_reading_count": 0, "block_cache_current_writing_count": 0, "block_cache_current_removing_count": 0 } ``` ##### Metric description[​](#metric-description "Direct link to Metric description") | **Metric** | **Description** | | --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | page\_cache\_mem\_quota\_bytes | Current memory limit of Page Cache. | | page\_cache\_mem\_used\_bytes | Current actual memory used by Page Cache. | | page\_cache\_mem\_used\_rate | Current memory usage rate of Page Cache. | | page\_cache\_hit\_count | Number of Page Cache hits. | | page\_cache\_miss\_count | Number of Page Cache misses. | | page\_cache\_hit\_rate | Hit rate of Page Cache. | | page\_cache\_hit\_count\_last\_minute | Number of Page Cache hits in the last minute. | | page\_cache\_miss\_count\_last\_minute | Number of Page Cache misses in the last minute. | | page\_cache\_hit\_rate\_last\_minute | Hit rate of Page Cache in the last minute. | | block\_cache\_status | Status of the Block Cache, including:`NORMAL`: The instance runs normally.`ABNORMAL`: Data cannot be read or written into the cache. The issue must be located using logs.`UPDATING`: The instance is being updated, such as the updating state during online scaling. | | block\_cache\_disk\_quota\_bytes | The disk cache capacity of Block Cache configured by users, in bytes. | | block\_cache\_disk\_used\_bytes | The disk cache space that has been used by Block Cache, in bytes. | | block\_cache\_disk\_used\_rate | The actual disk cache usage rate of Block Cache, in percentages. | | block\_cache\_disk\_spaces | The disk cache information of Block Cache configured by users, including each cache path and the cache size. | | block\_cache\_meta\_used\_bytes | The memory space used to cache Block Cache metadata, in bytes. | | block\_cache\_hit\_count | Number of Block Cache hits. | | block\_cache\_miss\_count | Number of cache misses. | | block\_cache\_hit\_rate | Hit rate of Block Cache. | | block\_cache\_hit\_bytes | Number of bytes that are hit in the Block Cache. | | block\_cache\_miss\_bytes | Number of bytes that are missed in the Block Cache. | | block\_cache\_hit\_count\_last\_minute | Number of Block Cache hits in the last minute. | | block\_cache\_miss\_count\_last\_minute | Number of Block Cache misses in the last minute. | | block\_cache\_hit\_bytes\_last\_minute | Number of types hit by Block Cache in the last minute. | | block\_cache\_miss\_bytes\_last\_minute | Number of types missed by Block Cache in the last minute. | | block\_cache\_buffer\_item\_count | The current number of Buffer instances in the Block Cache. Buffer instances refer to common data caches, such as when reading part of the raw data from a remote file and caching the data directly in memory or on disks. | | block\_cache\_buffer\_item\_bytes | Number of types used to cache the Buffer instance in Block Cache. | | block\_cache\_read\_disk\_bytes | Number of bytes read from Block Cache. | | block\_cache\_write\_bytes | Number of bytes written to Block Cache. | | block\_cache\_write\_success\_count | Number of successful Block Cache writes. | | block\_cache\_write\_fail\_count | Number of failed Block Cache writes. | | block\_cache\_remove\_bytes | Number of bytes removed from Block Cache. | | block\_cache\_remove\_success\_count | Number of successful remove operations from Block Cache. | | block\_cache\_remove\_fail\_count | Number of failed remove operations from Block Cache. | | block\_cache\_current\_reading\_count | Number of read operations currently being executed in Block Cache. | | block\_cache\_current\_writing\_count | Number of write operations currently being executed in Block Cache. | | block\_cache\_current\_removing\_count | Number of remove operations currently being executed in Block Cache. | --- ### Data Cache FAQ This topic describes some frequently asked questions (FAQ) and common issues about Data Cache and provides troubleshooting steps and solutions to these issues. #### Enabling Data Cache[​](#enabling-data-cache "Direct link to Enabling Data Cache") ##### How to confirm whether Data Cache is successfully enabled?[​](#how-to-confirm-whether-data-cache-is-successfully-enabled "Direct link to How to confirm whether Data Cache is successfully enabled?") In most cases, you can check whether Data Cache is successfully enabled by any of the following methods: * Execute `SHOW BACKENDS` (or `SHOW COMPUTE NODES`) from your SQL client, and check the value of `DataCacheMetrics`. You can confirm that Data Cache is enabled if the disk or memory cache quota is greater than 0. ```sql mysql> show backends \G *************************** 1. row *************************** BackendId: 89041 IP: X.X.X.X HeartbeatPort: 9050 BePort: 9060 HttpPort: 8040 BrpcPort: 8060 LastStartTime: 2025-05-29 14:45:37 LastHeartbeat: 2025-05-29 19:20:32 Alive: true SystemDecommissioned: false ClusterDecommissioned: false TabletNum: 10 DataUsedCapacity: 0.000 B AvailCapacity: 1.438 TB TotalCapacity: 1.718 TB UsedPct: 16.27 % MaxDiskUsedPct: 16.27 % ErrMsg: Version: main-c15b412 Status: {"lastSuccessReportTabletsTime":"2025-05-29 19:20:30"} DataTotalCapacity: 1.438 TB DataUsedPct: 0.00 % CpuCores: 8 MemLimit: 50.559GB NumRunningQueries: 0 MemUsedPct: 0.50 % CpuUsedPct: 0.2 % DataCacheMetrics: Status: Normal, DiskUsage: 44MB/1TB, MemUsage: 0B/0B Location: StatusCode: OK 1 row in set (0.00 sec) ``` In the above example, the disk cache quota of Data Cache is 1TB, and 44MB is currently in use; while the memory cache quota is 0B, thus memory cache is not enabled. * You can access the BE Web Console (`http://${BE_HOST}:${BE_HTTP_PORT}/api/datacache/stat`) to check the current Data Cache quota, hit rate and other metrics. You can confirm that Data Cache is enbaled if `disk_quota_bytes` or `mem_quota_bytes` is greater than 0. ![Data Cache FAQ - Web Console](/assets/images/data_cache_be_web_console-c0bf542a74f6593c8b61602da56d645d.png) ##### Why is Data Cache not enabled by default?[​](#why-is-data-cache-not-enabled-by-default "Direct link to Why is Data Cache not enabled by default?") From v3.3 onwards, BE will attempt to enable Data Cache upon startup. However, if there is insufficient space available on the current disk, Data Cache will not be enabled automatically. It may be caused by the following situations: * **Percentage**: The current disk usage is high. * **Remaining space**: The remaining disk space size is relatively low. Therefore, if Data Cache is not enabled by default, you can first check the current disk usage and increase disk capacity if necessary. Alternatively, you can manually enable Data Cache by configuring cache quota based on the current available disk space. ```text # disable Data Cache Automatic Adjustment datacache_auto_adjust_enable = false # Set Data Cache disk quota manually datacache_disk_size = 1T ``` #### Using Data Cache[​](#using-data-cache "Direct link to Using Data Cache") ##### What catalog types does Data Cache support?[​](#what-catalog-types-does-data-cache-support "Direct link to What catalog types does Data Cache support?") Data Cache currently supports external catalog types that use StarRocks Native File Reader (such as Parquet/ORC/CSV Reader), including Hive, Iceberg, Hudi, Delta Lake, and Paimon. Catalogs that access data based on JNI (such as JDBC Catalog) are not supported yet. note Some catalogs may use different data access methods based on certain conditions (such as file type, and data status). For example, for the Paimon catalog, StarRocks may automatically choose whether to use Native File Reader or JNI to access data based on the compaction status of the current data. When JNI is used to access Paimon data, Data Cache acceleration is not supported. ##### How can I know that a query hits the cache?[​](#how-can-i-know-that-a-query-hits-the-cache "Direct link to How can I know that a query hits the cache?") You can check Data Cache-related metrics in the corresponding query profile. The metrics `DataCacheReadBytes` and `DataCacheReadCounter` indicates the local cache hit status. ```text - DataCacheReadBytes: 518.73 MB - __MAX_OF_DataCacheReadBytes: 4.73 MB - __MIN_OF_DataCacheReadBytes: 16.00 KB - DataCacheReadCounter: 684 - __MAX_OF_DataCacheReadCounter: 4 - __MIN_OF_DataCacheReadCounter: 0 - DataCacheReadTimer: 737.357us - DataCacheWriteBytes: 7.65 GB - __MAX_OF_DataCacheWriteBytes: 64.39 MB - __MIN_OF_DataCacheWriteBytes: 0.00 - DataCacheWriteCounter: 7.887K (7887) - __MAX_OF_DataCacheWriteCounter: 65 - __MIN_OF_DataCacheWriteCounter: 0 - DataCacheWriteTimer: 23.467ms - __MAX_OF_DataCacheWriteTimer: 62.280ms - __MIN_OF_DataCacheWriteTimer: 0ns ``` ##### Why does a query miss the cache when Data Cache is enabled?[​](#why-does-a-query-miss-the-cache-when-data-cache-is-enabled "Direct link to Why does a query miss the cache when Data Cache is enabled?") Follow these steps for troubleshooting: 1. Check whether Data Cache support the current catalog type. 2. Confirm whether the query statement meets the cache population conditions. In certain cases, Data Cache will reject cache population for some queries. For details, see [Data Cache Population Rules](https://docs.starrocks.io/docs/data_source/data_cache.md#population-rules). The `EXPLAIN VERBOSE` command can be used to check whether a query triggers cache population. Example: ```sql mysql> EXPLAIN VERBOSE SELECT col1 FROM hudi_table; | 0:HudiScanNode | | TABLE: hudi_table | | partitions=3/3 | | cardinality=9084 | | avgRowSize=2.0 | | dataCacheOptions={populate: false} | | cardinality: 9084 | +-----------------------------------------+ ``` In the above example, the `populate` field of the `dataCacheOptions` section is `false`, indicating the cache will not be populated for the query. To enable Data Cache for such queries, you can modify the default population behavior by setting the system variable `populate_datacache_mode` to `always`. #### Data Cache Hit[​](#data-cache-hit "Direct link to Data Cache Hit") ##### Why sometimes the same query needs to be executed multiple times before fully hitting the cache?[​](#why-sometimes-the-same-query-needs-to-be-executed-multiple-times-before-fully-hitting-the-cache "Direct link to Why sometimes the same query needs to be executed multiple times before fully hitting the cache?") In the current version, Data Cache uses asynchronous population by default to reduce its impact on query performance. When using asynchronous population, the system will try to cache the accessed data in the background without affecting the read performance as much as possible. Therefore, executing a query for a single time can only cache a part of the data. You need to run the query for multiple times to cache all the data required by the query. You can also use the synchronous cache population by setting `enable_datacache_async_populate_mode=false`, or warm up the target data in advance by `CACHE SELECT`. ##### Why is it that all the data in the current query has been cached, but there is still a few data accessed remotely?[​](#why-is-it-that-all-the-data-in-the-current-query-has-been-cached-but-there-is-still-a-few-data-accessed-remotely "Direct link to Why is it that all the data in the current query has been cached, but there is still a few data accessed remotely?") In the current version, I/O adaptation is enabled by default to optimize cache performance when disk I/O load is high, which may result in a small number of requests directly accessing the remote storage in some cases. You can disable the I/O adaptation function by setting `enable_datacache_io-adapter` to `false`. #### Others[​](#others "Direct link to Others") ##### How to clear the cached data?[​](#how-to-clear-the-cached-data "Direct link to How to clear the cached data?") Currently, Data Cache does not provide a direct interface to clear the cached data, but you can choose one of the following methods to clear it: * You can clean up cached data by deleting all data (including block files and meta directories) in the `datacache` directory on the BE/CN nodes, and then restarting the nodes. (Recommended) * If you want to avoid restarting the BE/CN nodes, you can also clean cached data indirectly by scaling down the cache quota during runtime. For example, if you have set the disk cache quota to 2TB earlier, you can first scale it down to 0 (the system will automatically clean up the cached data), and then reset it to 2TB. Example: ```sql UPDATE be_configs SET VALUE="0" WHERE NAME="datacache_disk_size" and BE_ID=10005; UPDATE be_configs SET VALUE="2T" WHERE NAME="datacache_disk_size" and BE_ID=10005; ``` note When cleaning cached data during runtime, be cautious with the `WHERE` condition in the statement to avoid accidentally damaging other irrelevant parameters or nodes. ##### How to improve Data Cache performance?[​](#how-to-improve-data-cache-performance "Direct link to How to improve Data Cache performance?") With Data Cache, StarRocks essentially accesses local memory or disk instead of the remote storage. Therefore, the performance is directly related to the local cache medium. If you find that cache access latency is high due to high disk load, you may consider improving the performance of the local cache medium: * Prioritize using high-performance NVME disks as cache disks. * If high-performance disks are not available, you can also increase the number of disks to share the I/O pressure. * Increase the server memory of the BE/CN nodes (rather than the Data Cache memory quota), using the Page Cache of the operating system to reduce the number of direct disk accesses and the disk I/O pressure. --- ### Data Lakehouse ![DLA](/assets/images/1.1-8-dla-c67d601d709b092317fa11eb64ac1783.png) In addition to efficient analytics of local data, StarRocks can work as the compute engine to analyze data stored in data lakes such as Apache Hudi, Apache Iceberg, and Delta Lake. One of the key features of StarRocks is its external catalog, which acts as the linkage to an externally maintained metastore. This functionality provides users with the capability to query external data sources seamlessly, eliminating the need for data migration. As such, users can analyze data from different systems such as HDFS and Amazon S3, in various file formats such as Parquet, ORC, and CSV, etc. The preceding figure shows a data lake analytics scenario where StarRocks is responsible for data computing and analysis, and the data lake is responsible for data storage, organization, and maintenance. Data lakes allow users to store data in open storage formats and use flexible schemas to produce reports on "single source of truth" for various BI, AI, ad-hoc, and reporting use cases. StarRocks fully leverages the advantages of its vectorization engine and CBO, significantly improving the performance of data lake analytics. #### Key ideas[​](#key-ideas "Direct link to Key ideas") * Open Data Formats: Supports a variety of data types, including JSON, Parquet, and Avro, facilitating the storage and processing of both structured and unstructured data. * Metadata Management: Implements a shared metadata layer, often utilizing formats like the Iceberg table format, to organize and govern data efficiently. * Diverse Query Engines: Incorporates multiple engines, like enhanced versions of Presto and Spark, to cater to various analytics and AI use cases. * Governance and Security: Features robust built-in mechanisms for data security, privacy, and compliance, ensuring data integrity and trustworthiness. #### Advantages of Data Lakehouse architecture[​](#advantages-of-data-lakehouse-architecture "Direct link to Advantages of Data Lakehouse architecture") * Flexibility and Scalability: Seamlessly manages diverse data types and scales with the organization’s needs. * Cost-Effectiveness: Offers an economical alternative for data storage and processing, compared to traditional methods. * Enhanced Data Governance: Improves data control, management, and integrity, ensuring reliable and secure data handling. * AI and Analytics Readiness: Perfectly suited for complex analytical tasks, including machine learning and AI-driven data processing. #### StarRocks approach[​](#starrocks-approach "Direct link to StarRocks approach") The key things to consider are: * Standardizing the integration with catalog, or metadata services * Elastic scalability of compute nodes * Flexible caching mechanisms *** #### Catalogs[​](#catalogs "Direct link to Catalogs") StarRocks has two types of catalogs, internal and external. The internal catalog contains metadata for data stored within StarRocks databases. External catalogs are used to work with data stored externally, including the data managed by Hive, Iceberg, Delta Lake, and Hudi. There are many other external systems, links are in the More Information section at the bottom of the page. #### Compute node (CN) scaling[​](#compute-node-cn-scaling "Direct link to Compute node (CN) scaling") Separation of storage and compute reduces the complexity of scaling. Since the StarRocks compute nodes are only storing local cache, nodes can be added or removed based on load. #### Data cache[​](#data-cache "Direct link to Data cache") Cache on the compute nodes is optional. If your compute nodes are spinning up and down quickly based on quickly changing load patterns or your queries are often only on the most recent data it might not make sense to cache data. *** #### [🗃️ Catalog](https://docs.starrocks.io/docs/data_source/catalog/catalog_intro) [14 items](https://docs.starrocks.io/docs/data_source/catalog/catalog_intro) #### [🗃️ Data cache](https://docs.starrocks.io/docs/cover_pages/data_cache) [4 items](https://docs.starrocks.io/docs/cover_pages/data_cache) #### [📄️ External table](https://docs.starrocks.io/docs/data_source/External_table.md) [The External Table feature is no longer recommended except for certain corner usage cases, and might be deprecated in future releases.](https://docs.starrocks.io/docs/data_source/External_table.md) #### [📄️ File external table](https://docs.starrocks.io/docs/data_source/file_external_table.md) [File external table is a special type of external table.](https://docs.starrocks.io/docs/data_source/file_external_table.md) #### [📄️ Data lake FAQ](https://docs.starrocks.io/docs/data_source/datalake_faq.md) [FAQ for common data lake analytics issues in StarRocks, including catalog setup and query problems.](https://docs.starrocks.io/docs/data_source/datalake_faq.md) #### [📄️ Feature Support](https://docs.starrocks.io/docs/data_source/feature-support-data-lake-analytics.md) [From v2.3 onwards, StarRocks supports managing external data sources and analyzing data in data lakes via external catalogs.](https://docs.starrocks.io/docs/data_source/feature-support-data-lake-analytics.md) --- ### Data lake FAQ This topic describes some commonly asked questions (FAQ) about data lake and provides solutions to these issues. Some metrics mentioned in this topic can be obtained only from the profiles of the SQL queries. To obtain the profiles of SQL queries, you must specify `set enable_profile=true`. #### Slow HDFS DataNodes[​](#slow-hdfs-datanodes "Direct link to Slow HDFS DataNodes") ##### Issue description[​](#issue-description "Direct link to Issue description") When you access the data files stored in your HDFS cluster, you may find a huge difference between the values of the `__MAX_OF_FSIOTime` and `__MIN_OF_FSIOTime` metrics from the profiles of the SQL queries you run. This indicates that some DataNodes in the HDFS cluster are slow. The following example is a typical profile that indicates a slow HDFS DataNode issue: ```plaintext - InputStream: 0 - AppIOBytesRead: 22.72 GB - __MAX_OF_AppIOBytesRead: 187.99 MB - __MIN_OF_AppIOBytesRead: 64.00 KB - AppIOCounter: 964.862K (964862) - __MAX_OF_AppIOCounter: 7.795K (7795) - __MIN_OF_AppIOCounter: 1 - AppIOTime: 1s372ms - __MAX_OF_AppIOTime: 4s358ms - __MIN_OF_AppIOTime: 1.539ms - FSBytesRead: 15.40 GB - __MAX_OF_FSBytesRead: 127.41 MB - __MIN_OF_FSBytesRead: 64.00 KB - FSIOCounter: 1.637K (1637) - __MAX_OF_FSIOCounter: 12 - __MIN_OF_FSIOCounter: 1 - FSIOTime: 9s357ms - __MAX_OF_FSIOTime: 60s335ms - __MIN_OF_FSIOTime: 1.536ms ``` ##### Solution[​](#solution "Direct link to Solution") You can use one of the following solutions to resolve this issue: * **\[Recommended]** Enable the [data cache](https://docs.starrocks.io/docs/data_source/data_cache.md) feature, which eliminates the impact of slow HDFS DataNodes on queries by automatically caching the data from external storage systems to the BEs or CNs of your StarRocks cluster. * **\[Recommended]** Shorten the timeout duration between the HDFS client and DataNode. This solution is suitable when Data Cache cannot help resolve the slow HDFS DataNode issue. * Enable the [Hedged Read](https://hadoop.apache.org/docs/r2.8.3/hadoop-project-dist/hadoop-common/release/2.4.0/RELEASENOTES.2.4.0.html) feature. With this feature enabled, if a read from a block is slow, StarRocks starts up a new read, which runs in parallel to the original read, to read against a different block replica. Whenever one of the two reads returns, the other read is cancelled. **The Hedged Read feature can help accelerate reads, but it also significantly increases heap memory consumption on Java virtual machines (JVMs). Therefore, if your physical machines provide a small memory capacity, we recommend that you do not enable the Hedged Read feature.** ###### \[Recommended] Data Cache[​](#recommended-data-cache "Direct link to [Recommended] Data Cache") See [Data Cache](https://docs.starrocks.io/docs/data_source/data_cache.md). ###### \[Recommended] Shorten timeout duration between HDFS client and DataNode[​](#recommended-shorten-timeout-duration-between-hdfs-client-and-datanode "Direct link to [Recommended] Shorten timeout duration between HDFS client and DataNode") Configure the `dfs.client.socket-timeout` property in the `hdfs-site.xml` file to shorten the timeout duration between the HDFS client and DataNode. (The default timeout duration is 60s, which is a bit long.) As such, when StarRocks encounters a slow DataNode, the connection request from it can time out within a very short period of time and then be forwarded to another DataNode. The following example sets a 5-second timeout duration: ```xml dfs.client.socket-timeout 5000 ``` ###### Hedged Read[​](#hedged-read "Direct link to Hedged Read") Use the following parameters (supported from v3.0 onwards) in the BE or CN configuration file `be.conf` to enable and configure the Hedged Read feature in your HDFS cluster. | Parameter | Default value | Description | | --------------------------------------------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | hdfs\_client\_enable\_hedged\_read | false | Specifies whether to enable the hedged read feature. | | hdfs\_client\_hedged\_read\_threadpool\_size | 128 | Specifies the size of the Hedged Read thread pool on your HDFS client. The thread pool size limits the number of threads to dedicate to the running of hedged reads in your HDFS client. This parameter is equivalent to the `dfs.client.hedged.read.threadpool.size` parameter in the `hdfs-site.xml` file of your HDFS cluster. | | hdfs\_client\_hedged\_read\_threshold\_millis | 2500 | Specifies the number of milliseconds to wait before starting up a hedged read. For example, you have set this parameter to `30`. In this situation, if a read from a block has not returned within 30 milliseconds, your HDFS client immediately starts up a hedged read against a different block replica. This parameter is equivalent to the `dfs.client.hedged.read.threshold.millis` parameter in the `hdfs-site.xml` file of your HDFS cluster. | If the value of any of the following metrics in your query profiles exceeds `0`, the Hedged Read feature is enabled. | Metric | Description | | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | TotalHedgedReadOps | The number of hedged reads that are started up. | | TotalHedgedReadOpsInCurThread | The number of times that StarRocks has to start up a hedged read in the current thread instead of in a new thread because the Hedged Read thread pool has reached its maximum size specified by the `hdfs_client_hedged_read_threadpool_size` parameter. | | TotalHedgedReadOpsWin | The number of times that a hedged read beats its original read. | #### How do I resolve the error “ERROR 1064 (HY000): Type mismatches on column \[is\_refund], JDBC result type is Integer, please set the type to one of tinyint,smallint,int,bigint” when querying a table in the Hive Catalog?[​](#how-do-i-resolve-the-error-error-1064-hy000-type-mismatches-on-column-is_refund-jdbc-result-type-is-integer-please-set-the-type-to-one-of-tinyintsmallintintbigint-when-querying-a-table-in-the-hive-catalog "Direct link to How do I resolve the error “ERROR 1064 (HY000): Type mismatches on column [is_refund], JDBC result type is Integer, please set the type to one of tinyint,smallint,int,bigint” when querying a table in the Hive Catalog?") This issue is caused by an incorrect JDBC connection configuration. Add the parameter `tinyInt1isBit=false` to your JDBC URI to prevent this issue: ```sql "jdbc_uri" = "jdbc:mysql://xxx:3306?database=yl_spmibill&tinyInt1isBit=false" ``` #### Why can’t I query the latest updated data in the Iceberg Catalog (even after refresh or catalog rebuild), and how should I troubleshoot this?[​](#why-cant-i-query-the-latest-updated-data-in-the-iceberg-catalog-even-after-refresh-or-catalog-rebuild-and-how-should-i-troubleshoot-this "Direct link to Why can’t I query the latest updated data in the Iceberg Catalog (even after refresh or catalog rebuild), and how should I troubleshoot this?") First check whether the issue is caused by Data Cache being enabled. Follow these steps to verify: 1. Compare the scanned data files between StarRocks and Spark: * In StarRocks: `select file_path, spec_id from db.table_name$files;` * In Spark: `select file_path, spec_id from db.table_name.files;` 2. If the results are consistent, continue troubleshooting by disabling Data Cache and querying again to see whether the issue persists. Root cause: To update the Iceberg table data is to overwrite old files, which corrupts Iceberg’s historical data. The correct behavior is to generate new file names when writing updates. StarRocks Data Cache uses the file name, file size, and modification time to determine whether cached data is valid. Since Iceberg does not overwrite files and the modification time is always 0, StarRocks incorrectly treats the files as unchanged and reads from cache, resulting in outdated query results. #### FE constantly crashes after queries against tables in external catalog integrated with JuiceFS. How can I solve this?[​](#fe-constantly-crashes-after-queries-against-tables-in-external-catalog-integrated-with-juicefs-how-can-i-solve-this "Direct link to FE constantly crashes after queries against tables in external catalog integrated with JuiceFS. How can I solve this?") To solve this, restart FE after adding the following configuration items to **fe.conf**: ```properties proc_profile_mem_enable=false proc_profile_cpu_enable=false ``` --- ### External table note The External Table feature is no longer recommended except for certain corner usage cases, and might be deprecated in future releases. To manage and query data from external data sources in general scenarios, [External Catalog](https://docs.starrocks.io/docs/data_source/catalog/catalog_overview.md) is recommended. * From v3.0 onwards, we recommend that you use catalogs to query data from Hive, Iceberg, and Hudi. See [Hive catalog](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md), [Iceberg catalog](https://docs.starrocks.io/docs/data_source/catalog/iceberg/iceberg_catalog.md), and [Hudi catalog](https://docs.starrocks.io/docs/data_source/catalog/hudi_catalog.md). * From v3.1 onwards, we recommend that you use [JDBC catalog](https://docs.starrocks.io/docs/data_source/catalog/jdbc_catalog.md) to query data from MySQL and PostgreSQL and use [Elasticsearch catalog](https://docs.starrocks.io/docs/data_source/catalog/elasticsearch_catalog.md) to query data from Elasticsearch. * From v3.2.9 and v3.3.1, we recommend that you use [JDBC catalog](https://docs.starrocks.io/docs/data_source/catalog/jdbc_catalog.md) to query data from Oracle and SQL Server. * The External Table feature was designed to help with loading data into StarRocks, NOT to perform efficient queries against external systems as a normal operation. A more performant solution would be to load the data into StarRocks. StarRocks supports access to other data sources by using external tables. External tables are created based on data tables that are stored in other data sources. StarRocks only stores the metadata of the data tables. You can use external tables to directly query data in other data sources. Currently, except StarRocks external table, all other external tables are deprecated. **You can only write data from another StarRocks cluster into the current StarRocks cluster. You cannot read data from it. For data sources other than StarRocks, you can only read data from these data sources.** From 2.5 onwards, StarRocks provides the Data Cache feature, which accelerates hot data queriers on external data sources. For more information, see [Data Cache](https://docs.starrocks.io/docs/data_source/data_cache.md). #### StarRocks external table[​](#starrocks-external-table "Direct link to StarRocks external table") From StarRocks 1.19 onwards, StarRocks allows you to use a StarRocks external table to write data from one StarRocks cluster to another. This achieves read-write separation and provides better resource isolation. You can first create a destination table in the destination StarRocks cluster. Then, in the source StarRocks cluster, you can create a StarRocks external table that has the same schema as the destination table and specify the information of the destination cluster and table in the `PROPERTIES` field. Data can be written from a source cluster to a destination cluster by using INSERT INTO statement to write into a StarRocks external table. It can help realize the following goals: * Data synchronization between StarRocks clusters. * Read-write separation. Data is written to the source cluster, and data changes from the source cluster are synchronized to the destination cluster, which provides query services. The following code shows how to create a destination table and an external table. ```sql # Create a destination table in the destination StarRocks cluster. CREATE TABLE t ( k1 DATE, k2 INT, k3 SMALLINT, k4 VARCHAR(2048), k5 DATETIME ) ENGINE=olap DISTRIBUTED BY HASH(k1); # Create an external table in the source StarRocks cluster. CREATE EXTERNAL TABLE external_t ( k1 DATE, k2 INT, k3 SMALLINT, k4 VARCHAR(2048), k5 DATETIME ) ENGINE=olap DISTRIBUTED BY HASH(k1) PROPERTIES ( "host" = "127.0.0.1", "port" = "9020", "user" = "user", "password" = "passwd", "database" = "db_test", "table" = "t" ); # Write data from a source cluster to a destination cluster by writing data into the StarRocks external table. The second statement is recommended for the production environment. insert into external_t values ('2020-10-11', 1, 1, 'hello', '2020-10-11 10:00:00'); insert into external_t select * from other_table; ``` Parameters: * **EXTERNAL:** This keyword indicates that the table to be created is an external table. * **host:** This parameter specifies the IP address of the leader FE node of the destination StarRocks cluster. * **port:** This parameter specifies the RPC port of the FE node of the destination StarRocks cluster. note To ensure that the source cluster to which the StarRocks external tables belong can access the destination StarRocks cluster, you must configure your network and firewall to allow access to the following ports: * The RPC port of the FE node. See `rpc_port` in the FE configuration file **fe/fe.conf**. The default RPC port is `9020`. * The bRPC port of the BE node. See `brpc_port` in the BE configuration file **be/be.conf**. The default bRPC port is `8060`. * **user:** This parameter specifies the username used to access the destination StarRocks cluster. * **password:** This parameter specifies the password used to access the destination StarRocks cluster. * **database:** This parameter specifies the database to which the destination table belongs. * **table:** This parameter specifies the name of the destination table. The following limits apply when you use a StarRocks external table: * You can only run the INSERT INTO and SHOW CREATE TABLE commands on a StarRocks external table. Other data writing methods are not supported. In addition, you cannot query data from a StarRocks external table or perform DDL operations on the external table. * The syntax of creating an external table is the same as creating a normal table, but the column names and other information in the external table must be the same as the destination table. * The external table synchronizes table metadata from the destination table every 10 seconds. If a DDL operation is performed on the destination table, there may be a delay for data synchronization between the two tables. #### (Deprecated) External table for a JDBC-compatible database[​](#deprecated-external-table-for-a-jdbc-compatible-database "Direct link to (Deprecated) External table for a JDBC-compatible database") From v2.3.0, StarRocks provides external tables to query JDBC-compatible databases. This way, you can analyze the data of such databases in a blazing fast manner without the need to import the data into StarRocks. This section describes how to create an external table in StarRocks and query data in JDBC-compatible databases. ##### Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before you use a JDBC external table to query data, make sure that the FEs and BEs have access to the download URL of the JDBC driver. The download URL is specified by the `driver_url` parameter in the statement used for creating the JDBC resource. ##### Create and manage JDBC resources[​](#create-and-manage-jdbc-resources "Direct link to Create and manage JDBC resources") ###### Create a JDBC resource[​](#create-a-jdbc-resource "Direct link to Create a JDBC resource") Before you create an external table to query data from a database, you need to create a JDBC resource in StarRocks to manage the connection information of the database. The database must support the JDBC driver and is referred as the "target database". After creating the resource, you can use it to create external tables. Execute the following statement to create a JDBC resource named `jdbc0`: ```sql CREATE EXTERNAL RESOURCE jdbc0 PROPERTIES ( "type"="jdbc", "user"="postgres", "password"="changeme", "jdbc_uri"="jdbc:postgresql://127.0.0.1:5432/jdbc_test", "driver_url"="https://repo1.maven.org/maven2/org/postgresql/postgresql/42.3.3/postgresql-42.3.3.jar", "driver_class"="org.postgresql.Driver" ); ``` The required parameters in `PROPERTIES` are as follows: * `type`: the type of the resource. Set the value to `jdbc`. * `user`: the username that is used to connect to the target database. * `password`: the password that is used to connect to the target database. * `jdbc_uri`: the URI that the JDBC driver uses to connect to the target database. The URI format must satisfy the database URI syntax. For the URI syntax of some common databases, visit the official websites of [Oracle](https://docs.oracle.com/en/database/oracle/oracle-database/21/jjdbc/data-sources-and-URLs.html#GUID-6D8EFA50-AB0F-4A2B-88A0-45B4A67C361E), [PostgreSQL](https://jdbc.postgresql.org/documentation/head/connect.html), [SQL Server](https://learn.microsoft.com/en-us/sql/connect/jdbc/building-the-connection-url?view=sql-server-ver16). > Note: The URI must include the name of the target database. For example, in the preceding code example, `jdbc_test` is the name of the target database that you want to connect. * `driver_url`: the download URL of the JDBC driver JAR package. An HTTP URL or file URL is supported, for example, `https://repo1.maven.org/maven2/org/postgresql/postgresql/42.3.3/postgresql-42.3.3.jar` or `file:///home/disk1/postgresql-42.3.3.jar`. * `driver_class`: the class name of the JDBC driver. The JDBC driver class names of common databases are as follows: * MySQL: com.mysql.jdbc.Driver (MySQL 5.x and earlier), com.mysql.cj.jdbc.Driver (MySQL 6.x and later) * SQL Server: com.microsoft.sqlserver.jdbc.SQLServerDriver * Oracle: oracle.jdbc.driver.OracleDriver * PostgreSQL: org.postgresql.Driver When the resource is being created, the FE downloads the JDBC driver JAR package by using the URL that is specified in the `driver_url` parameter, generates a checksum, and uses the checksum to verify the JDBC driver downloaded by BEs. > Note: If the download of the JDBC driver JAR package fails, the creation of the resource also fails. When BEs query the JDBC external table for the first time and find that the corresponding JDBC driver JAR package does not exist on their machines, BEs download the JDBC driver JAR package by using the URL that is specified in the `driver_url` parameter, and all JDBC driver JAR packages are saved in the `${STARROCKS_HOME}/lib/jdbc_drivers` directory. ###### View JDBC resources[​](#view-jdbc-resources "Direct link to View JDBC resources") Execute the following statement to view all JDBC resources in StarRocks: ```sql SHOW RESOURCES; ``` > Note: The `ResourceType` column is `jdbc`. ###### Delete a JDBC resource[​](#delete-a-jdbc-resource "Direct link to Delete a JDBC resource") Execute the following statement to delete the JDBC resource named `jdbc0`: ```sql DROP RESOURCE "jdbc0"; ``` > Note: After a JDBC resource is deleted, all JDBC external tables that are created by using that JDBC resource are unavailable. However, the data in the target database is not lost. If you still need to use StarRocks to query data in the target database, you can create the JDBC resource and the JDBC external tables again. ##### Create a database[​](#create-a-database "Direct link to Create a database") Execute the following statement to create and access a database named `jdbc_test` in StarRocks: ```sql CREATE DATABASE jdbc_test; USE jdbc_test; ``` > Note: The database name that you specify in the preceding statement does not need to be same as the name of the target database. ##### Create a JDBC external table[​](#create-a-jdbc-external-table "Direct link to Create a JDBC external table") Execute the following statement to create a JDBC external table named `jdbc_tbl` in the database `jdbc_test`: ```sql create external table jdbc_tbl ( `id` bigint NULL, `data` varchar(200) NULL ) ENGINE=jdbc properties ( "resource" = "jdbc0", "table" = "dest_tbl" ); ``` The required parameters in `properties` are as follows: * `resource`: the name of the JDBC resource used to create the external table. * `table`: the target table name in the database. For supported data types and data type mapping between StarRocks and target databases, see \[Data type mapping]\(External\_table.md#Data type mapping). > Note: > > * Indexes are not supported. > * You cannot use PARTITION BY or DISTRIBUTED BY to specify data distribution rules. ##### Query a JDBC external table[​](#query-a-jdbc-external-table "Direct link to Query a JDBC external table") Before you query JDBC external tables, you must execute the following statement to enable the Pipeline engine: ```sql set enable_pipeline_engine=true; ``` > Note: If the Pipeline engine is already enabled, you can skip this step. Execute the following statement to query the data in the target database by using JDBC external tables. ```sql select * from JDBC_tbl; ``` StarRocks supports predicate pushdown by pushing down filter conditions to the target table. Executing filter conditions as close as possible to the data source can improve query performance. Currently, StarRocks can push down operators, including the binary comparison operators (`>`, `>=`, `=`, `<`, and `<=`), `IN`, `IS NULL`, and `BETWEEN ... AND ...` . However, StarRocks can not push down functions. ##### Data type mapping[​](#data-type-mapping "Direct link to Data type mapping") Currently, StarRocks can only query data of basic types in the target database, such as NUMBER, STRING, TIME, and DATE. If the ranges of data values in the target database are not supported by StarRocks, the query reports an error. The mapping between the target database and StarRocks varies based on the type of the target database. ###### **MySQL and StarRocks**[​](#mysql-and-starrocks "Direct link to mysql-and-starrocks") | MySQL | StarRocks | | ------------ | --------- | | BOOLEAN | BOOLEAN | | TINYINT | TINYINT | | SMALLINT | SMALLINT | | MEDIUMINTINT | INT | | BIGINT | BIGINT | | FLOAT | FLOAT | | DOUBLE | DOUBLE | | DECIMAL | DECIMAL | | CHAR | CHAR | | VARCHAR | VARCHAR | | DATE | DATE | | DATETIME | DATETIME | ###### **Oracle and StarRocks**[​](#oracle-and-starrocks "Direct link to oracle-and-starrocks") | Oracle | StarRocks | | --------------- | --------- | | CHAR | CHAR | | VARCHARVARCHAR2 | VARCHAR | | DATE | DATE | | SMALLINT | SMALLINT | | INT | INT | | BINARY\_FLOAT | FLOAT | | BINARY\_DOUBLE | DOUBLE | | DATE | DATE | | DATETIME | DATETIME | | NUMBER | DECIMAL | ###### **PostgreSQL and StarRocks**[​](#postgresql-and-starrocks "Direct link to postgresql-and-starrocks") | PostgreSQL | StarRocks | | ------------------- | --------- | | SMALLINTSMALLSERIAL | SMALLINT | | INTEGERSERIAL | INT | | BIGINTBIGSERIAL | BIGINT | | BOOLEAN | BOOLEAN | | REAL | FLOAT | | DOUBLE PRECISION | DOUBLE | | DECIMAL | DECIMAL | | TIMESTAMP | DATETIME | | DATE | DATE | | CHAR | CHAR | | VARCHAR | VARCHAR | | TEXT | VARCHAR | ###### **SQL Server and StarRocks**[​](#sql-server-and-starrocks "Direct link to sql-server-and-starrocks") | SQL Server | StarRocks | | ----------------- | --------- | | BOOLEAN | BOOLEAN | | TINYINT | TINYINT | | SMALLINT | SMALLINT | | INT | INT | | BIGINT | BIGINT | | FLOAT | FLOAT | | REAL | DOUBLE | | DECIMALNUMERIC | DECIMAL | | CHAR | CHAR | | VARCHAR | VARCHAR | | DATE | DATE | | DATETIMEDATETIME2 | DATETIME | ##### Limits[​](#limits "Direct link to Limits") * When you create JDBC external tables, you cannot create indexes on the tables or use PARTITION BY and DISTRIBUTED BY to specify data distribution rules for the tables. * When you query JDBC external tables, StarRocks cannot push down functions to the tables. #### (Deprecated) Elasticsearch external table[​](#deprecated-elasticsearch-external-table "Direct link to (Deprecated) Elasticsearch external table") StarRocks and Elasticsearch are two popular analytics systems. StarRocks is performant in large-scale distributed computing. Elasticsearch is ideal for full-text search. StarRocks combined with Elasticsearch can deliver a more complete OLAP solution. ##### Example of creating an Elasticsearch external table[​](#example-of-creating-an-elasticsearch-external-table "Direct link to Example of creating an Elasticsearch external table") ###### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE EXTERNAL TABLE elastic_search_external_table ( k1 DATE, k2 INT, k3 SMALLINT, k4 VARCHAR(2048), k5 DATETIME ) ENGINE=ELASTICSEARCH PROPERTIES ( "hosts" = "http://192.168.0.1:9200,http://192.168.0.2:9200", "user" = "root", "password" = "root", "index" = "tindex", "type" = "_doc", "es.net.ssl" = "true" ); ``` The following table describes the parameters. | **Parameter** | **Required** | **Default value** | **Description** | | ---------------------- | ------------ | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | hosts | Yes | None | The connection address of the Elasticsearch cluster. You can specify one or more addresses. StarRocks can parse the Elasticsearch version and index shard allocation from this address. StarRocks communicates with your Elasticsearch cluster based on the address returned by the `GET /_nodes/http` API operation. Therefore, the value of the `host` parameter must be the same as the address returned by the `GET /_nodes/http` API operation. Otherwise, BEs may not be able to communicate with your Elasticsearch cluster. | | index | Yes | None | The name of the Elasticsearch index that is created on the table in StarRocks. The name can be an alias. This parameter supports wildcards (`*`). For example, if you set `index` to `hello*`, StarRocks retrieves all indexes whose names start with `hello`. | | user | No | Empty | The username that is used to log in to the Elasticsearch cluster with basic authentication enabled. Make sure you have access to `/*cluster/state/*nodes/http` and the index. | | password | No | Empty | The password that is used to log in to the Elasticsearch cluster. | | type | No | `_doc` | The type of the index. Default value: `_doc`. If you want to query data in Elasticsearch 8 and later versions, you do not need to configure this parameter because the mapping types have been removed in Elasticsearch 8 and later versions. | | es.nodes.wan.only | No | `false` | Specifies whether StarRocks only uses the addresses specified by `hosts` to access the Elasticsearch cluster and fetch data.- `true`: StarRocks only uses the addresses specified by `hosts` to access the Elasticsearch cluster and fetch data and does not sniff data nodes on which the shards of the Elasticsearch index reside. If StarRocks cannot access the addresses of the data nodes inside the Elasticsearch cluster, you need to set this parameter to `true`.
- `false`: StarRocks uses the addresses specified by `host` to sniff data nodes on which the shards of the Elasticsearch cluster indexes reside. After StarRocks generates a query execution plan, the relevant BEs directly access the data nodes inside the Elasticsearch cluster to fetch data from the shards of indexes. If StarRocks can access the addresses of the data nodes inside the Elasticsearch cluster, we recommend that you retain the default value `false`. | | es.net.ssl | No | `false` | Specifies whether the HTTPS protocol can be used to access your Elasticsearch cluster. Only StarRocks 2.4 and later versions support configuring this parameter.- `true`: Both the HTTPS and HTTP protocols can be used to access your Elasticsearch cluster.
- `false`: Only the HTTP protocol can be used to access your Elasticsearch cluster. | | enable\_docvalue\_scan | No | `true` | Specifies whether to obtain the values of the target fields from Elasticsearch columnar storage. In most cases, reading data from columnar storage outperforms reading data from row storage. | | enable\_keyword\_sniff | No | `true` | Specifies whether to sniff TEXT-type fields in Elasticsearch based on KEYWORD-type fields. If this parameter is set to `false`, StarRocks performs matching after tokenization. | ###### Columnar scan for faster queries[​](#columnar-scan-for-faster-queries "Direct link to Columnar scan for faster queries") If you set `enable_docvalue_scan` to `true`, StarRocks follows these rules when it obtains data from Elasticsearch: * **Try and see**: StarRocks automatically checks if columnar storage is enabled for the target fields. If so, StarRocks obtains all values in the target fields from columnar storage. * **Auto-downgrading**: If any one of the target fields is unavailable in columnar storage, StarRocks parses and obtains all values in the target fields from row storage (`_source`). > **NOTE** > > * Columnar storage is unavailable for TEXT-type fields in Elasticsearch. Therefore, if you query fields containing TEXT-type values, StarRocks obtains the values of the fields from `_source`. > * If you query a large number (greater than or equal to 25) of fields, reading field values from `docvalue` does not show noticeable benefits compared with reading field values from `_source`. ###### Sniff KEYWORD-type fields[​](#sniff-keyword-type-fields "Direct link to Sniff KEYWORD-type fields") If you set `enable_keyword_sniff` to `true`, Elasticsearch allows direct data ingestion without an index because it will automatically create an index after ingestion. For STRING-type fields, Elasticsearch will create a field with both TEXT and KEYWORD types. This is how the Multi-Field feature of Elasticsearch works. The mapping is as follows: ```sql "k4": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } ``` For example, to conduct "=" filtering on `k4`, StarRocks on Elasticsearch will convert the filtering operation into an Elasticsearch TermQuery. The original SQL filter is as follows: ```sql k4 = "StarRocks On Elasticsearch" ``` The converted Elasticsearch query DSL is as follows: ```sql "term" : { "k4": "StarRocks On Elasticsearch" } ``` The first field of `k4` is TEXT, and it will be tokenized by the analyzer configured for `k4` (or by the standard analyzer if no analyzer has been configured for `k4`) after data ingestion. As a result, the first field will be tokenized into three terms: `StarRocks`, `On`, and `Elasticsearch`. The details are as follows: ```sql POST /_analyze { "analyzer": "standard", "text": "StarRocks On Elasticsearch" } ``` The tokenization results are as follows: ```sql { "tokens": [ { "token": "starrocks", "start_offset": 0, "end_offset": 5, "type": "", "position": 0 }, { "token": "on", "start_offset": 6, "end_offset": 8, "type": "", "position": 1 }, { "token": "elasticsearch", "start_offset": 9, "end_offset": 11, "type": "", "position": 2 } ] } ``` Suppose you conduct a query as follows: ```sql "term" : { "k4": "StarRocks On Elasticsearch" } ``` There is no term in the dictionary that matches the term `StarRocks On Elasticsearch`, and therefore no result will be returned. However, if you have set `enable_keyword_sniff` to `true`, StarRocks will convert `k4 = "StarRocks On Elasticsearch"` to `k4.keyword = "StarRocks On Elasticsearch"` to match the SQL semantics. The converted `StarRocks On Elasticsearch` query DSL is as follows: ```sql "term" : { "k4.keyword": "StarRocks On Elasticsearch" } ``` `k4.keyword` is of the KEYWORD type. Therefore, the data is written into Elasticsearch as a complete term, allowing for successful matching. ###### Mapping of column data types[​](#mapping-of-column-data-types "Direct link to Mapping of column data types") When you create an external table, you need to specify the data types of columns in the external table based on the data types of columns in the Elasticsearch table. The following table shows the mapping of column data types. | **Elasticsearch** | **StarRocks** | | ----------------- | --------------------------- | | BOOLEAN | BOOLEAN | | BYTE | TINYINT/SMALLINT/INT/BIGINT | | SHORT | SMALLINT/INT/BIGINT | | INTEGER | INT/BIGINT | | LONG | BIGINT | | FLOAT | FLOAT | | DOUBLE | DOUBLE | | KEYWORD | CHAR/VARCHAR | | TEXT | CHAR/VARCHAR | | DATE | DATE/DATETIME | | NESTED | CHAR/VARCHAR | | OBJECT | CHAR/VARCHAR | | ARRAY | ARRAY | > **Note** > > * StarRocks reads the data of the NESTED type by using JSON-related functions. > * Elasticsearch automatically flattens multi-dimensional arrays into one-dimensional arrays. StarRocks does the same. **The support for querying ARRAY data from Elasticsearch is added from v2.5.** ##### Predicate pushdown[​](#predicate-pushdown "Direct link to Predicate pushdown") StarRocks supports predicate pushdown. Filters can be pushed down to Elasticsearch for execution, which improves query performance. The following table lists the operators that support predicate pushdown. | SQL syntax | ES syntax | | -------------- | ---------------------- | | `=` | term query | | `in` | terms query | | `>=, <=, >, <` | range | | `and` | bool.filter | | `or` | bool.should | | `not` | bool.must\_not | | `not in` | bool.must\_not + terms | | `esquery` | ES Query DSL | ##### Examples[​](#examples "Direct link to Examples") The **esquery function** is used to push down queries **that cannot be expressed in SQL** (such as match and geoshape) to Elasticsearch for filtering. The first parameter in the esquery function is used to associate an index. The second parameter is a JSON expression of basic Query DSL, which is enclosed in brackets . **The JSON expression must have but only one root key**, such as match, geo\_shape, or bool. * match query ```sql select * from es_table where esquery(k4, '{ "match": { "k4": "StarRocks on elasticsearch" } }'); ``` * geo-related query ```sql select * from es_table where esquery(k4, '{ "geo_shape": { "location": { "shape": { "type": "envelope", "coordinates": [ [ 13, 53 ], [ 14, 52 ] ] }, "relation": "within" } } }'); ``` * bool query ```sql select * from es_table where esquery(k4, ' { "bool": { "must": [ { "terms": { "k1": [ 11, 12 ] } }, { "terms": { "k2": [ 100 ] } } ] } }'); ``` ##### Usage notes[​](#usage-notes "Direct link to Usage notes") * Elasticsearch earlier than 5.x scans data in a different way than that later than 5.x. Currently, **only versions later than 5.x** are supported. * Elasticsearch clusters with HTTP basic authentication enabled are supported. * Querying data from StarRocks may not be as fast as directly querying data from Elasticsearch, such as count-related queries. The reason is that Elasticsearch directly reads the metadata of target documents without the need to filter the real data, which accelerates the count query. #### (Deprecated) Hive external table[​](#deprecated-hive-external-table "Direct link to (Deprecated) Hive external table") Before using Hive external tables, make sure JDK 1.8 has been installed on your servers. ##### Create a Hive resource[​](#create-a-hive-resource "Direct link to Create a Hive resource") A Hive resource corresponds to a Hive cluster. You must configure the Hive cluster used by StarRocks, such as the Hive metastore address. You must specify the Hive resource that is used by the Hive external table. * Create a Hive resource named hive0. ```sql CREATE EXTERNAL RESOURCE "hive0" PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083" ); ``` * View the resources created in StarRocks. ```sql SHOW RESOURCES; ``` * Delete the resource named `hive0`. ```sql DROP RESOURCE "hive0"; ``` You can modify `hive.metastore.uris` of a Hive resource in StarRocks 2.3 and later versions. For more information, see [ALTER RESOURCE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Resource/ALTER_RESOURCE.md). ##### Create a database[​](#create-a-database-1 "Direct link to Create a database") ```sql CREATE DATABASE hive_test; USE hive_test; ``` ##### Create a Hive external table[​](#create-a-hive-external-table "Direct link to Create a Hive external table") Syntax ```sql CREATE EXTERNAL TABLE table_name ( col_name col_type [NULL | NOT NULL] [COMMENT "comment"] ) ENGINE=HIVE PROPERTIES ( "key" = "value" ); ``` Example: Create the external table `profile_parquet_p7` under the `rawdata` database in the Hive cluster corresponding to the `hive0` resource. ```sql CREATE EXTERNAL TABLE `profile_wos_p7` ( `id` bigint NULL, `first_id` varchar(200) NULL, `second_id` varchar(200) NULL, `p__device_id_list` varchar(200) NULL, `p__is_deleted` bigint NULL, `p_channel` varchar(200) NULL, `p_platform` varchar(200) NULL, `p_source` varchar(200) NULL, `p__city` varchar(200) NULL, `p__province` varchar(200) NULL, `p__update_time` bigint NULL, `p__first_visit_time` bigint NULL, `p__last_seen_time` bigint NULL ) ENGINE=HIVE PROPERTIES ( "resource" = "hive0", "database" = "rawdata", "table" = "profile_parquet_p7" ); ``` Description: * Columns in the external table * The column names must be the same as column names in the Hive table. * The column order **does not need to be** the same as column order in the Hive table. * You can select only **some of the columns in the Hive table**, but you must select all the **partition key columns**. * Partition key columns of an external table do not need to be specified by using `partition by`. They must be defined in the same description list as other columns. You do not need to specify partition information. StarRocks will automatically synchronize this information from the Hive table. * Set `ENGINE` to HIVE. * PROPERTIES: * **hive.resource**: the Hive resource that is used. * **database**: the Hive database. * **table**: the table in Hive. **view** is not supported. * The following table describes the column data type mapping between Hive and StarRocks. | Column type of Hive | Column type of StarRocks | Description | | ------------------- | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | INT/INTEGER | INT | | | BIGINT | BIGINT | | | TIMESTAMP | DATETIME | Precision and time zone information will be lost when you convert TIMESTAMP data into DATETIME data. You need to convert TIMESTAMP data into DATETIME data that does not have the time zone offset based on the time zone in sessionVariable. | | STRING | VARCHAR | | | VARCHAR | VARCHAR | | | CHAR | CHAR | | | DOUBLE | DOUBLE | | | FLOAT | FLOAT | | | DECIMAL | DECIMAL | | | ARRAY | ARRAY | | > Note: > > * Currently, the supported Hive storage formats are Parquet, ORC, and CSV. If the storage format is CSV, quotation marks cannot be used as escape characters. > * The SNAPPY and LZ4 compression formats are supported. > * The maximum length of a Hive string column that can be queried is 1 MB. If a string column exceeds 1 MB, it will be processed as a null column. ##### Use a Hive external table[​](#use-a-hive-external-table "Direct link to Use a Hive external table") Query the total number of rows of `profile_wos_p7`. ```sql select count(*) from profile_wos_p7; ``` ##### Update cached Hive table metadata[​](#update-cached-hive-table-metadata "Direct link to Update cached Hive table metadata") * Hive partition information and the related file information are cached in StarRocks. The cache is refreshed at intervals specified by `hive_meta_cache_refresh_interval_s`. The default value is 7200. * The cached data can also be refreshed manually. 1. If a partition is added or deleted from a table in Hive, you must run the `REFRESH EXTERNAL TABLE hive_t` command to refresh the table metadata cached in StarRocks. `hive_t` is the name of the Hive external table in StarRocks. 2. If data in some Hive partitions is updated, you must refresh the cached data in StarRocks by running the `REFRESH EXTERNAL TABLE hive_t PARTITION ('k1=01/k2=02', 'k1=03/k2=04')` command. `hive_t` is the name of the Hive external table in StarRocks. `'k1=01/k2=02'` and `'k1=03/k2=04'` are the names of Hive partitions whose data is updated. 3. When you run `REFRESH EXTERNAL TABLE hive_t`, StarRocks first checks if the column information of the Hive external table is the same as the column information of the Hive table returned by the Hive Metastore. If the schema of the Hive table changes, such as adding columns or removing columns, StarRocks synchronizes the changes to the Hive external table. After synchronization, the column order of the Hive external table remains the same as the column order of the Hive table, with the partition column being the last column. * When Hive data is stored in the Parquet, ORC, and CSV format, you can synchronize schema changes (such as ADD COLUMN and REPLACE COLUMN) of a Hive table to a Hive external table in StarRocks 2.3 and later versions. ##### Access object storage[​](#access-object-storage "Direct link to Access object storage") * The path of the FE configuration file is `fe/conf`, to which the configuration file can be added if you need to customize the Hadoop cluster. For example: If the HDFS cluster uses a highly available nameservice, you need to put `hdfs-site.xml` under `fe/conf`. If HDFS is configured with ViewFs, you need to put the `core-site.xml` under `fe/conf`. * The path of the BE configuration file is `be/conf`, to which the configuration file can be added if you need to customize the Hadoop cluster. For example, if the HDFS cluster using a highly available nameservice, you need to put `hdfs-site.xml` under `be/conf`. If HDFS is configured with ViewFs, you need to put `core-site.xml` under `be/conf`. * On the machine where BE is located, configure JAVA\_HOME as a JDK environment rather than a JRE environment in the BE **startup script** `bin/start_be.sh`, for example, `export JAVA_HOME = `. You must add this configuration at the beginning of the script and restart the BE for the configuration to take effect. * Configure Kerberos support: 1. To log in with `kinit -kt keytab_path principal` to all FE/BE machines, you need to have access to Hive and HDFS. The kinit command login is only good for a period of time and needs to be put into crontab to be executed regularly. 2. Put `hive-site.xml/core-site.xml/hdfs-site.xml` under `fe/conf`, and put `core-site.xml/hdfs-site.xml` under `be/conf`. 3. Add `-Djava.security.krb5.conf=/etc/krb5.conf` to the value of the `JAVA_OPTS` option in the **$FE\_HOME/conf/fe.conf** file. **/etc/krb5.conf** is the save path of the **krb5.conf** file. You can change the path based on your operating system. 4. Directly add `JAVA_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf"` to the **$BE\_HOME/conf/be.conf** file. **/etc/krb5.conf** is the save path of the **krb5.conf** file. You can change the path based on your operating system. 5. When you add a Hive resource, you must pass in a domain name to `hive.metastore.uris`. In addition, you must add the mapping between Hive/HDFS domain names and IP addresses in the **/etc/hosts** file. * Configure support for AWS S3: Add the following configuration to `fe/conf/core-site.xml` and `be/conf/core-site.xml`. ```xml fs.s3a.access.key ****** fs.s3a.secret.key ****** fs.s3a.endpoint s3.us-west-2.amazonaws.com fs.s3a.connection.maximum 500 ``` 1. `fs.s3a.access.key`: the AWS access key ID. 2. `fs.s3a.secret.key`: the AWS secret key. 3. `fs.s3a.endpoint`: the AWS S3 endpoint to connect to. 4. `fs.s3a.connection.maximum`: the maximum number of concurrent connections from StarRocks to S3. If an error `Timeout waiting for connection from poll` occurs during a query, you can set this parameter to a larger value. #### (Deprecated) Iceberg external table[​](#deprecated-iceberg-external-table "Direct link to (Deprecated) Iceberg external table") From v2.1.0, StarRocks allows you to query data from Apache Iceberg by using external tables. To query data in Iceberg, you need to create an Iceberg external table in StarRocks. When you create the table, you need to establish mapping between the external table and the Iceberg table you want to query. ##### Before you begin[​](#before-you-begin "Direct link to Before you begin") Make sure that StarRocks has permissions to access the metadata service (such as Hive metastore), file system (such as HDFS), and object storage system (such as Amazon S3 and Alibaba Cloud Object Storage Service) used by Apache Iceberg. ##### Precautions[​](#precautions "Direct link to Precautions") * Iceberg external tables can be used to query only the following types of data: * Iceberg v1 tables (Analytic Data Tables). ORC-formatted Iceberg v2 (Row-level Deletes) tables are supported from v3.0 onwards, and Parquet-formatted Iceberg v2 tables are supported from v3.1 onwards. For the differences between Iceberg v1 tables and Iceberg v2 tables, see [Iceberg Table Spec](https://iceberg.apache.org/spec/). * Tables that are compressed in gzip (default format), Zstd, LZ4, or Snappy format. * Files that are stored in Parquet or ORC format. * Iceberg external tables in StarRocks 2.3 and later versions support synchronizing schema changes of an Iceberg table while Iceberg external tables in versions earlier than StarRocks 2.3 do not. If the schema of an Iceberg table changes, you must delete the corresponding external table and create a new one. ##### Procedure[​](#procedure "Direct link to Procedure") ###### Step 1: Create an Iceberg resource[​](#step-1-create-an-iceberg-resource "Direct link to Step 1: Create an Iceberg resource") Before you create an Iceberg external table, you must create an Iceberg resource in StarRocks. The resource is used to manage the Iceberg access information. Additionally, you also need to specify this resource in the statement that is used to create the external table. You can create a resource based on your business requirements: * If the metadata of an Iceberg table is obtained from a Hive metastore, you can create a resource and set the catalog type to `HIVE`. * If the metadata of an Iceberg table is obtained from other services, you need to create a custom catalog. Then create a resource and set the catalog type to `CUSTOM`. ###### Create a resource whose catalog type is `HIVE`[​](#create-a-resource-whose-catalog-type-is-hive "Direct link to create-a-resource-whose-catalog-type-is-hive") For example, create a resource named `iceberg0` and set the catalog type to `HIVE`. ```sql CREATE EXTERNAL RESOURCE "iceberg0" PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "HIVE", "iceberg.catalog.hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083" ); ``` The following table describes the related parameters. | **Parameter** | **Description** | | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | type | The resource type. Set the value to `iceberg`. | | iceberg.catalog.type | The catalog type of the resource. Both Hive catalog and custom catalog are supported. If you specify a Hive catalog, set the value to `HIVE`.If you specify a custom catalog, set the value to `CUSTOM`. | | iceberg.catalog.hive.metastore.uris | The URI of the Hive metastore. The parameter value is in the following format: `thrift://< IP address of Iceberg metadata >:< port number >`. The port number defaults to 9083. Apache Iceberg uses a Hive catalog to access the Hive metastore and then queries the metadata of Iceberg tables. | ###### Create a resource whose catalog type is `CUSTOM`[​](#create-a-resource-whose-catalog-type-is-custom "Direct link to create-a-resource-whose-catalog-type-is-custom") A custom catalog needs to inherit the abstract class BaseMetastoreCatalog, and you need to implement the IcebergCatalog interface. Additionally, the class name of a custom catalog cannot be duplicated with the name of the class that already exists in StarRock. After the catalog is created, package the catalog and its related files, and place them under the **fe/lib** path of each frontend (FE). Then restart each FE. After you complete the preceding operations, you can create a resource whose catalog is a custom catalog. For example, create a resource named `iceberg1` and set the catalog type to `CUSTOM`. ```sql CREATE EXTERNAL RESOURCE "iceberg1" PROPERTIES ( "type" = "iceberg", "iceberg.catalog.type" = "CUSTOM", "iceberg.catalog-impl" = "com.starrocks.IcebergCustomCatalog" ); ``` The following table describes the related parameters. | **Parameter** | **Description** | | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | type | The resource type. Set the value to `iceberg`. | | iceberg.catalog.type | The catalog type of the resource. Both Hive catalog and custom catalog are supported. If you specify a Hive catalog, set the value to `HIVE`. If you specify a custom catalog, set the value to `CUSTOM`. | | iceberg.catalog-impl | The fully qualified class name of the custom catalog. FEs search for the catalog based on this name. If the catalog contains custom configuration items, you must add them to the `PROPERTIES` parameter as key-value pairs when you create an Iceberg external table. | You can modify `hive.metastore.uris` and `iceberg.catalog-impl`of a Iceberg resource in StarRocks 2.3 and later versions. For more information, see [ALTER RESOURCE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Resource/ALTER_RESOURCE.md). ###### View Iceberg resources[​](#view-iceberg-resources "Direct link to View Iceberg resources") ```sql SHOW RESOURCES; ``` ###### Drop an Iceberg resource[​](#drop-an-iceberg-resource "Direct link to Drop an Iceberg resource") For example, drop a resource named `iceberg0`. ```sql DROP RESOURCE "iceberg0"; ``` Dropping an Iceberg resource makes all external tables that reference this resource unavailable. However, the corresponding data in Apache Iceberg is not deleted. If you still need to query the data in Apache Iceberg, create a new resource and a new external table. ###### Step 2: (Optional) Create a database[​](#step-2-optional-create-a-database "Direct link to Step 2: (Optional) Create a database") For example, create a database named `iceberg_test` in StarRocks. ```sql CREATE DATABASE iceberg_test; USE iceberg_test; ``` > Note: The name of the database in StarRocks can be different from the name of the database in Apache Iceberg. ###### Step 3: Create an Iceberg external table[​](#step-3-create-an-iceberg-external-table "Direct link to Step 3: Create an Iceberg external table") For example, create an Iceberg external table named `iceberg_tbl` in the database `iceberg_test`. ```sql CREATE EXTERNAL TABLE `iceberg_tbl` ( `id` bigint NULL, `data` varchar(200) NULL ) ENGINE=ICEBERG PROPERTIES ( "resource" = "iceberg0", "database" = "iceberg", "table" = "iceberg_table" ); ``` The following table describes the related parameters. | **Parameter** | **Description** | | ------------- | -------------------------------------------------------------------- | | ENGINE | The engine name. Set the value to `ICEBERG`. | | resource | The name of the Iceberg resource that the external table references. | | database | The name of the database to which the Iceberg table belongs. | | table | The name of the Iceberg table. | > Note: > > * The name of the external table can be different from the name of the Iceberg table. > > * The column names of the external table must be the same as those in the Iceberg table. The column order of the two tables can be different. If you define configuration items in the custom catalog and want configuration items to take effect when you query data, you can add the configuration items to the `PROPERTIES` parameter as key-value pairs when you create an external table. For example, if you define a configuration item `custom-catalog.properties` in the custom catalog, you can run the following command to create an external table. ```sql CREATE EXTERNAL TABLE `iceberg_tbl` ( `id` bigint NULL, `data` varchar(200) NULL ) ENGINE=ICEBERG PROPERTIES ( "resource" = "iceberg0", "database" = "iceberg", "table" = "iceberg_table", "custom-catalog.properties" = "my_property" ); ``` When you create an external table, you need to specify the data types of columns in the external table based on the data types of columns in the Iceberg table. The following table shows the mapping of column data types. | **Iceberg table** | **Iceberg external table** | | ----------------- | -------------------------- | | BOOLEAN | BOOLEAN | | INT | TINYINT / SMALLINT / INT | | LONG | BIGINT | | FLOAT | FLOAT | | DOUBLE | DOUBLE | | DECIMAL(P, S) | DECIMAL | | DATE | DATE / DATETIME | | TIME | BIGINT | | TIMESTAMP | DATETIME | | STRING | STRING / VARCHAR | | UUID | STRING / VARCHAR | | FIXED(L) | CHAR | | BINARY | VARCHAR | | LIST | ARRAY | StarRocks does not support querying Iceberg data whose data type is TIMESTAMPTZ, STRUCT, and MAP. ###### Step 4: Query the data in Apache Iceberg[​](#step-4-query-the-data-in-apache-iceberg "Direct link to Step 4: Query the data in Apache Iceberg") After an external table is created, you can query the data in Apache Iceberg by using the external table. ```sql select count(*) from iceberg_tbl; ``` #### (Deprecated) Hudi external table[​](#deprecated-hudi-external-table "Direct link to (Deprecated) Hudi external table") From v2.2.0, StarRocks allows you to query data from Hudi data lakes by using Hudi external tables, thus facilitating blazing-fast data lake analytics. This topic describes how to create a Hudi external table in your StarRocks cluster and use the Hudi external table to query data from a Hudi data lake. ##### Before you begin[​](#before-you-begin-1 "Direct link to Before you begin") Make sure that your StarRocks cluster is granted access to the Hive metastore, HDFS cluster, or bucket with which you can register Hudi tables. ##### Precautions[​](#precautions-1 "Direct link to Precautions") * Hudi external tables for Hudi are read-only and can be used only for queries. * StarRocks supports querying Copy on Write and Merge On Read tables (MOR tables are supported from v2.5). For the differences between these two types of tables, see [Table & Query Types](https://hudi.apache.org/docs/table_types/). * StarRocks supports the following two query types of Hudi: Snapshot Queries and Read Optimized Queries (Hudi only supports performing Read Optimized Queries on Merge On Read tables). Incremental Queries are not supported. For more information about the query types of Hudi, see [Table & Query Types](https://hudi.apache.org/docs/next/table_types/#query-types). * StarRocks supports the following compression formats for Hudi files: gzip, zstd, LZ4, and Snappy. The default compression format for Hudi files is gzip. * StarRocks cannot synchronize schema changes from Hudi managed tables. For more information, see [Schema Evolution](https://hudi.apache.org/docs/schema_evolution/). If the schema of a Hudi managed table is changed, you must delete the associated Hudi external table from your StarRocks cluster and then re-create that external table. ##### Procedure[​](#procedure-1 "Direct link to Procedure") ###### Step 1: Create and manage Hudi resources[​](#step-1-create-and-manage-hudi-resources "Direct link to Step 1: Create and manage Hudi resources") You must create Hudi resources in your StarRocks cluster. The Hudi resources are used to manage the Hudi databases and external tables that you create in your StarRocks cluster. ###### Create a Hudi resource[​](#create-a-hudi-resource "Direct link to Create a Hudi resource") Execute the following statement to create a Hudi resource named `hudi0`: ```sql CREATE EXTERNAL RESOURCE "hudi0" PROPERTIES ( "type" = "hudi", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083" ); ``` The following table describes the parameters. | Parameter | Description | | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | type | The type of the Hudi resource. Set the vaue to hudi. | | hive.metastore.uris | The Thrift URI of the Hive metastore to which the Hudi resource connects. After connecting the Hudi resource to a Hive metastore, you can create and manage Hudi tables by using Hive. The Thrift URI is in the `:` format. The default port number is 9083. | From v2.3 onwards, StarRocks allows changing the `hive.metastore.uris` value of a Hudi resource. For more information, see [ALTER RESOURCE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Resource/ALTER_RESOURCE.md). ###### View Hudi resources[​](#view-hudi-resources "Direct link to View Hudi resources") Execute the following statement to view all Hudi resources that are created in your StarRocks cluster: ```sql SHOW RESOURCES; ``` ###### Delete a Hudi resource[​](#delete-a-hudi-resource "Direct link to Delete a Hudi resource") Execute the following statement to delete the Hudi resource named `hudi0`: ```sql DROP RESOURCE "hudi0"; ``` > Note: > > Deleting a Hudi resource causes unavailability of all Hudi external tables that are created by using that Hudi resource. However, the deletion does not affect your data stored in Hudi. If you still want to query your data from Hudi by using StarRocks, you must re-create Hudi resources, Hudi databases, and Hudi external tables in your StarRocks cluster. ###### Step 2: Create Hudi databases[​](#step-2-create-hudi-databases "Direct link to Step 2: Create Hudi databases") Execute the following statement to create and open a Hudi database named `hudi_test` in your StarRocks cluster: ```sql CREATE DATABASE hudi_test; USE hudi_test; ``` > Note: > > The name that you specify for the Hudi database in your StarRocks cluster does not need to be the same as the associated database in Hudi. ###### Step 3: Create Hudi external tables[​](#step-3-create-hudi-external-tables "Direct link to Step 3: Create Hudi external tables") Execute the following statement to create a Hudi external table named `hudi_tbl` in the `hudi_test` Hudi database: ```sql CREATE EXTERNAL TABLE `hudi_tbl` ( `id` bigint NULL, `data` varchar(200) NULL ) ENGINE=HUDI PROPERTIES ( "resource" = "hudi0", "database" = "hudi", "table" = "hudi_table" ); ``` The following table describes the parameters. | Parameter | Description | | --------- | ------------------------------------------------------------------------------------------------- | | ENGINE | The query engine of the Hudi external table. Set the value to `HUDI`. | | resource | The name of the Hudi resource in your StarRocks cluster. | | database | The name of the Hudi database to which the Hudi external table belongs in your StarRocks cluster. | | table | The Hudi managed table with which the Hudi external table is associated. | > Note: > > * The name that you specify for the Hudi external table does not need to be the same as the associated Hudi managed table. > > * The columns in the Hudi external table must have the same names but can be in a different sequence compared to their counterpart columns in the associated Hudi managed table. > > * You can select some or all columns from the associated Hudi managed table and create only the selected columns in the Hudi external table. The following table lists the mapping between the data types supported by Hudi and the data types supported by StarRocks. | Data types supported by Hudi | Data types supported by StarRocks | | ------------------------------- | --------------------------------- | | BOOLEAN | BOOLEAN | | INT | TINYINT/SMALLINT/INT | | DATE | DATE | | TimeMillis/TimeMicros | TIME | | TimestampMillis/TimestampMicros | DATETIME | | LONG | BIGINT | | FLOAT | FLOAT | | DOUBLE | DOUBLE | | STRING | CHAR/VARCHAR | | ARRAY | ARRAY | | DECIMAL | DECIMAL | > **Note** > > StarRocks does not support querying data of the STRUCT or MAP type, nor does it support querying data of the ARRAY type in Merge On Read tables. ###### Step 4: Query data from a Hudi external table[​](#step-4-query-data-from-a-hudi-external-table "Direct link to Step 4: Query data from a Hudi external table") After you create a Hudi external table associated with a specific Hudi managed table, you do not need to load data into the Hudi external table. To query data from Hudi, execute the following statement: ```sql SELECT COUNT(*) FROM hudi_tbl; ``` #### (Deprecated) MySQL external table[​](#deprecated-mysql-external-table "Direct link to (Deprecated) MySQL external table") In the star schema, data is generally divided into dimension tables and fact tables. Dimension tables have less data but involve UPDATE operations. Currently, StarRocks does not support direct UPDATE operations (update can be implemented by using the Unique Key table). In some scenarios, you can store dimension tables in MySQL for direct data read. To query MySQL data, you must create an external table in StarRocks and map it to the table in your MySQL database. You need to specify the MySQL connection information when creating the table. ```sql CREATE EXTERNAL TABLE mysql_external_table ( k1 DATE, k2 INT, k3 SMALLINT, k4 VARCHAR(2048), k5 DATETIME ) ENGINE=mysql PROPERTIES ( "host" = "127.0.0.1", "port" = "3306", "user" = "mysql_user", "password" = "mysql_passwd", "database" = "mysql_db_test", "table" = "mysql_table_test" ); ``` Parameters: * **host**: the connection address of the MySQL database * **port**: the port number of the MySQL database * **user**: the username to log in to MySQL * **password**: the password to log in to MySQL * **database**: the name of the MySQL database * **table**: the name of the table in the MySQL database --- ### Feature Support: Data Lake Analytics From v2.3 onwards, StarRocks supports managing external data sources and analyzing data in data lakes via external catalogs. This document outlines the feature support for external catalogs and the supported version of the features involved. #### Universal features[​](#universal-features "Direct link to Universal features") This section lists the universal features of the External Catalog feature, including storage systems, file readers, credentials, privileges, and Data Cache. ##### External storage systems[​](#external-storage-systems "Direct link to External storage systems") | Storage System | Supported Version | | ----------------------- | ----------------- | | HDFS | v2.3+ | | AWS S3 | v2.3+ | | Microsoft Azure Storage | v3.0+ | | Google GCS | v3.0+ | | Alibaba Cloud OSS | v3.1+ | | Huawei Cloud OBS | v3.1+ | | Tencent Cloud COS | v3.1+ | | Volcengine TOS | v3.1+ | | Kingsoft Cloud KS3 | v3.1+ | | MinIO | v3.1+ | | Ceph S3 | v3.1+ | In addition to the native support for the storage systems listed above, StarRocks also supports the following types of object storage services: * **HDFS-compatible object storage services such as COS Cloud HDFS, OSS-HDFS, and OBS PFS** * **Description**: You need to specify the object storage URI prefix in the BE configuration item `fallback_to_hadoop_fs_list`, and upload the .jar package provided by the cloud vendor to the directory **/lib/hadoop/hdfs/**. Note that you must create the external catalog using the prefix you specified in `fallback_to_hadoop_fs_list`. * **Supported Version(s)**: v3.1.9+, v3.2.4+ * **S3-compatible object storage services other than those listed above** * **Description**: You need to specify the object storage URI prefix in the BE configuration item `s3_compatible_fs_list`. Note that you must create the external catalog using the prefix you specified in `s3_compatible_fs_list`. * **Supported Version(s)**: v3.1.9+, v3.2.4+ ##### Compression formats[​](#compression-formats "Direct link to Compression formats") This section only lists the compression formats supported by each file format. For the file formats supported by each external catalog, please refer to the section on the corresponding external catalog. | File Format | Compression Formats | | ------------ | ----------------------------------------------------------------------------------------------- | | Parquet | NO\_COMPRESSION, SNAPPY, LZ4, ZSTD, GZIP, LZO (v3.1.5+) | | ORC | NO\_COMPRESSION, ZLIB, SNAPPY, LZO, LZ4, ZSTD | | Text | NO\_COMPRESSION, LZO (v3.1.5+) | | Avro | NO\_COMPRESSION (v3.2.1+), DEFLATE (v3.2.1+), SNAPPY (v3.2.1+), BZIP2 (v3.2.1+) | | RCFile | NO\_COMPRESSION (v3.2.1+), DEFLATE (v3.2.1+), SNAPPY (v3.2.1+), GZIP (v3.2.1+) | | SequenceFile | NO\_COMPRESSION (v3.2.1+), DEFLATE (v3.2.1+), SNAPPY (v3.2.1+), BZIP2 (v3.2.1+), GZIP (v3.2.1+) | note The Avro, RCFile, and SequenceFile file formats are read by Java Native Interface (JNI) instead of the native readers within StarRocks. Therefore, the read performance for these file formats may not be as good as that of Parquet and ORC. ##### Management, credential, and access control[​](#management-credential-and-access-control "Direct link to Management, credential, and access control") | Feature | Description | Supported Version(s) | | ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- | | Information Schema | Supports Information Schema for external catalogs. | v3.2+ | | Data lake access control | Supports StarRocks' native RBAC model for external catalogs. You can manage the privileges of databases, tables, and views (currently, Hive views and Iceberge views only) in external catalogs just like those in the default catalog of StarRocks. | v3.0+ | | Reuse external services on Apache Ranger | Supports reusing the external service (such as the Hive Service) on Apache Ranger for access control. | v3.1.9+ | | Kerberos authentication | Supports Kerberos authentication for HDFS or Hive Metastore. | v2.3+ | ##### Data Cache[​](#data-cache "Direct link to Data Cache") | Feature | Description | Supported Version(s) | | -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------- | | Data Cache (Block Cache) | From v2.5 onwards, StarRocks supported the Data Cache feature (then called Block Cache) implemented using CacheLib, which led to limited optimization potential for its extensibility. Starting from v3.0, StarRocks refactored the cache implementation and added new features to Data Cache, resulting in better performance with each subsequent version. | v2.5+ | | Data rebalancing among local disks | Supports data rebalancing strategy to ensure that data skew is controlled under 10%. | v3.2+ | | Replace Block Cache with Data Cache | **Parameter changes**
BE Configurations:- Replace `block_cache_enable` with `datacache_enable`.
- Replace `block_cache_mem_size` with `datacache_mem_size`.
- Replace `block_cache_disk_size` with `datacache_disk_size`.
- Replace `block_cache_disk_path` with `datacache_disk_path`.
- Replace `block_cache_meta_path` with `datacache_meta_path`.
- Replace `block_cache_block_size` with `datacache_block_size`.Session Variables:- Replace `enable_scan_block_cache` with `enable_scan_datacache`. * Replace `enable_populate_block_cache` with `enable_populate_datacache`.After the cluster is upgraded to a version where Data Cache is available, the Block Cache parameters still take effect. The new parameters will override the old ones once Data Cache is enabled. The mixed usage of both groups of parameters is not allowed. Otherwise, some parameters will not take effect. | v3.2+ | | New metrics for API that monitors Data Cache | Supports an individual API that monitors Data Cache including the cache capacity and hits. You can view Data Cache metrics via the interface `http://${BE_HOST}:${BE_HTTP_PORT}/api/datacache/stat`. | v3.2.3+ | | Memory Tracker for Data Cache | Supports Memory Tracker for Data Cache. You can view the memory-related metrics via the interface `http://${BE_HOST}:${BE_HTTP_PORT}/mem_tracker`. | v3.1.8+ | | Data Cache Warmup | By executing CACHE SELECT, you can proactively populate the cache with the desired data from remote storage in advance to prevent the first query from taking too much time fetching the data. CACHE SELECT will not print data or incur calculations. It only fetches data. | v3.3+ | #### Hive Catalog[​](#hive-catalog "Direct link to Hive Catalog") ##### Metadata[​](#metadata "Direct link to Metadata") While executing queries against Hive data through Hive catalogs, StarRocks will cache the table metadata, to reduce the costs from frequent access to the remote storage. This mechanism ensures query performance while maintaining data freshness through the asynchronous refresh and expiration policy. ##### Cached metadata[​](#cached-metadata "Direct link to Cached metadata") StarRocks will cache the following metadata during queries: * **Table- or partition-level metadata** * Content: * Table information: database, table schema, column names, and partition keys * Partition information: partition list, and partition location * Influence: detecting the table existence (whether a table is deleted and/or recreated) * Catalog properties: * `enable_metastore_cache`: Controls whether to enable the metastore cache. Default value: `true`. * `metastore_cache_refresh_interval_sec`: Controls the time interval at which the cached metadata is considered fresh. Default value: `60`. Unit: Seconds. * Location: Metastore (HMS or Glue) * **Partition name list** * Content: List of partition names used to find and prune partitions. Although the partition name list has been collected as the partition information in the above section, there is a separate configuration to enable or disable this feature under certain circumstances. * Influence: detecting the partition existence (whether there is a new partition or a partition is deleted and/or recreated) * Catalog properties: * `enable_cache_list_names`: Controls whether to enable the partition name list cache. Default value: `true`. * `metastore_cache_refresh_interval_sec`: Controls the time interval at which the cached metadata is considered fresh. Default value: `60`. Unit: Seconds. * Location: Metastore (HMS or Glue) * **File-level metadata** * Content: Paths to files under the partition folder. * Influence: Load data into an existing partition. * Catalog properties: * `enable_remote_file_cache`: Controls whether to enable the metadata cache for files in the remote storage. Default value: `true`. * `remote_file_cache_refresh_interval_sec`: Controls the time interval at which the file metadata is considered fresh. Default value: `60`. Unit: Seconds. * `remote_file_cache_memory_ratio`: Controls the ratio of memory that can be used for the file metadata cache. Default value: `0.1` (10%). * Location: Remote storage (HDFS or S3) ##### Asynchronous update policy[​](#asynchronous-update-policy "Direct link to Asynchronous update policy") The following FE configuration item controls the asynchronous metadata update policy: | Configuration item | Default | Description | | -------------------------------------------------------------------- | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | enable\_background\_refresh\_connector\_metadata | `true` in v3.0
`false` in v2.5 | Whether to enable the periodic metadata cache refresh. After it is enabled, StarRocks polls the metastore, and refreshes the cached metadata of the frequently accessed external catalogs to perceive data changes. `true` indicates to enable the Hive metadata cache refresh, and `false` indicates to disable it. This item is an [FE dynamic parameter](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#configure-fe-dynamic-parameters). You can modify it using the [ADMIN SET FRONTEND CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md) command. | | background\_refresh\_metadata\_interval\_millis | `600000` (10 minutes) | The interval between two consecutive metadata cache refreshes. Unit: millisecond. This item is an [FE dynamic parameter](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#configure-fe-dynamic-parameters). You can modify it using the [ADMIN SET FRONTEND CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md) command. | | background\_refresh\_metadata\_time\_secs\_since\_last\_access\_secs | `86400` (24 hours) | The expiration time of a metadata cache refresh task. For the external catalog that has been accessed, if it has not been accessed for more than the specified time, StarRocks stops refreshing its cached metadata. For the external catalog that has not been accessed, StarRocks will not refresh its cached metadata. Unit: second. This item is an [FE dynamic parameter](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#configure-fe-dynamic-parameters). You can modify it using the [ADMIN SET FRONTEND CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md) command. | ##### Metadata cache behavior[​](#metadata-cache-behavior "Direct link to Metadata cache behavior") This section uses the default behavior to explain the metadata behavior during metadata updates and queries. By default, when a table is queried, StarRocks caches the metadata of the table, partitions, and files, and keeps it active for the next 24 hours. During the 24 hours, the system will ensure that the cache is refreshed at least every 10 minutes (note that 10 minutes is the estimated time for a metadata refresh round. If there are excessive external tables that are pending metadata refresh, the overall metadata refresh interval may be longer than 10 minutes). If a table has not been accessed for more than 24 hours, StarRocks discards the associated metadata. In other words, any query you make within 24 hours will, at worst, use metadata from 10 minutes ago. ![Metadata Behavior](/assets/images/hive_metadata_behavior-8456001517ecaccbe7d64c7a5d634e02.png) In details: 1. Suppose the first query involves the partition `P1` of table `A`. StarRocks caches table-level metadata, partition name lists, and file path information under `P1`. The cache is synchronously populated while the query is executed. 2. If a second query is submitted within 60 seconds after the cache is populated, and hits the partition `P1` of table `A`, StarRocks uses the metadata cache directly, and at this point StarRocks considers all cached metadata to be fresh (`metastore_cache_refresh_interval_sec` and `remote_file_cache_refresh_interval_sec` control the time window in which StarRocks considers metadata to be fresh). 3. If a third query is submitted after 90 seconds, and hits the partition `P1` of table `A`, StarRocks will still use the metadata cache directly to complete the query. However, since it has been more than 60 seconds since the last metadata refresh, StarRocks will consider the metadata to be expired. So StarRocks will start an asynchronous refresh for the expired metadata. The asynchronous refresh will not affect the result of the current query because the query will still use the outdated metadata. 4. Because the partition `P1` of table `A` has been queried, it is estimated that the metadata will be refreshed every 10 minutes (controlled by `background_refresh_metadata_interval_millis`) for the next 24 hours (controlled by `background_refresh_metadata_time_secs_since_last_access_secs`). The actual interval between rounds of the metadata refresh also depends on the overall pending refresh tasks within the system. 5. If table `A` is not involved in any query within 24 hours, StarRocks will remove its metadata cache after 24 hours. ##### Best practices[​](#best-practices "Direct link to Best practices") Hive Catalog's support for Hive Metastore (HMS) and AWS Glue mostly overlaps except that the automatic incremental update feature for HMS is not recommended. The default configuration is recommended in most cases. The performance of metadata retrieval largely depends on the performance of the user's HMS or HDFS NameNode. Please consider all factors and base your judgment on test results. * **\[Default and Recommended] Best performance with a tolerance of minute-level data inconsistency** * **Configuration**: You can use the default setting. Data updated within 10 minutes (by default) is not visible. Old data will be returned to queries within this duration. * **Advantage**: Best query performance. * **Disadvantage**: Data inconsistency caused by latency. * **Supported Version(s)**: v2.5.5+ (Disabled by default in v2.5 and enabled by default in v3.0+) * **Instant visibility of newly loaded data (files) without manual refresh** * **Configuration**: Disable the cache for the metadata of the underlying data files by setting the catalog property `enable_remote_file_cache` to `false`. * **Advantage**: Visibility of file changes with no delay. * **Disadvantage**: Lower performance when the file metadata cache is disabled. Each query must access the file list. * **Supported Version(s)**: v2.5.5+ * **Instant visibility of partition changes without manual refresh** * **Configuration**: Disable the cache for the Hive partition names by setting the catalog property `enable_cache_list_names` to `false`. * **Advantage**: Visibility of partition changes with no delay * **Disadvantage**: Lower performance when the partition name cache is disabled. Each query must access the partition list. * **Supported Version(s)**: v2.5.5+ tip If you demand real-time updates on the data changes whilst the performance of your HMS is not optimized, you can enable the cache, disable the automatic incremental update, and manually refresh the metadata (using REFRESH EXTERNAL TABLE) via a scheduling system whenever there is a data change upstream. ##### Storage system[​](#storage-system "Direct link to Storage system") | Feature | Description | Supported Version(s) | | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------- | | Recursive sub-directory listing | Enable recursive sub-directory listing by setting the Catalog property `enable_recursive_listing` to `true`. When recursive listing is enabled, StarRocks will read data from a table and its partitions and from the subdirectories within the physical locations of the table and its partitions. This feature is designed to address the issue of multi-layer nested directories. | v2.5.9+
v3.0.4+ (Disabled by default in v2.5 and v3.0, and enabled by default in v3.1+) | ##### File formats and data types[​](#file-formats-and-data-types "Direct link to File formats and data types") ###### File formats[​](#file-formats "Direct link to File formats") | Feature | Supported File Formats | | ------- | ---------------------------------------------- | | Read | Parquet, ORC, TEXT, Avro, RCFile, SequenceFile | | Sink | Parquet (v3.2+), ORC (v3.3+), TEXT (v3.3+) | ###### Data types[​](#data-types "Direct link to Data types") INTERVAL, BINARY, and UNION types are not supported. TEXT-formatted Hive table does not support MAP and STRUCT types. ###### Table types[​](#table-types "Direct link to Table types") Reading Hive transactional tables is not supported. ##### Hive views[​](#hive-views "Direct link to Hive views") StarRocks supports querying Hive views from v3.1.0 onwards. note While StarRocks executes queries against a Hive view, it will try to parse the definition of the view using the syntax of StarRocks and Trino. An error will be returned if StarRocks cannot parse the definition of the view. There is a possibility that StarRocks failed to parse the Hive views created with functions exclusive to Hive or Spark. ##### Query statistics interfaces[​](#query-statistics-interfaces "Direct link to Query statistics interfaces") | Feature | Supported Version(s) | | ------------------------------------------------------------- | -------------------- | | Supports SHOW CREATE TABLE to view Hive table schema | v3.0+ | | Supports ANALYZE to collect statistics | v3.2+ | | Supports collecting histograms and STRUCT subfield statistics | v3.3+ | ##### Data sinking[​](#data-sinking "Direct link to Data sinking") | Feature | Supported Version(s) | Note | | ---------------------- | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | CREATE DATABASE | v3.2+ | You can choose to specify the location for a database created in Hive or not. If you do not specify the location for the database, you will need to specify the location for the tables created under the database. Otherwise, an error will be returned. If you have specified the location for the database, tables without the location specified will inherit the location of the database. And if you have specified locations for both the database and the table, the table's location will take effect eventually. | | CREATE TABLE | v3.2+ | For both partitioned and non-partitioned tables. | | CREATE TABLE AS SELECT | v3.2+ | | | INSERT INTO/OVERWRITE | v3.2+ | For both partitioned and non-partitioned tables. | | CREATE TABLE LIKE | v3.2.4+ | | | Sink file size | v3.3+ | You can define the maximum size of each data file to be sunk using the session variable `connector_sink_target_max_file_size`. | #### Iceberg Catalog[​](#iceberg-catalog "Direct link to Iceberg Catalog") ##### Metadata[​](#metadata-1 "Direct link to Metadata") While executing queries against Iceberg data through Iceberg catalogs, StarRocks will cache the table metadata, to reduce the costs from frequent access to the remote storage. This mechanism ensures query performance while maintaining data freshness through the asynchronous refresh and expiration policy. ##### Cached metadata[​](#cached-metadata-1 "Direct link to Cached metadata") StarRocks will cache the following metadata during queries: * **Metadata pointer cache** * Content: JSON file of the metadata pointer * Snapshot ID * Location of the manifest lists * Influence: detecting data changes (If a data changes occurs, the snapshot ID will change.) * Catalog properties: * `enable_iceberg_metadata_cache`: Controls whether to enable the Iceberg metadata cache. Default value: `true`. * `iceberg_table_cache_refresh_interval_sec`: Controls the time interval at which the cached metadata is considered fresh. Default value: `60`. Unit: Seconds. * **Metadata cache** * Content: * Manifest for data file path * Manifest for delete file path * Database * Partition (for materialized view rewrite) * Influence: * Will not affect the data freshness for queries because the manifests for data or delete files cannot be changed. * May affect materialized view rewrite, causing queries to miss the materialized view. The partition metadata will be deleted when the snapshot ID is refreshed. Therefore, the new snapshot ID lacks partition metadata, causing misses of the materialized view rewrite. * Catalog properties: * `enable_cache_list_names`: Controls whether to enable the partition name list cache. Default value: `true`. * `metastore_cache_refresh_interval_sec`: Controls the time interval at which the cached metadata is considered fresh. Default value: `60`. Unit: Seconds. * `iceberg_data_file_cache_memory_usage_ratio`: Controls the ratio of memory that can be used for the data file metadata cache. Default value: `0.1` (10%). * `iceberg_delete_file_cache_memory_usage_ratio`: Controls the ratio of memory that can be used for the delete file metadata cache. Default value: `0.1` (10%). ##### Asynchronous update policy[​](#asynchronous-update-policy-1 "Direct link to Asynchronous update policy") The following FE configuration item controls the asynchronous metadata update policy: | Configuration item | Default | Description | | -------------------------------------------------------------------- | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | enable\_background\_refresh\_connector\_metadata | `true` in v3.0
`false` in v2.5 | Whether to enable the periodic metadata cache refresh. After it is enabled, StarRocks polls the metastore, and refreshes the cached metadata of the frequently accessed external catalogs to perceive data changes. `true` indicates to enable the Hive metadata cache refresh, and `false` indicates to disable it. This item is an [FE dynamic parameter](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#configure-fe-dynamic-parameters). You can modify it using the [ADMIN SET FRONTEND CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md) command. | | background\_refresh\_metadata\_interval\_millis | `600000` (10 minutes) | The interval between two consecutive metadata cache refreshes. Unit: millisecond. This item is an [FE dynamic parameter](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#configure-fe-dynamic-parameters). You can modify it using the [ADMIN SET FRONTEND CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md) command. | | background\_refresh\_metadata\_time\_secs\_since\_last\_access\_secs | `86400` (24 hours) | The expiration time of a metadata cache refresh task. For the external catalog that has been accessed, if it has not been accessed for more than the specified time, StarRocks stops refreshing its cached metadata. For the external catalog that has not been accessed, StarRocks will not refresh its cached metadata. Unit: second. This item is an [FE dynamic parameter](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#configure-fe-dynamic-parameters). You can modify it using the [ADMIN SET FRONTEND CONFIG](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md) command. | ##### Metadata cache behavior[​](#metadata-cache-behavior-1 "Direct link to Metadata cache behavior") This section uses the default behavior to explain the metadata behavior during metadata updates and queries. By default, when a table is queried, StarRocks caches the metadata of the table, and keeps it active for the next 24 hours. During the 24 hours, the system will ensure that the cache is refreshed at least every 10 minutes (note that 10 minutes is the estimated time for a metadata refresh round. If there are excessive external tables that are pending metadata refresh, the overall metadata refresh interval may be longer than 10 minutes). If a table has not been accessed for more than 24 hours, StarRocks discards the associated metadata. In other words, any query you make within 24 hours will, at worst, use metadata from 10 minutes ago. ![Metadata Behavior](/assets/images/iceberg_metadata_behavior-7544cc5cb78ab4260b20ac0b21f7350c.png) In details: 1. Suppose the first query involves the table `A`. StarRocks caches its latest snapshot and metadata. The cache is synchronously populated while the query is executed. 2. If a second query is submitted within 60 seconds after the cache is populated, and hits the table `A`, StarRocks uses the metadata cache directly, and at this point StarRocks considers all cached metadata to be fresh (`iceberg_table_cache_refresh_interval_sec` controls the time window in which StarRocks considers metadata to be fresh). 3. If a third query is submitted after 90 seconds, and hits the table `A`, StarRocks will still use the metadata cache directly to complete the query. However, since it has been more than 60 seconds since the last metadata refresh, StarRocks will consider the metadata to be expired. So StarRocks will start an asynchronous refresh for the expired metadata. The asynchronous refresh will not affect the result of the current query because the query will still use the outdated metadata. 4. Because the table `A` has been queried, it is estimated that the metadata will be refreshed every 10 minutes (controlled by `background_refresh_metadata_interval_millis`) for the next 24 hours (controlled by `background_refresh_metadata_time_secs_since_last_access_secs`). The actual interval between rounds of the metadata refresh also depends on the overall pending refresh tasks within the system. 5. If table `A` is not involved in any query within 24 hours, StarRocks will remove its metadata cache after 24 hours. ##### Best practices[​](#best-practices-1 "Direct link to Best practices") Iceberg Catalog supports HMS, Glue, and Tabular as its metastore. The default configuration is recommended in most cases. Please note that the default value of the session variable `enable_iceberg_metadata_cache` has been changed to accommodate different scenarios: * From v3.2.1 to v3.2.3, this parameter is set to `true` by default, regardless of what metastore service is used. * In v3.2.4 and later, if the Iceberg cluster uses AWS Glue as metastore, this parameter still defaults to `true`. However, if the Iceberg cluster uses other metastore services such as Hive metastore, this parameter defaults to `false`. * From v3.3.0 onwards, the default value of this parameter is set to `true` again because StarRocks supports the new Iceberg metadata framework. Iceberg Catalog and Hive Catalog now use the same metadata polling mechanism and FE configuration item `background_refresh_metadata_interval_millis`. | Feature | Supported Version(s) | | ----------------------------------------------------------------------------------------------------- | -------------------- | | Distributed metadata plan (Recommended for scenarios with a large volume of metadata) | v3.3+ | | Manifest Cache (Recommended for scenarios with a small volume of metadata but high demand on latency) | v3.3+ | From v3.3.0 onwards, StarRocks supports the metadata reading and caching policies described above. The system will automatically adjust the choice of policy according to the machines in your cluster. Usually, you do not need to change it. Since metadata caching is enabled, it is possible that metadata freshness may be compromised due to performance considerations. Therefore, you can adjust it according to your specific query requirements: * **\[Default and recommended] Optimal performance with tolerance of minute-level data inconsistencies** * **Setting**: No additional setting is required. By default, data updated within 10 minutes is not visible. During this time, queries will return old data. * **Advantages**: Best query performance. * **Disadvantage**: data inconsistency caused by delays. * **New data files generated by import and partition additions or deletions are immediately visible, and no manual refresh is required** * **Setting**: Set the Catalog property `iceberg_meta_cache_ttl_sec` to `0` to allow StarRocks to fetch a new snapshot for each query. * **Advantages**: File and partition changes are visible without delay. * **Disadvantage**: Lower performance due to the snapshot fetching behavior for each query. ##### File formats[​](#file-formats-1 "Direct link to File formats") | Feature | Supported File Formats | | ------- | ---------------------- | | Read | Parquet, ORC | | Sink | Parquet | * Both Parquet-formatted and ORC-formatted Iceberg V1 tables support position deletes and equality deletes. * ORC-formatted Iceberg V2 tables support position deletes from v3.0.0, and Parquet-formatted ones support position deletes from v3.1.0. * ORC-formatted Iceberg V2 tables support equality deletes from v3.1.8 and v3.2.3, and Parquet-formatted ones support equality deletes from v3.2.5. ##### Iceberg views[​](#iceberg-views "Direct link to Iceberg views") StarRocks supports Iceberg views on REST from v3.3.2 and on Hive from v3.4.1. Currently, only Iceberg views created through StarRocks are supported. Starting with v3.5, adding StarRocks syntax style definitions to existing Iceberg views is supported. ##### Query statistics interfaces[​](#query-statistics-interfaces-1 "Direct link to Query statistics interfaces") | Feature | Supported Version(s) | | ------------------------------------------------------------- | -------------------- | | Supports SHOW CREATE TABLE to view Iceberg table schema | v3.0+ | | Supports ANALYZE to collect statistics | v3.2+ | | Supports collecting histograms and STRUCT subfield statistics | v3.3+ | ##### Data sinking[​](#data-sinking-1 "Direct link to Data sinking") | Feature | Supported Version(s) | Note | | ---------------------- | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | CREATE DATABASE | v3.1+ | You can choose to specify the location for a database created in Iceberg or not. If you do not specify the location for the database, you will need to specify the location for the tables created under the database. Otherwise, an error will be returned. If you have specified the location for the database, tables without the location specified will inherit the location of the database. And if you have specified locations for both the database and the table, the table's location will take effect eventually. | | CREATE TABLE | v3.1+ | Supports partitioned and non-partitioned tables. Starting from v4.0, supports creating tables with hidden partitions. | | CREATE TABLE AS SELECT | v3.1+ | | | INSERT INTO/OVERWRITE | v3.1+ | For both partitioned and non-partitioned tables. | ##### Miscellaneous supports[​](#miscellaneous-supports "Direct link to Miscellaneous supports") | Feature | Supported Version(s) | | -------------------------------------------------------------------------------------------- | ---------------------------------- | | Supports reading TIMESTAMP-type partition formats `yyyy-MM-ddTHH:mm` and `yyyy-MM-dd HH:mm`. | v2.5.19+
v3.1.9+
v3.2.3+ | | Supports Iceberg metadata table | v3.4.1+ | | Supports Iceberg TimeTravel | v3.4.0+ | #### Hudi Catalog[​](#hudi-catalog "Direct link to Hudi Catalog") * StarRocks supports querying the Parquet-formatted data in Hudi, and supports SNAPPY, LZ4, ZSTD, GZIP, and NO\_COMPRESSION compression formats for Parquet files. * StarRocks fully supports Hudi's Copy On Write (COW) tables and Merge On Read (MOR) tables. * StarRocks supports SHOW CREATE TABLE to view Hudi table schema from v3.0.0 onwards. * StarRocks v3.5.0 supports Hudi 0.15.0. #### Delta Lake Catalog[​](#delta-lake-catalog "Direct link to Delta Lake Catalog") * StarRocks supports querying the Parquet-formatted data in Delta Lake, and supports SNAPPY, LZ4, ZSTD, GZIP, and NO\_COMPRESSION compression formats for Parquet files. * StarRocks does not support querying the MAP-type and STRUCT-type data in Delta Lake. * StarRocks supports SHOW CREATE TABLE to view Delta Lake table schema from v3.0.0 onwards. * Currently, Delta Lake catalogs support the following table features: * V2 Checkpoint (From v3.3.0 onwards) * Timestamp without Timezone (From v3.3.1 onwards) * Column mapping (From v3.3.6 onwards) * Deletion Vector (From v3.4.1 onwards) #### JDBC Catalog[​](#jdbc-catalog "Direct link to JDBC Catalog") | Catalog type | Supported Version(s) | | ------------ | -------------------- | | MySQL | v3.0+ | | PostgreSQL | v3.0+ | | ClickHouse | v3.3+ | | Oracle | v3.2.9+ | | SQL Server | v3.2.9+ | ##### MySQL[​](#mysql "Direct link to MySQL") | Feature | Supported Version(s) | | -------------- | -------------------- | | Metadata cache | v3.3+ | ###### Data type correspondance[​](#data-type-correspondance "Direct link to Data type correspondance") | MySQL | StarRocks | Supported Version(s) | | ----------------- | ------------------- | -------------------- | | BOOLEAN | BOOLEAN | v2.3+ | | BIT | BOOLEAN | v2.3+ | | SIGNED TINYINT | TINYINT | v2.3+ | | UNSIGNED TINYINT | SMALLINT | v3.0.6+
v3.1.2+ | | SIGNED SMALLINT | SMALLINT | v2.3+ | | UNSIGNED SMALLINT | INT | v3.0.6+
v3.1.2+ | | SIGNED INTEGER | INT | v2.3+ | | UNSIGNED INTEGER | BIGINT | v3.0.6+
v3.1.2+ | | SIGNED BIGINT | BIGINT | v2.3+ | | UNSIGNED BIGINT | LARGEINT | v3.0.6+
v3.1.2+ | | FLOAT | FLOAT | v2.3+ | | REAL | FLOAT | v3.0.1+ | | DOUBLE | DOUBLE | v2.3+ | | DECIMAL | DECIMAL32 | v2.3+ | | CHAR | VARCHAR(columnsize) | v2.3+ | | VARCHAR | VARCHAR | v2.3+ | | TEXT | VARCHAR(columnsize) | v3.0.1+ | | DATE | DATE | v2.3+ | | TIME | TIME | v3.1.9+
v3.2.4+ | | TIMESTAMP | DATETIME | v2.3+ | ##### PostgreSQL[​](#postgresql "Direct link to PostgreSQL") ###### Data type correspondance[​](#data-type-correspondance-1 "Direct link to Data type correspondance") | PGSQL | StarRocks | Supported Version(s) | | --------- | ------------------- | -------------------- | | BIT | BOOLEAN | v2.3+ | | SMALLINT | SMALLINT | v2.3+ | | INTEGER | INT | v2.3+ | | BIGINT | BIGINT | v2.3+ | | REAL | FLOAT | v2.3+ | | DOUBLE | DOUBLE | v2.3+ | | NUMERIC | DECIMAL32 | v2.3+ | | CHAR | VARCHAR(columnsize) | v2.3+ | | VARCHAR | VARCHAR | v2.3+ | | TEXT | VARCHAR(columnsize) | v2.3+ | | DATE | DATE | v2.3+ | | TIMESTAMP | DATETIME | v2.3+ | | UUID | VARBINARY | v3.5.3+ | ##### ClickHouse[​](#clickhouse "Direct link to ClickHouse") Supported from v3.3.0 onwards. ##### Oracle[​](#oracle "Direct link to Oracle") Supported from v3.2.9 onwards. ##### SQL Server[​](#sql-server "Direct link to SQL Server") Supported from v3.2.9 onwards. #### Elasticsearch Catalog[​](#elasticsearch-catalog "Direct link to Elasticsearch Catalog") Elasticsearch Catalog is supported from v3.1.0 onwards. #### Paimon Catalog[​](#paimon-catalog "Direct link to Paimon Catalog") Paimon Catalog is supported from v3.1.0 onwards. #### MaxCompute Catalog[​](#maxcompute-catalog "Direct link to MaxCompute Catalog") MaxCompute Catalog is supported from v3.3.0 onwards. #### Kudu Catalog[​](#kudu-catalog "Direct link to Kudu Catalog") Kudu Catalog is supported from v3.3.0 onwards. --- ### File external table File external table is a special type of external table. It allows you to directly query Parquet and ORC data files in external storage systems without loading data into StarRocks. In addition, file external tables do not rely on a metastore. In the current version, StarRocks supports the following external storage systems: HDFS, Amazon S3, and other S3-compatible storage systems. This feature is supported from StarRocks v2.5. note * From v3.1 onwards, StarRocks supports directly loading data from files on cloud storage using the [INSERT](https://docs.starrocks.io/docs/loading/InsertInto.md#insert-data-directly-from-files-in-an-external-source-using-files) command and the [FILES](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md) function, thereby you do not need to create an external catalog or file external table first. Besides, FILES() can automatically infer the table schema of the files, greatly simplifying the process of data loading. * The File External Table feature was designed to help with loading data into StarRocks, NOT to perform efficient queries against external systems as a normal operation. A more performant solution would be to load the data into StarRocks. #### Limits[​](#limits "Direct link to Limits") * File external tables must be created in databases within the [default\_catalog](https://docs.starrocks.io/docs/data_source/catalog/default_catalog.md). You can run [SHOW CATALOGS](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SHOW_CATALOGS.md) to query catalogs created in the cluster. * Only Parquet, ORC, Avro, RCFile, and SequenceFile data files are supported. * You can only use file external tables to query data in the target data file. Data write operations such as INSERT, DELETE, and DROP are not supported. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before you create a file external table, you must configure your StarRocks cluster so that StarRocks can access the external storage system where the target data file is stored. The configurations required for a file external table are the same as those required for a Hive catalog, except that you do not need to configure a metastore. See [Hive catalog - Integration preparations](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md#integration-preparations) for more information about configurations. #### Create a database (Optional)[​](#create-a-database-optional "Direct link to Create a database (Optional)") After connecting to your StarRocks cluster, you can create a file external table in an existing database or create a new database to manage file external tables. To query existing databases in the cluster, run [SHOW DATABASES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATABASES.md). Then you can run `USE ` to switch to the target database. The syntax for creating a database is as follows. ```sql CREATE DATABASE [IF NOT EXISTS] ``` #### Create a file external table[​](#create-a-file-external-table "Direct link to Create a file external table") After accessing the target database, you can create a file external table in this database. ##### Syntax[​](#syntax "Direct link to Syntax") ```sql CREATE EXTERNAL TABLE ( [NULL | NOT NULL] [COMMENT ""] ) ENGINE=file COMMENT ["comment"] PROPERTIES ( FileLayoutParams, StorageCredentialParams ) ``` ##### Parameters[​](#parameters "Direct link to Parameters") | Parameter | Required | Description | | ---------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | table\_name | Yes | The name of the file external table. The naming conventions are as follows:- The name can contain letters, digits (0-9), and underscores (\_). It must start with a letter.
- The name cannot exceed 64 characters in length. | | col\_name | Yes | The column name in the file external table. The column names in the file external table must be the same as those in the target data file but are not case-sensitive. The order of columns in the file external table can be different from that in the target data file. | | col\_type | Yes | The column type in the file external table. You need to specify this parameter based on the column type in the target data file. For more information, see [Mapping of column types](#mapping-of-column-types). | | NULL | NOT NULL | No | Whether the column in the file external table is allowed to be NULL. - NULL: NULL is allowed.
- NOT NULL: NULL is not allowed.You must specify this modifier based on the following rules:- If this parameter is not specified for the columns in the target data file, you can choose not to specify it for the columns in the file external table or specify NULL for the columns in the file external table. * If NULL is specified for the columns in the target data file, you can choose not to specify this parameter for the columns in the file external table or specify NULL for the columns in the file external table. * If NOT NULL is specified for the columns in the target data file, you must also specify NOT NULL for the columns in the file external table. | | comment | No | The comment of column in the file external table. | | ENGINE | Yes | The type of engine. Set the value to file. | | comment | No | The description of the file external table. | | PROPERTIES | Yes | - `FileLayoutParams`: specifies the path and format of the target file. This property is required.
- `StorageCredentialParams`: specifies the authentication information required for accessing object storage systems. This property is required only for AWS S3 and other S3-compatible storage systems. | ###### FileLayoutParams[​](#filelayoutparams "Direct link to FileLayoutParams") A set of parameters for accessing the target data file. ```sql "path" = "", "format" = "" "enable_recursive_listing" = "{ true | false }" "enable_wildcards" = "{ true | false }" ``` | Parameter | Required | Description | | -------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | path | Yes | The path of the data file. - If the data file is stored in HDFS, the path format is `hdfs://:/`. The default port number is 8020. If you use the default port, you do not need to specify it.
- If the data file is stored in AWS S3 or other S3-compatible storage system, the path format is `s3:////`. Note the following rules when you enter the path:- If you want to access all files in a path, end this parameter with a slash (`/`), such as `hdfs://x.x.x.x/user/hive/warehouse/array2d_parq/data/`. When you run a query, StarRocks traverses all data files under the path. It does not traverse data files by using recursion. * If you want to access a single file, enter a path that directly points to this file, such as `hdfs://x.x.x.x/user/hive/warehouse/array2d_parq/data`. When you run a query, StarRocks only scans this data file. | | format | Yes | The format of the data file. Valid values: `parquet`, `orc`, `avro`, `rctext` or `rcbinary`, and `sequence`. | | enable\_recursive\_listing | No | Specifies whether to recursively transverse all files under the current path. Default value: `true`. The value `true` specifies to recursively list subdirectories, and the value `false` specifies to ignore subdirectories. | | enable\_wildcards | No | Whether to support using wildcards (`*`) in `path`. Default value: `false`. For example, `2024-07-*` is to match all files with the `2024-07-` prefix. This parameter is supported from v3.1.9. | ###### StorageCredentialParams (Optional)[​](#storagecredentialparams-optional "Direct link to StorageCredentialParams (Optional)") A set of parameters about how StarRocks integrates with the target storage system. This parameter set is **optional**. You need to configure `StorageCredentialParams` only when the target storage system is AWS S3 or other S3-compatible storage. For other storage systems, you can ignore `StorageCredentialParams`. ###### AWS S3[​](#aws-s3 "Direct link to AWS S3") If you need to access a data file stored in AWS S3, configure the following authentication parameters in `StorageCredentialParams`. * If you choose the instance profile-based authentication method, configure `StorageCredentialParams` as follows: ```javascript "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "" ``` * If you choose the assumed role-based authentication method, configure `StorageCredentialParams` as follows: ```javascript "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "", "aws.s3.region" = "" ``` * If you choose the IAM user-based authentication method, configure `StorageCredentialParams` as follows: ```javascript "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "" ``` | Parameter name | Required | Description | | ----------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | aws.s3.use\_instance\_profile | Yes | Specifies whether to enable the instance profile-based authentication method and the assumed role-based authentication method when you access AWS S3. Valid values: `true` and `false`. Default value: `false`. | | aws.s3.iam\_role\_arn | Yes | The ARN of the IAM role that has privileges on your AWS S3 bucket.
If you use the assumed role-based authentication method to access AWS S3, you must specify this parameter. Then, StarRocks will assume this role when it accesses the target data file. | | aws.s3.region | Yes | The region in which your AWS S3 bucket resides. Example: us-west-1. | | aws.s3.access\_key | No | The access key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter. | | aws.s3.secret\_key | No | The secret key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter. | For information about how to choose an authentication method for accessing AWS S3 and how to configure an access control policy in the AWS IAM Console, see [Authentication parameters for accessing AWS S3](https://docs.starrocks.io/docs/integrations/authenticate_to_aws_resources.md#authentication-parameters-for-accessing-aws-s3). ###### S3-compatible storage[​](#s3-compatible-storage "Direct link to S3-compatible storage") If you need to access an S3-compatible storage system, such as MinIO, configure `StorageCredentialParams` as follows to ensure a successful integration: ```sql "aws.s3.enable_ssl" = "false", "aws.s3.enable_path_style_access" = "true", "aws.s3.endpoint" = "", "aws.s3.access_key" = "", "aws.s3.secret_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | Parameter | Required | Description | | ---------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | aws.s3.enable\_ssl | Yes | Specifies whether to enable SSL connection.
Valid values: `true` and `false`. Default value: `true`. | | aws.s3.enable\_path\_style\_access | Yes | Specifies whether to enable path-style access.
Valid values: `true` and `false`. Default value: `false`. For MinIO, you must set the value to `true`.
Path-style URLs use the following format: `https://s3..amazonaws.com//`. For example, if you create a bucket named `DOC-EXAMPLE-BUCKET1` in the US West (Oregon) Region, and you want to access the `alice.jpg` object in that bucket, you can use the following path-style URL: `https://s3.us-west-2.amazonaws.com/DOC-EXAMPLE-BUCKET1/alice.jpg`. | | aws.s3.endpoint | Yes | The endpoint used to connect to an S3-compatible storage system instead of AWS S3. | | aws.s3.access\_key | Yes | The access key of your IAM user. | | aws.s3.secret\_key | Yes | The secret key of your IAM user. | ###### Mapping of column types[​](#mapping-of-column-types "Direct link to Mapping of column types") The following table provides the mapping of column types between the target data file and the file external table. | Data file | File external table | | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | INT/INTEGER | INT | | BIGINT | BIGINT | | TIMESTAMP | DATETIME.
Note that TIMESTAMP is converted to DATETIME without a time zone based on the time zone setting of the current session and loses some of its precision. | | STRING | STRING | | VARCHAR | VARCHAR | | CHAR | CHAR | | DOUBLE | DOUBLE | | FLOAT | FLOAT | | DECIMAL | DECIMAL | | BOOLEAN | BOOLEAN | | ARRAY | ARRAY | | MAP | MAP | | STRUCT | STRUCT | ##### Examples[​](#examples "Direct link to Examples") ###### HDFS[​](#hdfs "Direct link to HDFS") Create a file external table named `t0` to query Parquet data files stored in an HDFS path. ```sql USE db_example; CREATE EXTERNAL TABLE t0 ( name string, id int ) ENGINE=file PROPERTIES ( "path"="hdfs://x.x.x.x:8020/user/hive/warehouse/person_parq/", "format"="parquet" ); ``` ###### AWS S3[​](#aws-s3-1 "Direct link to AWS S3") Example 1: Create a file external table and use **instance profile** to access **a single Parquet file** in AWS S3. ```sql USE db_example; CREATE EXTERNAL TABLE table_1 ( name string, id int ) ENGINE=file PROPERTIES ( "path" = "s3://bucket-test/folder1/raw_0.parquet", "format" = "parquet", "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "us-west-2" ); ``` Example 2: Create a file external table and use **assumed role** to access **all the ORC files** under the target file path in AWS S3. ```sql USE db_example; CREATE EXTERNAL TABLE table_1 ( name string, id int ) ENGINE=file PROPERTIES ( "path" = "s3://bucket-test/folder1/", "format" = "orc", "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::51234343412:role/role_name_in_aws_iam", "aws.s3.region" = "us-west-2" ); ``` Example 3: Create a file external table and use **IAM user** to access **all the ORC files** under the file path in AWS S3. ```sql USE db_example; CREATE EXTERNAL TABLE table_1 ( name string, id int ) ENGINE=file PROPERTIES ( "path" = "s3://bucket-test/folder1/", "format" = "orc", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "us-west-2" ); ``` #### Query a file external table[​](#query-a-file-external-table "Direct link to Query a file external table") Syntax: ```sql SELECT FROM ``` For example, to query data from the file external table `t0` created in [Examples - HDFS](#examples), run the following command: ```plain SELECT * FROM t0; +--------+------+ | name | id | +--------+------+ | jack | 2 | | lily | 1 | +--------+------+ 2 rows in set (0.08 sec) ``` #### Manage file external tables[​](#manage-file-external-tables "Direct link to Manage file external tables") You can view the schema of the table using [DESC](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DESCRIBE.md) or drop the table using [DROP TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DROP_TABLE.md). --- ### Iceberg Lakehouse tutorial ### Apache Iceberg Lakehouse This guide will get you up and running with Apache Iceberg™ using StarRocks™, including sample code to highlight some powerful features. ##### Docker-Compose[​](#docker-compose "Direct link to Docker-Compose") The fastest way to get started is to use a docker-compose file that uses the `starrocks/fe-ubuntu` and `starrocks/be-ubuntu` images which contain a local StarRocks cluster with a configured Iceberg catalog. To use this, you'll need to install the Docker CLI. Once you have Docker installed, save the yaml below into a file named docker-compose.yml: ```yml services: starrocks-fe: image: starrocks/fe-ubuntu:4.0-latest hostname: starrocks-fe container_name: starrocks-fe user: root command: | bash /opt/starrocks/fe/bin/start_fe.sh --host_type FQDN ports: - 8030:8030 - 9020:9020 - 9030:9030 networks: iceberg_net: environment: - AWS_ACCESS_KEY_ID=admin - AWS_SECRET_ACCESS_KEY=password - AWS_REGION=us-east-1 healthcheck: test: 'mysql -u root -h starrocks-fe -P 9030 -e "SHOW FRONTENDS\G" |grep "Alive: true"' interval: 10s timeout: 5s retries: 3 starrocks-be: image: starrocks/be-ubuntu:4.0-latest command: - /bin/bash - -c - | ulimit -n 65535; echo "# Enable data cache" >> /opt/starrocks/be/conf/be.conf echo "block_cache_enable = true" >> /opt/starrocks/be/conf/be.conf echo "block_cache_mem_size = 536870912" >> /opt/starrocks/be/conf/be.conf echo "block_cache_disk_size = 1073741824" >> /opt/starrocks/be/conf/be.conf sleep 15s mysql --connect-timeout 2 -h starrocks-fe -P 9030 -u root -e "ALTER SYSTEM ADD BACKEND \"starrocks-be:9050\";" bash /opt/starrocks/be/bin/start_be.sh ports: - 8040:8040 hostname: starrocks-be container_name: starrocks-be user: root depends_on: starrocks-fe: condition: service_healthy healthcheck: test: 'mysql -u root -h starrocks-fe -P 9030 -e "SHOW BACKENDS\G" |grep "Alive: true"' interval: 10s timeout: 5s retries: 3 networks: iceberg_net: environment: - HOST_TYPE=FQDN - AWS_EC2_METADATA_DISABLED=true rest: image: apache/iceberg-rest-fixture container_name: iceberg-rest networks: iceberg_net: aliases: - iceberg-rest.minio ports: - 8181:8181 environment: - AWS_ACCESS_KEY_ID=admin - AWS_SECRET_ACCESS_KEY=password - AWS_REGION=us-east-1 - CATALOG_WAREHOUSE=s3://warehouse/ - CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO - CATALOG_S3_ENDPOINT=http://minio:9000 minio: image: minio/minio:RELEASE.2024-10-29T16-01-48Z container_name: minio environment: - MINIO_ROOT_USER=admin - MINIO_ROOT_PASSWORD=password - MINIO_DOMAIN=minio networks: iceberg_net: aliases: - warehouse.minio ports: - 9001:9001 - 9000:9000 command: ["server", "/data", "--console-address", ":9001"] mc: depends_on: - minio image: minio/mc:RELEASE.2024-10-29T15-34-59Z container_name: mc networks: iceberg_net: environment: - AWS_ACCESS_KEY_ID=admin - AWS_SECRET_ACCESS_KEY=password - AWS_REGION=us-east-1 entrypoint: > /bin/sh -c " until (/usr/bin/mc config host add minio http://minio:9000 admin password) do echo '...waiting...' && sleep 1; done; /usr/bin/mc rm -r --force minio/warehouse; /usr/bin/mc mb minio/warehouse; /usr/bin/mc policy set public minio/warehouse; tail -f /dev/null " networks: iceberg_net: ``` Next, start up the docker containers with this command: ```plain docker compose up --detach --wait --wait-timeout 400 ``` You can then run any of the following commands to start a StarRocks session. ```bash docker exec -it starrocks-fe \ mysql -P 9030 -h 127.0.0.1 -u root --prompt="StarRocks > " ``` ##### Adding and Using a Catalog[​](#adding-and-using-a-catalog "Direct link to Adding and Using a Catalog") ```sql CREATE EXTERNAL CATALOG 'demo' COMMENT "External catalog to Apache Iceberg on MinIO" PROPERTIES ( "type"="iceberg", "iceberg.catalog.type"="rest", "iceberg.catalog.uri"="http://iceberg-rest:8181", "iceberg.catalog.warehouse"="warehouse", "aws.s3.access_key"="admin", "aws.s3.secret_key"="password", "aws.s3.endpoint"="http://minio:9000", "aws.s3.enable_path_style_access"="true" ); ``` ```sql SHOW CATALOGS\G ``` ```sql *************************** 1. row *************************** Catalog: default_catalog Type: Internal Comment: An internal catalog contains this cluster's self-managed tables. *************************** 2. row *************************** Catalog: demo Type: Iceberg Comment: External catalog to Apache Iceberg on MinIO 2 rows in set (0.00 sec) ``` ```sql SET CATALOG demo; ``` ##### Creating and using a database[​](#creating-and-using-a-database "Direct link to Creating and using a database") ```sql CREATE DATABASE nyc; ``` ```sql USE nyc; ``` ##### Creating a table[​](#creating-a-table "Direct link to Creating a table") ```sql CREATE TABLE demo.nyc.taxis ( trip_id bigint, trip_distance float, fare_amount double, store_and_fwd_flag string, vendor_id bigint ) PARTITION BY (vendor_id); ``` ##### Writing Data to a Table[​](#writing-data-to-a-table "Direct link to Writing Data to a Table") ```sql INSERT INTO demo.nyc.taxis VALUES (1000371, 1.8, 15.32, 'N', 1), (1000372, 2.5, 22.15, 'N', 2), (1000373, 0.9, 9.01, 'N', 2), (1000374, 8.4, 42.13, 'Y', 1); ``` ##### Reading Data from a Table[​](#reading-data-from-a-table "Direct link to Reading Data from a Table") ```sql SELECT * FROM demo.nyc.taxis; ``` ##### Verify that the data is stored in object storage[​](#verify-that-the-data-is-stored-in-object-storage "Direct link to Verify that the data is stored in object storage") When you added and used the external catalog, Starrocks started using MinIO as the object store for the `demo.nyc.taxis` table. If you navigate to and then navigate through the Object Browser menu to `warehouse/nyc/taxis/` you can confirm that StarRocks is using MinIO for the storage. tip The username and password for MinIO are in the docker-compose.yml file. You will be prompted to change the password to something better, just ignore this advice for the tutorial. ![img](/assets/images/MinIO-Iceberg-data-8ade61c31be69444bd02b00acafe263c.png) ##### Next Steps[​](#next-steps "Direct link to Next Steps") ###### Adding Iceberg to StarRocks[​](#adding-iceberg-to-starrocks "Direct link to Adding Iceberg to StarRocks") If you already have a StarRocks 3.2.0, or later, environment, it comes with the Iceberg 1.6.0 included. No additional downloads or jars are needed. ###### Learn More[​](#learn-more "Direct link to Learn More") Now that you're up and running with Iceberg and StarRocks, check out the [StarRocks-Iceberg docs](https://docs.starrocks.io/docs/data_source/catalog/iceberg/iceberg_catalog.md) to learn more! --- ## Deployment ### Deploy Shared-nothing StarRocks Manually tip The preparations for manual deployment are outlined in the [Deployment prerequisites](https://docs.starrocks.io/docs/deployment/deployment_prerequisites.md) and [Check environment configurations](https://docs.starrocks.io/docs/deployment/environment_configurations.md) documents. Please start there if you are planning a production deployment. If you are getting started with StarRocks and would like to follow one of the Quick Starts, please refer to [Quick Starts](https://docs.starrocks.io/docs/quick_start.md). This topic describes how to manually deploy a shared-nothing StarRocks cluster (in which BE is responsible for both storage and computing). For other modes of installation, see [Deployment Overview](https://docs.starrocks.io/docs/deployment/deployment_overview.md). To deploy a shared-data StarRocks cluster (decoupled storage and computing), see [Deploy Shared-data StarRocks Manually](https://docs.starrocks.io/docs/deployment/deploy_shared_data_manually.md). #### Step 1: Start the Leader FE node[​](#step-1-start-the-leader-fe-node "Direct link to Step 1: Start the Leader FE node") The following procedures are performed on an FE instance. 1. Create a dedicated directory for metadata storage. We recommend storing metadata in a separate directory from the FE deployment files. Make sure that this directory exists and that you have write access to it. ```yaml # Replace with the metadata directory you want to create. mkdir -p ``` 2. Navigate to the directory that stores the [StarRocks FE deployment files](https://docs.starrocks.io/docs/deployment/prepare_deployment_files.md) you prepared earlier, and modify the FE configuration file **fe/conf/fe.conf**. a. Specify the metadata directory in the configuration item `meta_dir`. ```yaml # Replace with the metadata directory you have created. meta_dir = ``` b. If any of the FE ports mentioned in the [Environment Configuration Checklist](https://docs.starrocks.io/docs/deployment/environment_configurations.md#fe-ports) are occupied, you must assign valid alternatives in the FE configuration file. ```yaml http_port = aaaa # Default: 8030 rpc_port = bbbb # Default: 9020 query_port = cccc # Default: 9030 edit_log_port = dddd # Default: 9010 ``` > **CAUTION** > > If you want to deploy multiple FE nodes in a cluster, you must assign the same `http_port` to each FE node. c. If you want to enable IP address access for your cluster, you must add the configuration item `priority_networks` in the configuration file and assign a dedicated IP address (in the CIDR format) to the FE node. You can ignore this configuration item if you want to enable [FQDN access](https://docs.starrocks.io/docs/administration/management/enable_fqdn.md) for your cluster. ```yaml priority_networks = x.x.x.x/x ``` > **NOTE** > > * You can run `ifconfig` in your terminal to view the IP address(es) owned by the instance. > * From v3.3.0, StarRocks supports deployment based on IPv6. d. If you have multiple JDKs installed on the instance, and you want to use a specific JDK that is different from the one specified in the environment variable `JAVA_HOME`, you must specify the path where the chosen JDK is installed by adding the configuration item `JAVA_HOME` in the configuration file. ```yaml # Replace with the path where the chosen JDK is installed. JAVA_HOME = ``` For information about advanced configuration items, see [Parameter Configuration - FE configuration items](https://docs.starrocks.io/docs/administration/management/FE_configuration.md). 3. Start the FE node. * To enable IP address access for your cluster, run the following command to start the FE node: ```bash ./fe/bin/start_fe.sh --daemon ``` * To enable FQDN access for your cluster, run the following command to start the FE node: ```bash ./fe/bin/start_fe.sh --host_type FQDN --daemon ``` Note that you only need to specify the parameter `--host_type` ONCE when you start the node for the first time. > **CAUTION** > > Before starting the FE node with FQDN access enabled, make sure you have assigned hostnames for all instances in **/etc/hosts**. See [Environment Configuration Checklist - Hostnames](https://docs.starrocks.io/docs/deployment/environment_configurations.md#hostnames) for more information. 4. Check the FE logs to verify if the FE node is started successfully. ```bash cat fe/log/fe.log | grep thrift ``` A record of log like "2022-08-10 16:12:29,911 INFO (UNKNOWN x.x.x.x\_9010\_1660119137253(-1)|1) \[FeServer.start():52] thrift server started with port 9020." suggests that the FE node is started properly. #### Step 2: Start the BE service[​](#step-2-start-the-be-service "Direct link to Step 2: Start the BE service") note You can only add BE nodes to shared-nothing clusters and CN nodes to shared-data clusters. Otherwise, it may lead to unknown behaviors. The following procedures are performed on the BE instances. 1. Create a dedicated directory for data storage. We recommend storing data in a separate directory from the BE deployment directory. Make sure that this directory exists and you have write access to it. ```yaml # Replace with the data storage directory you want to create. mkdir -p ``` 2. Navigate to the directory that stores the [StarRocks BE deployment files](https://docs.starrocks.io/docs/deployment/prepare_deployment_files.md) you prepared earlier, and modify the BE configuration file **be/conf/be.conf**. a. Specify the data directory in the configuration item `storage_root_path`. Multiple volumes are separated by semicolon (;). Example: `/data1;/data2`. ```yaml # Replace with the data directory you have created. storage_root_path = ``` b. If any of the BE ports mentioned in the [Environment Configuration Checklist](https://docs.starrocks.io/docs/deployment/environment_configurations.md#be-ports) are occupied, you must assign valid alternatives in the BE configuration file. ```yaml be_port = vvvv # Default: 9060 be_http_port = xxxx # Default: 8040 heartbeat_service_port = yyyy # Default: 9050 brpc_port = zzzz # Default: 8060 starlet_port = uuuu # Default: 9070 ``` c. If you want to enable IP address access for your cluster, you must add the configuration item `priority_networks` in the configuration file and assign a dedicated IP address (in the CIDR format) to the BE node. You can ignore this configuration item if you want to enable FQDN access for your cluster. ```yaml priority_networks = x.x.x.x/x ``` > **NOTE** > > * You can run `ifconfig` in your terminal to view the IP address(es) owned by the instance. > * From v3.3.0, StarRocks supports deployment based on IPv6. d. If you have multiple JDKs installed on the instance, and you want to use a specific JDK that is different from the one specified in the environment variable `JAVA_HOME`, you must specify the path where the chosen JDK is installed by adding the configuration item `JAVA_HOME` in the configuration file. ```yaml # Replace with the path where the chosen JDK is installed. JAVA_HOME = ``` For information about advanced configuration items, see [Parameter Configuration - BE configuration items](https://docs.starrocks.io/docs/administration/management/BE_configuration.md). 3. Start the BE node. ```bash ./be/bin/start_be.sh --daemon ``` > **CAUTION** > > * Before starting the BE node with FQDN access enabled, make sure you have assigned hostnames for all instances in **/etc/hosts**. See [Environment Configuration Checklist - Hostnames](https://docs.starrocks.io/docs/deployment/environment_configurations.md#hostnames) for more information. > * You do not need to specify the parameter `--host_type` when you start BE nodes. 4. Check the BE logs to verify if the BE node is started successfully. ```bash cat be/log/be.INFO | grep heartbeat ``` A record of log like "I0810 16:18:44.487284 3310141 task\_worker\_pool.cpp:1387] Waiting to receive first heartbeat from frontend" suggests that the BE node is started properly. 5. You can start new BE nodes by repeating the above procedures on other BE instances. > **NOTE** > > * A high-availability cluster of BEs is automatically formed when at least three BE nodes are deployed and added to a StarRocks cluster. > > * If you want to deploy just one BE node, you must set `default_replication_num` to `1` in the FE configuration file **fe/conf/fe.conf**. > > ```yaml > default_replication_num = 1 > > ``` #### Step 3: Set up the cluster[​](#step-3-set-up-the-cluster "Direct link to Step 3: Set up the cluster") After all FE and BE nodes are started properly, you can set up the StarRocks cluster. The following procedures are performed on a MySQL client. You must have MySQL client 5.5.0 or later installed. 1. Connect to StarRocks via your MySQL client. You need to log in with the initial account `root`, and the password is empty by default. ```bash # Replace with the IP address (priority_networks) or FQDN # of the Leader FE node, and replace (Default: 9030) # with the query_port you specified in fe.conf. mysql -h -P -uroot ``` 2. Check the status of the Leader FE node by executing the following SQL. ```sql SHOW PROC '/frontends'\G ``` Example: ```plain MySQL [(none)]> SHOW PROC '/frontends'\G *************************** 1. row *************************** Name: x.x.x.x_9010_1686810741121 IP: x.x.x.x EditLogPort: 9010 HttpPort: 8030 QueryPort: 9030 RpcPort: 9020 Role: LEADER ClusterId: 919351034 Join: true Alive: true ReplayedJournalId: 1220 LastHeartbeat: 2023-06-15 15:39:04 IsHelper: true ErrMsg: StartTime: 2023-06-15 14:32:28 Version: 3.0.0-48f4d81 1 row in set (0.01 sec) ``` * If the field `Alive` is `true`, this FE node is properly started and added to the cluster. * If the field `Role` is `FOLLOWER`, this FE node is eligible to be elected as the Leader FE node. * If the field `Role` is `LEADER`, this FE node is the Leader FE node. 3. Add the BE nodes to the cluster. ```sql -- Replace with the IP address (priority_networks) -- or FQDN of the BE nodes, and replace -- with the heartbeat_service_port (Default: 9050) you specified in be.conf. ALTER SYSTEM ADD BACKEND ":"; ``` > **NOTE** > > You can use the preceding command to add multiple BE nodes at a time. Each `:` pair represents one BE node. 4. Check the status of the BE nodes by executing the following SQL. ```sql SHOW PROC '/backends'\G ``` Example: ```plain MySQL [(none)]> SHOW PROC '/backends'\G *************************** 1. row *************************** BackendId: 10007 IP: 172.26.195.67 HeartbeatPort: 9050 BePort: 9060 HttpPort: 8040 BrpcPort: 8060 LastStartTime: 2023-06-15 15:23:08 LastHeartbeat: 2023-06-15 15:57:30 Alive: true SystemDecommissioned: false ClusterDecommissioned: false TabletNum: 30 DataUsedCapacity: 0.000 AvailCapacity: 341.965 GB TotalCapacity: 1.968 TB UsedPct: 83.04 % MaxDiskUsedPct: 83.04 % ErrMsg: Version: 3.0.0-48f4d81 Status: {"lastSuccessReportTabletsTime":"2023-06-15 15:57:08"} DataTotalCapacity: 341.965 GB DataUsedPct: 0.00 % CpuCores: 16 NumRunningQueries: 0 MemUsedPct: 0.01 % CpuUsedPct: 0.0 % ``` If the field `Alive` is `true`, this BE node is properly started and added to the cluster. #### Step 4: (Optional) Deploy a high-availability FE cluster[​](#step-4-optional-deploy-a-high-availability-fe-cluster "Direct link to Step 4: (Optional) Deploy a high-availability FE cluster") A high-availability FE cluster requires at least THREE Follower FE nodes in the StarRocks cluster. After the Leader FE node is started successfully, you can then start two new FE nodes to deploy a high-availability FE cluster. 1. Connect to StarRocks via your MySQL client. You need to log in with the initial account `root`, and the password is empty by default. ```bash # Replace with the IP address (priority_networks) or FQDN # of the Leader FE node, and replace (Default: 9030) # with the query_port you specified in fe.conf. mysql -h -P -uroot ``` 2. Add the new Follower FE node to the cluster by executing the following SQL. ```sql -- Replace with the IP address (priority_networks) -- or FQDN of the new Follower FE node, and replace -- with the edit_log_port (Default: 9010) you specified in fe.conf. ALTER SYSTEM ADD FOLLOWER ":"; ``` > **NOTE** > > * You can use the preceding command to add a single Follower FE nodes each time. > * If you want to add Observer FE nodes, execute `ALTER SYSTEM ADD OBSERVER ":"=`. For detailed instructions, see [ALTER SYSTEM - FE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md). 3. Launch a terminal on the new FE instance, create a dedicated directory for metadata storage, navigate to the directory that stores the StarRocks FE deployment files, and modify the FE configuration file **fe/conf/fe.conf**. For more instructions, see [Step 1: Start the Leader FE node](#step-1-start-the-leader-fe-node). Basically, you can repeat the procedures in Step 1 **except for the command used to start the FE node**. After configuring the Follower FE node, execute the following SQL to assign a helper node for Follower FE node and start the Follower FE node. > **NOTE** > > When adding new Follower FE node to a cluster, you must assign a helper node (essentially an existing Follower FE node) to the new Follower FE node to synchronize the metadata. * To start a new FE node with IP address access, run the following command to start the FE node: ```bash # Replace with the IP address (priority_networks) # of the Leader FE node, and replace (Default: 9010) with # the Leader FE node's edit_log_port. ./fe/bin/start_fe.sh --helper : --daemon ``` Note that you only need to specify the parameter `--helper` ONCE when you start the node for the first time. * To start a new FE node with FQDN access, run the following command to start the FE node: ```bash # Replace with the FQDN of the Leader FE node, # and replace (Default: 9010) with the Leader FE node's edit_log_port. ./fe/bin/start_fe.sh --helper : \ --host_type FQDN --daemon ``` Note that you only need to specify the parameters `--helper` and `--host_type` ONCE when you start the node for the first time. 4. Check the FE logs to verify if the FE node is started successfully. ```bash cat fe/log/fe.log | grep thrift ``` A record of log like "2022-08-10 16:12:29,911 INFO (UNKNOWN x.x.x.x\_9010\_1660119137253(-1)|1) \[FeServer.start():52] thrift server started with port 9020." suggests that the FE node is started properly. 5. Repeat the preceding procedure 2, 3, and 4 until you have start all the new Follower FE nodes properly, and then check the status of the FE nodes by executing the following SQL from your MySQL client: ```sql SHOW PROC '/frontends'\G ``` Example: ```plain MySQL [(none)]> SHOW PROC '/frontends'\G *************************** 1. row *************************** Name: x.x.x.x_9010_1686810741121 IP: x.x.x.x EditLogPort: 9010 HttpPort: 8030 QueryPort: 9030 RpcPort: 9020 Role: LEADER ClusterId: 919351034 Join: true Alive: true ReplayedJournalId: 1220 LastHeartbeat: 2023-06-15 15:39:04 IsHelper: true ErrMsg: StartTime: 2023-06-15 14:32:28 Version: 3.0.0-48f4d81 *************************** 2. row *************************** Name: x.x.x.x_9010_1686814080597 IP: x.x.x.x EditLogPort: 9010 HttpPort: 8030 QueryPort: 9030 RpcPort: 9020 Role: FOLLOWER ClusterId: 919351034 Join: true Alive: true ReplayedJournalId: 1219 LastHeartbeat: 2023-06-15 15:39:04 IsHelper: true ErrMsg: StartTime: 2023-06-15 15:38:53 Version: 3.0.0-48f4d81 *************************** 3. row *************************** Name: x.x.x.x_9010_1686814090833 IP: x.x.x.x EditLogPort: 9010 HttpPort: 8030 QueryPort: 9030 RpcPort: 9020 Role: FOLLOWER ClusterId: 919351034 Join: true Alive: true ReplayedJournalId: 1219 LastHeartbeat: 2023-06-15 15:39:04 IsHelper: true ErrMsg: StartTime: 2023-06-15 15:37:52 Version: 3.0.0-48f4d81 3 rows in set (0.02 sec) ``` * If the field `Alive` is `true`, this FE node is properly started and added to the cluster. * If the field `Role` is `FOLLOWER`, this FE node is eligible to be elected as the Leader FE node. * If the field `Role` is `LEADER`, this FE node is the Leader FE node. #### Stop the StarRocks cluster[​](#stop-the-starrocks-cluster "Direct link to Stop the StarRocks cluster") You can stop the StarRocks cluster by running the following commands on the corresponding instances. * Stop an FE node. ```bash ./fe/bin/stop_fe.sh ``` * Stop a BE node. ```bash ./be/bin/stop_be.sh ``` #### Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") Try the following steps to identify the errors that occur when you start the FE or BE nodes: * If an FE node is not started properly, you can identify the problem by checking its log in **fe/log/fe.warn.log**. ```bash cat fe/log/fe.warn.log ``` Having identified and resolved the problem, you must first terminate the current FE process, delete the existing **meta** directory, create a new metadata storage directory, and then restart the FE node with the correct configuration. * If a BE node is not started properly, you can identify the problem by checking its log in **be/log/be.WARNING**. ```bash cat be/log/be.WARNING ``` Having identified and resolved the problem, you must first terminate the existing BE process, delete the existing **storage** directory, create a new data storage directory, and then restart the BE node with the correct configuration. #### What to do next[​](#what-to-do-next "Direct link to What to do next") Having deployed your StarRocks cluster, you can move on to [Post-deployment Setup](https://docs.starrocks.io/docs/deployment/post_deployment_setup.md) for instructions on initial management measures. --- ### Deploy Shared-data StarRocks Manually tip The preparations for manual deployment are outlined in the [Deployment prerequisites](https://docs.starrocks.io/docs/deployment/deployment_prerequisites.md) and [Check environment configurations](https://docs.starrocks.io/docs/deployment/environment_configurations.md) documents. Please start there if you are planning a production deployment. If you are getting started with StarRocks and would like to follow one of the Quick Starts, please refer to [Quick Starts](https://docs.starrocks.io/docs/quick_start.md). This topic describes how to manually deploy a shared-data StarRocks cluster (in which storage and computing are decoupled). For other modes of installation, see [Deployment Overview](https://docs.starrocks.io/docs/deployment/deployment_overview.md). To deploy a shared-nothing StarRocks cluster (in which BE is responsible for both storage and computing), see [Deploy Shared-nothing StarRocks Manually](https://docs.starrocks.io/docs/deployment/deploy_manually.md). #### Overview[​](#overview "Direct link to Overview") The shared-data StarRocks cluster is specifically engineered for the cloud on the premise of separation of storage and computing. It allows data to be stored in remote storage (for example, HDFS, AWS S3, Google GCS, Azure Blob Storage, Azure Data Lake Storage Gen2, and MinIO). You can achieve not only cheaper storage and better resource isolation, but elastic scalability for your cluster. The query performance of the shared-data StarRocks cluster aligns with that of a shared-nothing StarRocks cluster when the local disk cache is hit. The StarRocks shared-data cluster is made up of Frontend Engines (FEs) and Compute Nodes (CNs), which replace the classic Backend Engines (BEs) in shared-nothing clusters. Compared to the classic shared-nothing StarRocks architecture, separation of storage and computing offers a wide range of benefits. By decoupling these components, StarRocks provides: * Inexpensive and seamlessly scalable storage. * Elastic scalable computing. Because data is not stored in Compute Nodes (CNs), scaling can be done without data migration or shuffling across nodes. * Local disk cache for hot data to boost query performance. * Asynchronous data ingestion into remote storage, allowing a significant improvement in loading performance. The architecture of the shared-data cluster is as follows: ![Shared-data Architecture](/assets/images/share_data_arch-2aee703af4e8afcf46e5bf9866cd2f2a.png) #### Step 1: Start the Leader FE node[​](#step-1-start-the-leader-fe-node "Direct link to Step 1: Start the Leader FE node") The following procedures are performed on an FE instance. 1. Create a dedicated directory for metadata storage. We recommend storing metadata in a separate directory from the FE deployment files. Make sure that this directory exists and that you have write access to it. ```yaml # Replace with the metadata directory you want to create. mkdir -p ``` 2. Navigate to the directory that stores the [StarRocks FE deployment files](https://docs.starrocks.io/docs/deployment/prepare_deployment_files.md) you prepared earlier, and modify the FE configuration file **fe/conf/fe.conf**. a. Set shared-data run mode. ```yaml run_mode = shared_data ``` b. Specify the metadata directory in the configuration item `meta_dir`. ```yaml # Replace with the metadata directory you have created. meta_dir = ``` c. If any of the FE ports mentioned in the [Environment Configuration Checklist](https://docs.starrocks.io/docs/deployment/environment_configurations.md#fe-ports) are occupied, you must assign valid alternatives in the FE configuration file. ```yaml http_port = aaaa # Default: 8030 rpc_port = bbbb # Default: 9020 query_port = cccc # Default: 9030 edit_log_port = dddd # Default: 9010 cloud_native_meta_port = eeee # Default: 6090 ``` > **CAUTION** > > If you want to deploy multiple FE nodes in a cluster, you must assign the same `http_port` to each FE node. d. If you want to enable IP address access for your cluster, you must add the configuration item `priority_networks` in the configuration file and assign a dedicated IP address (in the CIDR format) to the FE node. You can ignore this configuration item if you want to enable [FQDN access](https://docs.starrocks.io/docs/administration/management/enable_fqdn.md) for your cluster. ```yaml priority_networks = x.x.x.x/x ``` > **NOTE** > > * You can run `ifconfig` in your terminal to view the IP address(es) owned by the instance. > * From v3.3.0, StarRocks supports deployment based on IPv6. e. If you have multiple JDKs installed on the instance, and you want to use a specific JDK that is different from the one specified in the environment variable `JAVA_HOME`, you must specify the path where the chosen JDK is installed by adding the configuration item `JAVA_HOME` in the configuration file. ```yaml # Replace with the path where the chosen JDK is installed. JAVA_HOME = ``` For information about advanced configuration items, see [Parameter Configuration - FE configuration items](https://docs.starrocks.io/docs/administration/management/FE_configuration.md). 3. Start the FE node. * To enable IP address access for your cluster, run the following command to start the FE node: ```bash ./fe/bin/start_fe.sh --daemon ``` * To enable FQDN access for your cluster, run the following command to start the FE node: ```bash ./fe/bin/start_fe.sh --host_type FQDN --daemon ``` Note that you only need to specify the parameter `--host_type` ONCE when you start the node for the first time. > **CAUTION** > > Before starting the FE node with FQDN access enabled, make sure you have assigned hostnames for all instances in **/etc/hosts**. See [Environment Configuration Checklist - Hostnames](https://docs.starrocks.io/docs/deployment/environment_configurations.md#hostnames) for more information. 4. Check the FE logs to verify if the FE node is started successfully. ```bash cat fe/log/fe.log | grep thrift ``` A record of log like "2022-08-10 16:12:29,911 INFO (UNKNOWN x.x.x.x\_9010\_1660119137253(-1)|1) \[FeServer.start():52] thrift server started with port 9020." suggests that the FE node is started properly. #### Step 2: Start the CN service[​](#step-2-start-the-cn-service "Direct link to Step 2: Start the CN service") note You can only add BE nodes to shared-nothing clusters and CN nodes to shared-data clusters. Otherwise, it may lead to unknown behaviors. The following procedures are performed on the CN instances. You can deploy CN nodes with the BE deployment files. 1. Create a dedicated directory for data cache. We recommend caching data in a separate directory from the CN deployment directory. Make sure that this directory exists and you have write access to it. ```yaml # Replace with the data cache directory you want to create. mkdir -p ``` 2. Navigate to the directory that stores the [StarRocks BE deployment files](https://docs.starrocks.io/docs/deployment/prepare_deployment_files.md) you prepared earlier, and modify the CN configuration file **be/conf/cn.conf**. a. Specify the data directory in the configuration item `storage_root_path`. Multiple volumes are separated by semicolon (;). Example: `/data1;/data2`. ```yaml # Replace with the data directory you have created. storage_root_path = ``` Local cache is effective when queries are frequent and the data being queried is recent, but there are cases that you may wish to turn off the local cache completely. * In a Kubernetes environment with CN pods that scale up and down in number on demand, the pods may not have storage volumes attached. * When the data being queried is in a data lake in remote storage and most of it is archive (old) data, if the queries are infrequent, the data cache will have a low hit ratio and the benefit may not be worth having the cache. To turn off the data cache set: ```yaml storage_root_path = ``` > **NOTE** > > The data is cached under the directory **`/starlet_cache`**. b. If any of the CN ports mentioned in the [Environment Configuration Checklist](https://docs.starrocks.io/docs/deployment/environment_configurations.md) are occupied, you must assign valid alternatives in the CN configuration file. ```yaml be_port = vvvv # Default: 9060 be_http_port = xxxx # Default: 8040 heartbeat_service_port = yyyy # Default: 9050 brpc_port = zzzz # Default: 8060 starlet_port = uuuu # Default: 9070 ``` c. If you want to enable IP address access for your cluster, you must add the configuration item `priority_networks` in the configuration file and assign a dedicated IP address (in the CIDR format) to the CN node. You can ignore this configuration item if you want to enable FQDN access for your cluster. ```yaml priority_networks = x.x.x.x/x ``` > **NOTE** > > * You can run `ifconfig` in your terminal to view the IP address(es) owned by the instance. > * From v3.3.0, StarRocks supports deployment based on IPv6. d. If you have multiple JDKs installed on the instance, and you want to use a specific JDK that is different from the one specified in the environment variable `JAVA_HOME`, you must specify the path where the chosen JDK is installed by adding the configuration item `JAVA_HOME` in the configuration file. ```yaml # Replace with the path where the chosen JDK is installed. JAVA_HOME = ``` For information about advanced configuration items, see [Parameter Configuration - BE configuration items](https://docs.starrocks.io/docs/administration/management/BE_configuration.md) because most of CN's parameters are inherited from BE. 3. Start the CN node. ```bash ./be/bin/start_cn.sh --daemon ``` > **CAUTION** > > * Before starting the CN node with FQDN access enabled, make sure you have assigned hostnames for all instances in **/etc/hosts**. See [Environment Configuration Checklist - Hostnames](https://docs.starrocks.io/docs/deployment/environment_configurations.md#hostnames) for more information. > * You do not need to specify the parameter `--host_type` when you start CN nodes. 4. Check the CN logs to verify if the CN node is started successfully. ```bash cat be/log/cn.INFO | grep heartbeat ``` A record of log like "I0313 15:03:45.820030 412450 thrift\_server.cpp:375] heartbeat has started listening port on 9050" suggests that the CN node is started properly. 5. You can start new CN nodes by repeating the above procedures on other instances. #### Step 3: Set up the cluster[​](#step-3-set-up-the-cluster "Direct link to Step 3: Set up the cluster") After all FE and CN nodes are started properly, you can set up the StarRocks cluster. The following procedures are performed on a MySQL client. You must have MySQL client 5.5.0 or later installed. 1. Connect to StarRocks via your MySQL client. You need to log in with the initial account `root`, and the password is empty by default. ```bash # Replace with the IP address (priority_networks) or FQDN # of the Leader FE node, and replace (Default: 9030) # with the query_port you specified in fe.conf. mysql -h -P -uroot ``` 2. Check the status of the Leader FE node by executing the following SQL. ```sql SHOW PROC '/frontends'\G ``` Example: ```plain MySQL [(none)]> SHOW PROC '/frontends'\G *************************** 1. row *************************** Name: x.x.x.x_9010_1686810741121 IP: x.x.x.x EditLogPort: 9010 HttpPort: 8030 QueryPort: 9030 RpcPort: 9020 Role: LEADER ClusterId: 919351034 Join: true Alive: true ReplayedJournalId: 1220 LastHeartbeat: 2023-06-15 15:39:04 IsHelper: true ErrMsg: StartTime: 2023-06-15 14:32:28 Version: 3.0.0-48f4d81 1 row in set (0.01 sec) ``` * If the field `Alive` is `true`, this FE node is properly started and added to the cluster. * If the field `Role` is `FOLLOWER`, this FE node is eligible to be elected as the Leader FE node. * If the field `Role` is `LEADER`, this FE node is the Leader FE node. 3. Add the CN nodes to the cluster. ```sql -- Replace with the IP address (priority_networks) -- or FQDN of the CN node, and replace -- with the heartbeat_service_port (Default: 9050) you specified in cn.conf. ALTER SYSTEM ADD COMPUTE NODE ":"; ``` > **NOTE** > > You can add multiple CN nodes with one SQL. Each `:` pair represents one CN node. 4. Check the status of the CN nodes by executing the following SQL. ```sql SHOW PROC '/compute_nodes'\G ``` Example: ```plain MySQL [(none)]> SHOW PROC '/compute_nodes'\G *************************** 1. row *************************** ComputeNodeId: 10003 IP: x.x.x.x HeartbeatPort: 9050 BePort: 9060 HttpPort: 8040 BrpcPort: 8060 LastStartTime: 2023-03-13 15:11:13 LastHeartbeat: 2023-03-13 15:11:13 Alive: true SystemDecommissioned: false ClusterDecommissioned: false ErrMsg: Version: 2.5.2-c3772fb 1 row in set (0.00 sec) ``` If the field `Alive` is `true`, this CN node is properly started and added to the cluster. #### Step 4: (Optional) Deploy a high-availability FE cluster[​](#step-4-optional-deploy-a-high-availability-fe-cluster "Direct link to Step 4: (Optional) Deploy a high-availability FE cluster") A high-availability FE cluster requires at least THREE Follower FE nodes in the StarRocks cluster. After the Leader FE node is started successfully, you can then start two new FE nodes to deploy a high-availability FE cluster. 1. Connect to StarRocks via your MySQL client. You need to log in with the initial account `root`, and the password is empty by default. ```bash # Replace with the IP address (priority_networks) or FQDN # of the Leader FE node, and replace (Default: 9030) # with the query_port you specified in fe.conf. mysql -h -P -uroot ``` 2. Add the new Follower FE node to the cluster by executing the following SQL. ```sql -- Replace with the IP address (priority_networks) -- or FQDN of the new Follower FE node, and replace -- with the edit_log_port (Default: 9010) you specified in fe.conf. ALTER SYSTEM ADD FOLLOWER ":"; ``` > **NOTE** > > * You can use the preceding command to add a single Follower FE nodes each time. > * If you want to add Observer FE nodes, execute `ALTER SYSTEM ADD OBSERVER ":"=`. For detailed instructions, see [ALTER SYSTEM - FE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md). 3. Launch a terminal on the new FE instance, create a dedicated directory for metadata storage, navigate to the directory that stores the StarRocks FE deployment files, and modify the FE configuration file **fe/conf/fe.conf**. For more instructions, see [Step 1: Start the Leader FE node](#step-1-start-the-leader-fe-node). Basically, you can repeat the procedures in Step 1 **except for the command used to start the FE node**. After configuring the Follower FE node, execute the following SQL to assign a helper node for Follower FE node and start the Follower FE node. > **NOTE** > > When adding new Follower FE node to a cluster, you must assign a helper node (essentially an existing Follower FE node) to the new Follower FE node to synchronize the metadata. * To start a new FE node with IP address access, run the following command to start the FE node: ```bash # Replace with the IP address (priority_networks) # of the Leader FE node, and replace (Default: 9010) with # the Leader FE node's edit_log_port. ./fe/bin/start_fe.sh --helper : --daemon ``` Note that you only need to specify the parameter `--helper` ONCE when you start the node for the first time. * To start a new FE node with FQDN access, run the following command to start the FE node: ```bash # Replace with the FQDN of the Leader FE node, # and replace (Default: 9010) with the Leader FE node's edit_log_port. ./fe/bin/start_fe.sh --helper : \ --host_type FQDN --daemon ``` Note that you only need to specify the parameters `--helper` and `--host_type` ONCE when you start the node for the first time. 4. Check the FE logs to verify if the FE node is started successfully. ```bash cat fe/log/fe.log | grep thrift ``` A record of log like "2022-08-10 16:12:29,911 INFO (UNKNOWN x.x.x.x\_9010\_1660119137253(-1)|1) \[FeServer.start():52] thrift server started with port 9020." suggests that the FE node is started properly. 5. Repeat the preceding procedure 2, 3, and 4 until you have start all the new Follower FE nodes properly, and then check the status of the FE nodes by executing the following SQL from your MySQL client: ```sql SHOW PROC '/frontends'\G ``` Example: ```plain MySQL [(none)]> SHOW PROC '/frontends'\G *************************** 1. row *************************** Name: x.x.x.x_9010_1686810741121 IP: x.x.x.x EditLogPort: 9010 HttpPort: 8030 QueryPort: 9030 RpcPort: 9020 Role: LEADER ClusterId: 919351034 Join: true Alive: true ReplayedJournalId: 1220 LastHeartbeat: 2023-06-15 15:39:04 IsHelper: true ErrMsg: StartTime: 2023-06-15 14:32:28 Version: 3.0.0-48f4d81 *************************** 2. row *************************** Name: x.x.x.x_9010_1686814080597 IP: x.x.x.x EditLogPort: 9010 HttpPort: 8030 QueryPort: 9030 RpcPort: 9020 Role: FOLLOWER ClusterId: 919351034 Join: true Alive: true ReplayedJournalId: 1219 LastHeartbeat: 2023-06-15 15:39:04 IsHelper: true ErrMsg: StartTime: 2023-06-15 15:38:53 Version: 3.0.0-48f4d81 *************************** 3. row *************************** Name: x.x.x.x_9010_1686814090833 IP: x.x.x.x EditLogPort: 9010 HttpPort: 8030 QueryPort: 9030 RpcPort: 9020 Role: FOLLOWER ClusterId: 919351034 Join: true Alive: true ReplayedJournalId: 1219 LastHeartbeat: 2023-06-15 15:39:04 IsHelper: true ErrMsg: StartTime: 2023-06-15 15:37:52 Version: 3.0.0-48f4d81 3 rows in set (0.02 sec) ``` * If the field `Alive` is `true`, this FE node is properly started and added to the cluster. * If the field `Role` is `FOLLOWER`, this FE node is eligible to be elected as the Leader FE node. * If the field `Role` is `LEADER`, this FE node is the Leader FE node. #### Step 5: Create and set the default storage volume[​](#step-5-create-and-set-the-default-storage-volume "Direct link to Step 5: Create and set the default storage volume") To give your shared-data StarRocks cluster permission to store data in your remote storage, you must reference a storage volume when you create databases or cloud-native tables. A storage volume consists of the properties and credential information of the remote data storage. If you have deployed a new shared-data StarRocks cluster, you must define a default storage volume before you can create databases and tables in the cluster. Choose your cloud provider and service, and execute the corresponding statements to create and set the default storage volume. * AWS S3 * Google Cloud Storage * Azure Blob Storage * Azure Data Lake Storage Gen2 * MinIO * HDFS The following example creates a storage volume `def_volume` for an AWS S3 bucket `defaultbucket` with the IAM user-based credential (Access Key and Secret Key), enables the [Partitioned Prefix](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md#partitioned-prefix) feature, and sets it as the default storage volume: ```sql CREATE STORAGE VOLUME def_volume TYPE = S3 LOCATIONS = ("s3://defaultbucket") PROPERTIES ( "enabled" = "true", "aws.s3.region" = "us-west-2", "aws.s3.endpoint" = "https://s3.us-west-2.amazonaws.com", "aws.s3.use_aws_sdk_default_behavior" = "false", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "xxxxxxxxxx", "aws.s3.secret_key" = "yyyyyyyyyy", "aws.s3.enable_partitioned_prefix" = "true" ); SET def_volume AS DEFAULT STORAGE VOLUME; ``` The following example creates a storage volume `def_volume` for a GCS bucket `defaultbucket` with Service Account-based credentials, enables the storage volume, and sets it as the default storage volume: ```sql CREATE STORAGE VOLUME def_volume TYPE = GS LOCATIONS = ("gs://defaultbucket") PROPERTIES ( "enabled" = "true", "gcp.gcs.use_compute_engine_service_account" = "false", "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "" ); SET def_volume AS DEFAULT STORAGE VOLUME; ``` The following example creates a storage volume `def_volume` for an Azure Blob Storage bucket `defaultbucket` with shared key access, enables the storage volume, and sets it as the default storage volume: ```sql CREATE STORAGE VOLUME def_volume TYPE = AZBLOB LOCATIONS = ("azblob://defaultbucket/test/") PROPERTIES ( "enabled" = "true", "azure.blob.endpoint" = "", "azure.blob.shared_key" = "" ); SET def_volume AS DEFAULT STORAGE VOLUME; ``` The following example creates a storage volume `adls2` for an Azure Data Lake Storage Gen2 file system `testfilesystem` with SAS token, and disables the storage volume: ```sql CREATE STORAGE VOLUME adls2 TYPE = ADLS2 LOCATIONS = ("adls2://testfilesystem/starrocks") PROPERTIES ( "enabled" = "false", "azure.adls2.endpoint" = "", "azure.adls2.sas_token" = "" ); ``` The following example creates a storage volume `def_volume` for a MinIO bucket `defaultbucket` with Access Key and Secret Key credentials, enables the [Partitioned Prefix](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md#partitioned-prefix) feature, and sets it as the default storage volume: ```sql CREATE STORAGE VOLUME def_volume TYPE = S3 LOCATIONS = ("s3://defaultbucket") PROPERTIES ( "enabled" = "true", "aws.s3.region" = "us-west-2", "aws.s3.endpoint" = "https://hostname.domainname.com:portnumber", "aws.s3.access_key" = "xxxxxxxxxx", "aws.s3.secret_key" = "yyyyyyyyyy", "aws.s3.enable_partitioned_prefix" = "true" ); SET def_volume AS DEFAULT STORAGE VOLUME; ``` The following example creates a storage volume `def_volume` for an HDFS storage, enables the storage volume, and sets it as the default storage volume: ```sql CREATE STORAGE VOLUME def_volume TYPE = HDFS LOCATIONS = ("hdfs://127.0.0.1:9000/user/starrocks/"); SET def_volume AS DEFAULT STORAGE VOLUME; ``` In addition to the authentication methods used above, StarRocks also supports a variety of credentials to access your remote storage. See [CREATE STORAGE VOLUME - Credential information](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md#credential-information) for detailed instructions. #### Use shared-data StarRocks[​](#use-shared-data-starrocks "Direct link to Use shared-data StarRocks") The usage of shared-data StarRocks clusters is also similar to that of a classic shared-nothing StarRocks cluster, except that the shared-data cluster uses storage volumes and cloud-native tables to store data in remote storage. ##### Create a database and a cloud-native table[​](#create-a-database-and-a-cloud-native-table "Direct link to Create a database and a cloud-native table") After you create a default storage volume, you can then create a database and a cloud-native table using this storage volume. Shared-data StarRocks clusters support all [StarRocks table types](https://docs.starrocks.io/docs/table_design/table_types.md). The following example creates a database `cloud_db` and a table `detail_demo` based on Duplicate Key table type, enables the local disk cache, sets the hot data validity duration to one month, and disables asynchronous data ingestion into remote storage: ```sql CREATE DATABASE cloud_db; USE cloud_db; CREATE TABLE IF NOT EXISTS detail_demo ( recruit_date DATE NOT NULL COMMENT "YYYY-MM-DD", region_num TINYINT COMMENT "range [-128, 127]", num_plate SMALLINT COMMENT "range [-32768, 32767] ", tel INT COMMENT "range [-2147483648, 2147483647]", id BIGINT COMMENT "range [-2^63 + 1 ~ 2^63 - 1]", password LARGEINT COMMENT "range [-2^127 + 1 ~ 2^127 - 1]", name CHAR(20) NOT NULL COMMENT "range char(m),m in (1-255) ", profile VARCHAR(500) NOT NULL COMMENT "upper limit value 65533 bytes", ispass BOOLEAN COMMENT "true/false") DUPLICATE KEY(recruit_date, region_num) DISTRIBUTED BY HASH(recruit_date, region_num) PROPERTIES ( "storage_volume" = "def_volume", "datacache.enable" = "true", "datacache.partition_duration" = "1 MONTH" ); ``` > **NOTE** > > The default storage volume is used when you create a database or a cloud-native table in a shared-data StarRocks cluster if no storage volume is specified. ###### `PROPERTIES`[​](#properties "Direct link to properties") In addition to the regular table `PROPERTIES`, you need to specify the following `PROPERTIES` when creating a table for shared-data StarRocks cluster: ###### `datacache.enable`[​](#datacacheenable "Direct link to datacacheenable") Whether to enable the local disk cache. * `true` (Default) When this property is set to `true`, the data to be loaded is simultaneously written into the remote storage and the local disk (as the cache for query acceleration). * `false` When this property is set to `false`, the data is loaded only into the remote storage. > **NOTE** > > To enable the local disk cache, you must specify the directory of the disk in the CN configuration item `storage_root_path`. ###### `datacache.partition_duration`[​](#datacachepartition_duration "Direct link to datacachepartition_duration") The validity duration of the hot data. When the local disk cache is enabled, all data is loaded into the cache. When the cache is full, StarRocks deletes the less recently used data from the cache. When a query needs to scan the deleted data, StarRocks checks if the data is within the duration of validity starting from the current time. If the data is within the duration, StarRocks loads the data into the cache again. If the data is not within the duration, StarRocks does not load it into the cache. This property is a string value that can be specified with the following units: `YEAR`, `MONTH`, `DAY`, and `HOUR`, for example, `7 DAY` and `12 HOUR`. If it is not specified, all data is cached as the hot data. > **NOTE** > > This property is available only when `datacache.enable` is set to `true`. ##### View table information[​](#view-table-information "Direct link to View table information") You can view the information of tables in a specific database using `SHOW PROC "/dbs/"`. See [SHOW PROC](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md) for more information. Example: ```plain mysql> SHOW PROC "/dbs/xxxxx"; +---------+-------------+----------+---------------------+--------------+--------+--------------+--------------------------+--------------+---------------+------------------------------+ | TableId | TableName | IndexNum | PartitionColumnName | PartitionNum | State | Type | LastConsistencyCheckTime | ReplicaCount | PartitionType | StoragePath | +---------+-------------+----------+---------------------+--------------+--------+--------------+--------------------------+--------------+---------------+------------------------------+ | 12003 | detail_demo | 1 | NULL | 1 | NORMAL | CLOUD_NATIVE | NULL | 8 | UNPARTITIONED | s3://xxxxxxxxxxxxxx/1/12003/ | +---------+-------------+----------+---------------------+--------------+--------+--------------+--------------------------+--------------+---------------+------------------------------+ ``` The `Type` of a table in shared-data StarRocks cluster is `CLOUD_NATIVE`. In the field `StoragePath`, StarRocks returns the remote storage directory where the table is stored. ##### Load data into a shared-data StarRocks cluster[​](#load-data-into-a-shared-data-starrocks-cluster "Direct link to Load data into a shared-data StarRocks cluster") Shared-data StarRocks clusters support all loading methods provided by StarRocks. See [Loading options](https://docs.starrocks.io/docs/loading/Loading_intro.md) for more information. ##### Query in a shared-data StarRocks cluster[​](#query-in-a-shared-data-starrocks-cluster "Direct link to Query in a shared-data StarRocks cluster") Tables in a shared-data StarRocks cluster support all types of queries provided by StarRocks. See StarRocks [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md) for more information. > **NOTE** > > Shared-data StarRocks clusters support [synchronous materialized views](https://docs.starrocks.io/docs/using_starrocks/Materialized_view-single_table.md) from v3.4.0. #### Stop the StarRocks cluster[​](#stop-the-starrocks-cluster "Direct link to Stop the StarRocks cluster") You can stop the StarRocks cluster by running the following commands on the corresponding instances. * Stop an FE node. ```bash ./fe/bin/stop_fe.sh ``` * Stop a CN node. ```bash ./be/bin/stop_cn.sh ``` #### Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") Try the following steps to identify the errors that occur when you start the FE or CN nodes: * If an FE node is not started properly, you can identify the problem by checking its log in **fe/log/fe.warn.log**. ```bash cat fe/log/fe.warn.log ``` Having identified and resolved the problem, you must first terminate the current FE process, delete the existing **meta** directory, create a new metadata storage directory, and then restart the FE node with the correct configuration. * If a CN node is not started properly, you can identify the problem by checking its log in **be/log/cn.WARNING**. ```bash cat be/log/cn.WARNING ``` Having identified and resolved the problem, you must first terminate the existing CN process, and then restart the CN node with the correct configuration. #### What to do next[​](#what-to-do-next "Direct link to What to do next") Having deployed your StarRocks cluster, you can move on to [Post-deployment Setup](https://docs.starrocks.io/docs/deployment/post_deployment_setup.md) for instructions on initial management measures. --- ### Deployment Overview This chapter describes how to deploy, upgrade, and downgrade a StarRocks cluster in a production environment. #### Deployment Procedure[​](#deployment-procedure "Direct link to Deployment Procedure") A summary of the deployment procedure is as follows and later topics provide the details. The deployment of StarRocks generally follows the steps outlined here: 1. Confirm the [hardware and software requirements](https://docs.starrocks.io/docs/deployment/deployment_prerequisites.md) for your StarRocks deployment. Check the prerequisites that your servers must meet before deploying StarRocks, including CPU, memory, storage, network, operating system, and dependencies. 2. [Plan your cluster size](https://docs.starrocks.io/docs/deployment/plan_cluster.md). Plan the number of FE nodes and BE/CN nodes in your cluster, and the hardware specifications of the servers. 3. [Check environment configurations](https://docs.starrocks.io/docs/deployment/environment_configurations.md). When your servers are ready, you need to check and modify some environment configurations before deploying StarRocks on them. 4. [Prepare deployment files](https://docs.starrocks.io/docs/deployment/prepare_deployment_files.md). * If you want to deploy StarRocks on x86 architecture, you can directly download and extract the software package provided on our official website. * If you want to deploy StarRocks on ARM architecture, you need to prepare the deployment files from the StarRocks Docker image. * If you want to deploy StarRocks on Kubernetes, you can skip this step. 5. Deploy StarRocks. * If you want to deploy a shared-data StarRocks cluster, which features a disaggregated storage and compute architecture, see [Deploy and use shared-data StarRocks](https://docs.starrocks.io/docs/deployment/deploy_shared_data_manually.md) for instructions. * If you want to deploy a shared-nothing StarRocks cluster, which uses local storage, you have the following options: * [Deploy StarRocks manually](https://docs.starrocks.io/docs/deployment/deploy_manually.md). * [Deploy StarRocks on Kubernetes with operator](https://docs.starrocks.io/docs/deployment/sr_operator.md). * [Deploy StarRocks on Kubernetes with Helm](https://docs.starrocks.io/docs/deployment/helm.md). 6. Perform necessary [post-deployment setup](https://docs.starrocks.io/docs/deployment/post_deployment_setup.md) measures. Further setup measures are needed before your StarRocks cluster is put into production. These measures include securing the initial account and setting some performance-related system variables. #### Upgrade and Downgrade[​](#upgrade-and-downgrade "Direct link to Upgrade and Downgrade") If you plan to upgrade an existing StarRocks cluster to a later version rather than install StarRocks for the first time, see [Upgrade StarRocks](https://docs.starrocks.io/docs/deployment/upgrade.md) for information about upgrade procedures and issues that you should consider before upgrading. For instructions to downgrade your StarRocks cluster, see [Downgrade StarRocks](https://docs.starrocks.io/docs/deployment/downgrade.md). --- ### Deployment Prerequisites This topic describes the hardware and software requirements that your servers must meet before deploying StarRocks. For recommended hardware specifications of your StarRocks cluster, see [Plan your StarRocks cluster](https://docs.starrocks.io/docs/deployment/plan_cluster.md). #### Hardware[​](#hardware "Direct link to Hardware") ##### CPU[​](#cpu "Direct link to CPU") StarRocks relies on AVX2 instruction sets to fully unleash its vectorization capability. Therefore, in a production environment, we highly recommend you deploy StarRocks on machines with x86 architecture CPUs. You can run the following command in your terminal to check if the CPUs on your machines support the AVX2 instruction sets: ```bash cat /proc/cpuinfo | grep avx2 ``` ##### Memory[​](#memory "Direct link to Memory") No specific requirement is imposed on memory kits used for StarRocks. See [Plan StarRocks cluster - CPU and Memory](https://docs.starrocks.io/docs/deployment/plan_cluster.md#cpu-and-memory) for the recommended memory size. ##### Storage[​](#storage "Direct link to Storage") StarRocks supports both HDD and SSD as storage medium. If your applications require real-time data analytics, intensive data scans, or random disk access, we strongly recommend you use SSD storage. If your applications involve [Primary Key tables](https://docs.starrocks.io/docs/table_design/table_types/primary_key_table.md) with the persistent index, you must use SSD storage. ##### Network[​](#network "Direct link to Network") We recommend that you use 10 Gigabit Ethernet networking to ensure stable data transmission across nodes within your StarRocks cluster. #### Operating System[​](#operating-system "Direct link to Operating System") StarRocks supports deployments on Red Hat Enterprise Linux 7.9, CentOS Linux 7.9 or Ubuntu Linux 22.04. #### Software[​](#software "Direct link to Software") You must install the corresponding JDK version on the server to run StarRocks. * For StarRocks v3.3 and v3.4, use JDK 11 or later. * For StarRocks v3.5 and later, use JDK 17 or later. important StarRocks does not support JRE. --- ### Downgrade StarRocks This topic describes how to downgrade your StarRocks cluster. If an exception occurs after you upgrade a StarRocks cluster, you can downgrade it to the earlier version to quickly recover the cluster. #### Overview[​](#overview "Direct link to Overview") Review the information in this section before downgrading. Perform any recommended actions. ##### Downgrade paths[​](#downgrade-paths "Direct link to Downgrade paths") * **For patch version downgrade** You can downgrade your StarRocks cluster across patch versions, for example, from v3.5.11 directly to v3.5.6. * **For minor version downgrade** For compatibility and safety reasons, we strongly recommend you downgrade your StarRocks cluster **consecutively from one minor version to another**. For example, to downgrade a StarRocks v3.5 cluster to v3.2, you need to downgrade it in the following order: v3.5.x --> v3.4.x --> v3.3.x --> v3.2.x. * **For major version downgrade** You can only downgrade your StarRocks v4.1 cluster to v4.0.6 and later versions. warning **Downgrade Notes** * After upgrading StarRocks to v4.1, DO NOT downgrade to any v4.0 version below v4.0.6. Due to internal changes in data layout introduced in v4.1 (related to tablet splitting and distribution mechanisms), clusters upgraded to v4.1 may generate metadata and storage structures that are not fully compatible with earlier versions. As a result, downgrade from v4.1 is only supported to v4.0.6 or later. Downgrading to versions prior to v4.0.6 is not supported. This limitation is due to backward compatibility constraints in how earlier versions interpret tablet layout and distribution metadata. ##### Downgrade procedure[​](#downgrade-procedure "Direct link to Downgrade procedure") StarRocks' downgrade procedure is the reverse order of the [upgrade procedure](https://docs.starrocks.io/docs/deployment/upgrade.md#upgrade-procedure). Therefore, you need to **downgrade** **FEs** **first and then BEs and CNs**. Downgrading them in the wrong order may lead to incompatibility between FEs and BEs/CNs, and thereby cause the service to crash. For FE nodes, you must first downgrade all Follower FE nodes before downgrading the Leader FE node. #### Before you begin[​](#before-you-begin "Direct link to Before you begin") During preparation, you must perform the compatibility configuration if you are up for a minor or major version downgrade. You also need to perform the downgrade availability test on one of the FEs or BEs before downgrading all nodes in the cluster. ##### Perform compatibility configuration[​](#perform-compatibility-configuration "Direct link to Perform compatibility configuration") If you want to downgrade your StarRocks cluster to an earlier minor or major version, you must perform the compatibility configuration. In addition to the universal compatibility configuration, detailed configurations vary depending on the version of the StarRocks cluster you downgrade from. * **Universal compatibility configuration** Before downgrading your StarRocks cluster, you must disable tablet clone. You can skip this step if you have disabled the balancer. ```sql ADMIN SET FRONTEND CONFIG ("tablet_sched_max_scheduling_tablets" = "0"); ADMIN SET FRONTEND CONFIG ("tablet_sched_max_balancing_tablets" = "0"); ADMIN SET FRONTEND CONFIG ("disable_balance"="true"); ADMIN SET FRONTEND CONFIG ("disable_colocate_balance"="true"); ``` After the downgrade, you can enable tablet clone again if the status of all BE nodes becomes `Alive`. ```sql ADMIN SET FRONTEND CONFIG ("tablet_sched_max_scheduling_tablets" = "10000"); ADMIN SET FRONTEND CONFIG ("tablet_sched_max_balancing_tablets" = "500"); ADMIN SET FRONTEND CONFIG ("disable_balance"="false"); ADMIN SET FRONTEND CONFIG ("disable_colocate_balance"="false"); ``` #### Downgrade FE[​](#downgrade-fe "Direct link to Downgrade FE") note To downgrade a cluster from v3.3.0 or later to v3.2, follow these steps before downgrading: 1. Ensure that all ALTER TABLE SCHEMA CHANGE transactions initiated in the v3.3 cluster are either completed or canceled before downgrading. 2. Clear all transaction history by executing the following command: ```sql ADMIN SET FRONTEND CONFIG ("history_job_keep_max_second" = "0"); ``` 3. Verify that there are no remaining historical records by running the following command: ```sql SHOW PROC '/jobs//schema_change'; ``` After the compatibility configuration and the availability test, you can downgrade the FE nodes. You must first downgrade the Follower FE nodes and then the Leader FE node. 1. Create a metadata snapshot. a. Run [ALTER SYSTEM CREATE IMAGE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md) to create a metadata snapshot. b. You can check whether the image file has been synchronized by viewing the log file **fe.log** of the Leader FE. A record of log like "push image.\* from subdir \[] to other nodes. totally xx nodes, push successful xx nodes" suggests that the image file has been successfully synchronized. 2. Navigate to the working directory of the FE node and stop the node. ```bash # Replace with the deployment directory of the FE node. cd /fe ./bin/stop_fe.sh ``` 3. Replace the original deployment files under **bin**, **lib**, and **spark-dpp** with the ones of the earlier version. ```bash mv lib lib.bak mv bin bin.bak mv spark-dpp spark-dpp.bak cp -r /tmp/StarRocks-x.x.x/fe/lib . cp -r /tmp/StarRocks-x.x.x/fe/bin . cp -r /tmp/StarRocks-x.x.x/fe/spark-dpp . ``` 4. Start the FE node. ```bash ./bin/start_fe.sh --daemon ``` 5. Check if the FE node is started successfully. ```bash ps aux | grep StarRocksFE ``` 6. Repeat the above Step 2 to Step 5 to downgrade other Follower FE nodes, and finally the Leader FE node. > **CAUTION** > > Suppose you have downgraded your cluster after a failed upgrade and you want to upgrade the cluster again, for example, 3.5->4.0->3.5->4.0. To prevent metadata upgrade failure for some Follower FEs, repeat Step 1 to trigger a new snapshot before upgrading. #### Downgrade BE[​](#downgrade-be "Direct link to Downgrade BE") Having downgraded the FE nodes, you can then downgrade the BE nodes in the cluster. 1. Navigate to the working directory of the BE node and stop the node. ```bash # Replace with the deployment directory of the BE node. cd /be ./bin/stop_be.sh ``` 2. Replace the original deployment files under **bin** and **lib** with the ones of the earlier version. ```bash mv lib lib.bak mv bin bin.bak cp -r /tmp/StarRocks-x.x.x/be/lib . cp -r /tmp/StarRocks-x.x.x/be/bin . ``` 3. Start the BE node. ```bash ./bin/start_be.sh --daemon ``` 4. Check if the BE node is started successfully. ```bash ps aux | grep starrocks_be ``` 5. Repeat the above procedures to downgrade other BE nodes. #### Downgrade CN[​](#downgrade-cn "Direct link to Downgrade CN") 1. Navigate to the working directory of the CN node and stop the node gracefully. ```bash # Replace with the deployment directory of the CN node. cd /be ./bin/stop_cn.sh --graceful ``` 2. Replace the original deployment files under **bin** and **lib** with the ones of the earlier version. ```bash mv lib lib.bak mv bin bin.bak cp -r /tmp/StarRocks-x.x.x/be/lib . cp -r /tmp/StarRocks-x.x.x/be/bin . ``` 3. Start the CN node. ```bash ./bin/start_cn.sh --daemon ``` 4. Check if the CN node is started successfully. ```bash ps aux | grep starrocks_be ``` 5. Repeat the above procedures to downgrade other CN nodes. --- ### Check environment configurations This topic lists all environment and system configuration items that you must check and set before deploying StarRocks. Setting these configuration items properly allows your StarRocks cluster to work with high availability and performance. #### Ports[​](#ports "Direct link to Ports") StarRocks uses specific ports for different services. Check whether these ports are occupied on each instance if you have deployed other services on these instances. ##### FE ports[​](#fe-ports "Direct link to FE ports") On the instances used for the FE deployment, you need to check the following ports: * `8030`: FE HTTP server port (`http_port`) * `9020`: FE Thrift server port (`rpc_port`) * `9030`: FE MySQL server port (`query_port`) * `9010`: FE internal communication port (`edit_log_port`) * `6090`: FE cloud-native metadata server RPC listen port (`cloud_native_meta_port`) Run the following commands on the FE instances to check whether these ports are occupied: ```bash netstat -tunlp | grep 8030 netstat -tunlp | grep 9020 netstat -tunlp | grep 9030 netstat -tunlp | grep 9010 netstat -tunlp | grep 6090 ``` If any of the above ports are occupied, you must find alternatives and specify them later when you deploy FE nodes. For detailed instructions, see [Deploy StarRocks - Start the Leader FE node](https://docs.starrocks.io/docs/deployment/deploy_manually.md#step-1-start-the-leader-fe-node). ##### BE ports[​](#be-ports "Direct link to BE ports") On the instances used for the BE deployment, you need to check the following ports: * `9060`: BE Thrift server port (`be_port`) * `8040`: BE HTTP server port (`be_http_port`) * `9050`: BE heartbeat service port (`heartbeat_service_port`) * `8060`: BE bRPC port (`brpc_port`) * `9070`: An extra agent service port for BE and CN (`starlet_port`) Run the following commands on the BE instances to check whether these ports are occupied: ```bash netstat -tunlp | grep 9060 netstat -tunlp | grep 8040 netstat -tunlp | grep 9050 netstat -tunlp | grep 8060 netstat -tunlp | grep 9070 ``` If any of the above ports are occupied, you must find alternatives and specify them later when you deploy BE nodes. For detailed instructions, see [Deploy StarRocks - Start the BE service](https://docs.starrocks.io/docs/deployment/deploy_manually.md#step-2-start-the-be-service). ##### CN ports[​](#cn-ports "Direct link to CN ports") On the instances used for the CN deployment, you need to check the following ports: * `9060`: CN Thrift server port (`be_port`) * `8040`: CN HTTP server port (`be_http_port`) * `9050`: CN heartbeat service port (`heartbeat_service_port`) * `8060`: CN bRPC port (`brpc_port`) * `9070`: An extra agent service port for BE and CN (`starlet_port`) Run the following commands on the CN instances to check whether these ports are occupied: ```bash netstat -tunlp | grep 9060 netstat -tunlp | grep 8040 netstat -tunlp | grep 9050 netstat -tunlp | grep 8060 netstat -tunlp | grep 9070 ``` If any of the above ports are occupied, you must find alternatives and specify them later when you deploy CN nodes. For detailed instructions, see [Deploy StarRocks - Start the CN service](https://docs.starrocks.io/docs/deployment/deploy_shared_data_manually.md#step-2-start-the-cn-service). #### Hostnames[​](#hostnames "Direct link to Hostnames") If you want to [enable FQDN access](https://docs.starrocks.io/docs/administration/management/enable_fqdn.md) for your StarRocks cluster, you must assign a hostname to each instance. In the file **/etc/hosts** on each instance, you must specify the IP addresses and corresponding hostnames of all the other instances in the cluster. important All IP addresses in the file **/etc/hosts** must be unique. #### JDK configuration[​](#jdk-configuration "Direct link to JDK configuration") StarRocks relies on the environment variable `JAVA_HOME` to locate the Java dependency on the instance. Run the following command to check the environment variable `JAVA_HOME`: ```bash echo $JAVA_HOME ``` Follow these steps to set `JAVA_HOME`: 1. Set `JAVA_HOME` in the file **/etc/profile**: ```bash sudo vi /etc/profile # Replace with the path where JDK is installed. export JAVA_HOME= export PATH=$PATH:$JAVA_HOME/bin ``` 2. Bring the change into effect: ```bash source /etc/profile ``` Run the following command to verify the change: ```bash java -version ``` #### CPU scaling governor[​](#cpu-scaling-governor "Direct link to CPU scaling governor") This configuration item is **optional**. You can skip it if your CPU does not support the scaling governor. The CPU scaling governor controls the CPU power mode. If your CPU supports it, we recommend you set it to `performance` for better CPU performance: ```bash echo 'performance' | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor ``` #### Memory configurations[​](#memory-configurations "Direct link to Memory configurations") ##### Memory Overcommit[​](#memory-overcommit "Direct link to Memory Overcommit") Memory Overcommit allows the operating system to overcommit memory resources to processes. We recommend you enable Memory Overcommit. ```bash # Modify the configuration file. cat >> /etc/sysctl.conf << EOF vm.overcommit_memory=1 EOF # Bring the change into effect. sysctl -p ``` ##### Transparent Huge Pages[​](#transparent-huge-pages "Direct link to Transparent Huge Pages") Transparent Huge Pages are enabled by default. We recommend you disable this feature because it can interfere with the memory allocator, and thereby lead to a drop in performance. ```bash # Change the configuration temporarily. echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/defrag # Change the configuration permanently. cat >> /etc/rc.d/rc.local << EOF if test -f /sys/kernel/mm/transparent_hugepage/enabled; then echo madvise > /sys/kernel/mm/transparent_hugepage/enabled fi if test -f /sys/kernel/mm/transparent_hugepage/defrag; then echo madvise > /sys/kernel/mm/transparent_hugepage/defrag fi EOF chmod +x /etc/rc.d/rc.local ``` ##### Swap Space[​](#swap-space "Direct link to Swap Space") We recommend you disable Swap Space. Follow these steps to check and disable Swap Space: 1. Disable Swap Space. ```sql swapoff / swapoff -a ``` 2. Delete the Swap Space information from the configuration file **/etc/fstab**. ```bash / swap swap defaults 0 0 ``` 3. Verify that Swap Space is disabled. ```bash free -m ``` ##### Swappiness[​](#swappiness "Direct link to Swappiness") We recommend you disable swappiness to eliminate its impact on performance. ```bash # Modify the configuration file. cat >> /etc/sysctl.conf << EOF vm.swappiness=0 EOF # Bring the change into effect. sysctl -p ``` #### Storage configurations[​](#storage-configurations "Direct link to Storage configurations") We recommend that you choose your suitable scheduler algorithm in accordance with the storage medium you use. You can run the following command to check the scheduler algorithm that you are using: ```bash cat /sys/block/${disk}/queue/scheduler # For example, run cat /sys/block/vdb/queue/scheduler ``` We recommend you use the mq-deadline scheduler for SATA disks and the kyber scheduler algorithm for SSD and NVMe disks. ##### SATA[​](#sata "Direct link to SATA") The mq-deadline scheduler algorithm suits SATA disks. ```bash # Change the configuration temporarily. echo mq-deadline | sudo tee /sys/block/${disk}/queue/scheduler # Change the configuration permanently. cat >> /etc/rc.d/rc.local << EOF echo mq-deadline | sudo tee /sys/block/${disk}/queue/scheduler EOF chmod +x /etc/rc.d/rc.local ``` ##### SSD and NVMe[​](#ssd-and-nvme "Direct link to SSD and NVMe") * If your NVMe or SSD disks support the kyber scheduler algorithm: ```bash # Change the configuration temporarily. echo kyber | sudo tee /sys/block/${disk}/queue/scheduler # Change the configuration permanently. cat >> /etc/rc.d/rc.local << EOF echo kyber | sudo tee /sys/block/${disk}/queue/scheduler EOF chmod +x /etc/rc.d/rc.local ``` * If your NVMe or SSD disks support the none (or noop) scheduler. ```bash # Change the configuration temporarily. echo none | sudo tee /sys/block/vdb/queue/scheduler # Change the configuration permanently. cat >> /etc/rc.d/rc.local << EOF echo none | sudo tee /sys/block/${disk}/queue/scheduler EOF chmod +x /etc/rc.d/rc.local ``` #### SELinux[​](#selinux "Direct link to SELinux") We recommend you disable SELinux. ```bash # Change the configuration temporarily. setenforce 0 # Change the configuration permanently. sed -i 's/SELINUX=.*/SELINUX=disabled/' /etc/selinux/config sed -i 's/SELINUXTYPE/#SELINUXTYPE/' /etc/selinux/config ``` #### Firewall[​](#firewall "Direct link to Firewall") Open the internal ports for FE nodes, BE nodes, and Broker if your firewall is enabled. ```bash systemctl stop firewalld.service systemctl disable firewalld.service ``` #### LANG variable[​](#lang-variable "Direct link to LANG variable") Run the following command to check and configure the LANG variable manually: ```bash # Modify the configuration file. echo "export LANG=en_US.UTF8" >> /etc/profile # Bring the change into effect. source /etc/profile ``` #### Time zone[​](#time-zone "Direct link to Time zone") Set this item in accordance with your actual time zone. The following example sets the time zone to `/Asia/Shanghai`. ```bash cp -f /usr/share/zoneinfo/Asia/Shanghai /etc/localtime hwclock ``` #### ulimit configurations[​](#ulimit-configurations "Direct link to ulimit configurations") Problems can occur with StarRocks if the values of **max file descriptors** and **max user processes** are abnormally small. We recommend you enlarge these values. ```bash cat >> /etc/security/limits.conf << EOF * soft nproc 65535 * hard nproc 65535 * soft nofile 655350 * hard nofile 655350 * soft stack unlimited * hard stack unlimited * hard memlock unlimited * soft memlock unlimited EOF cat >> /etc/security/limits.d/20-nproc.conf << EOF * soft nproc 65535 root soft nproc 65535 EOF ``` #### File system configuration[​](#file-system-configuration "Direct link to File system configuration") We recommend you use the ext4 or xfs journaling file system. You can run the following command to check the mount type: ```bash df -Th ``` #### Network configuration[​](#network-configuration "Direct link to Network configuration") ##### tcp\_abort\_on\_overflow[​](#tcp_abort_on_overflow "Direct link to tcp_abort_on_overflow") Allow the system to reset new connections if the system is currently overflowed with new connection attempts that the daemon(s) can not handle: ```bash # Modify the configuration file. cat >> /etc/sysctl.conf << EOF net.ipv4.tcp_abort_on_overflow=1 EOF # Bring the change into effect. sysctl -p ``` ##### somaxconn[​](#somaxconn "Direct link to somaxconn") Specify the maximum number of connection requests queued for any listening socket to `1024`: ```bash # Modify the configuration file. cat >> /etc/sysctl.conf << EOF net.core.somaxconn=1024 EOF # Bring the change into effect. sysctl -p ``` #### NTP configuration[​](#ntp-configuration "Direct link to NTP configuration") You must configure time synchronization between nodes within your StarRocks cluster to ensure linear consistency of transactions. You can either use the internet time service provided by pool.ntp.org, or use the NTP service built in an offline environment. For example, you can use the NTP service provided by your cloud service provider. 1. Check if the NTP time server or Chrony service exists. ```bash rpm -qa | grep ntp systemctl status chronyd ``` 2. Install the NTP service if there is not one. ```bash sudo yum install ntp ntpdate && \ sudo systemctl start ntpd.service && \ sudo systemctl enable ntpd.service ``` 3. Check the NTP service. ```bash systemctl list-unit-files | grep ntp ``` 4. Check the connectivity and monitoring status of the NTP service. ```bash netstat -tunlp | grep ntp ``` 5. Check if your application is synchronized with the NTP server. ```bash ntpstat ``` 6. Check the state of all the configured NTP servers in your network. ```bash ntpq -p ``` #### High concurrency configurations[​](#high-concurrency-configurations "Direct link to High concurrency configurations") If your StarRocks cluster has a high load concurrency, we recommend you set the following configurations. ##### max\_map\_count[​](#max_map_count "Direct link to max_map_count") Specify the maximum number of memory map areas a process may have as `262144`: ```bash # Modify the configuration file. cat >> /etc/sysctl.conf << EOF vm.max_map_count = 262144 EOF # Bring the change into effect. sysctl -p ``` ##### Other[​](#other "Direct link to Other") ```bash echo 120000 > /proc/sys/kernel/threads-max echo 200000 > /proc/sys/kernel/pid_max ``` --- ### Feature Support: Shared-data Clusters tip Each of the features below lists the version number that they were added in. If you are deploying a new cluster please deploy the latest patch release of version 3.2 or higher. #### Overview[​](#overview "Direct link to Overview") Shared-data StarRocks clusters feature a disaggregated storage and compute architecture. This allows data to be stored in remote storage, leading to lower storage costs, optimized resource isolation, and better service elasticity compared to a shared-nothing cluster. This document outlines the feature support for shared-data clusters, covering deployment methods, storage configurations, caching mechanisms, Compaction, Primary Key table functionalities, and performance test results. #### Deployment[​](#deployment "Direct link to Deployment") Shared-data clusters support deployments on physical/virtual machines and on Kubernetes via Operator. Both deployment solutions have the following limitations: * Mixed deployment of shared-nothing and shared-data mode is not supported. * The transformation from a shared-nothing cluster to a shared-data cluster or vice versa is not supported. * Heterogeneous deployments are not supported, meaning the hardware specifications of all CN nodes within a cluster must be the same. ##### StarRocks Kubernetes Operator[​](#starrocks-kubernetes-operator "Direct link to StarRocks Kubernetes Operator") StarRocks offers the [StarRocks Kubernetes Operator](https://github.com/StarRocks/starrocks-kubernetes-operator/releases) for shared-data deployment on Kubernetes. You can scale shared-data clusters by the following methods: * Manual operations. * Automatic scaling using Kubernetes HPA (Horizontal Pod Autoscaler) strategies. #### Storage[​](#storage "Direct link to Storage") Shared-data clusters support building storage volumes on HDFS and object storage. ##### HDFS[​](#hdfs "Direct link to HDFS") ###### Location[​](#location "Direct link to Location") StarRocks supports the following locations for HDFS storage volume: * HDFS: `hdfs://:/` > **NOTE** > > From v3.2, storage volumes support HDFS clusters with the NameNode HA mode enabled. * WebHDFS (Supported from v3.2): `webhdfs://:/` * ViewFS (Supported from v3.2): `viewfs:///` ###### Authentication[​](#authentication "Direct link to Authentication") StarRocks supports the following authentication methods for HDFS storage volume: * Basic * Username (Supported from v3.2) * Kerberos Ticket Cache (Supported from v3.2) > **NOTE** > > StarRocks does not support automatic ticket refresh. You need to set up crontab tasks to refresh the ticket. Authentication using Kerberos Keytab and Principal ID is not yet supported. ###### Usage notes[​](#usage-notes "Direct link to Usage notes") StarRocks supports storage volumes on HDFS and object storage. However, only one HDFS storage volume is allowed in each StarRocks instance. Creating multiple HDFS storage volumes may cause unknown behaviors of StarRocks. ##### Object storage[​](#object-storage "Direct link to Object storage") ###### Location[​](#location-1 "Direct link to Location") StarRocks supports the following object storage services for storage volumes: * S3-compatible object storage services: `s3://` * AWS S3 * GCS, OSS, OBS, COS, TOS, KS3, MinIO, and Ceph S3 * Azure Blob Storage (Supported from v3.1.1): `azblob://` * Azure Data Lake Storage Gen2 (Supported from v3.4.1): `adls2:///` ###### Authentication[​](#authentication-1 "Direct link to Authentication") StarRocks supports the following authentication methods for different object storage services: * AWS S3 * AWS SDK * IAM user-based Credential * Instance Profile * Assumed Role * GCS, OSS, OBS, COS, TOS, KS3, MinIO, and Ceph S3 * Access Key pair * Azure Blob Storage * Shared Key * Shared Access Signatures (SAS) * Azure Data Lake Storage Gen2 * Shared Key * Shared Access Signatures (SAS) ###### Partitioned Prefix[​](#partitioned-prefix "Direct link to Partitioned Prefix") From v3.2.4, StarRocks supports creating storage volumes with the Partitioned Prefix feature for S3-compatible object storage systems. When this feature is enabled, StarRocks distributes the data into multiple partitions (sub-paths) under the bucket. It can easily multiply StarRocks' read and write performance on data files stored in the bucket. ##### Storage volumes[​](#storage-volumes "Direct link to Storage volumes") * From v3.1.0 onwards, storage volumes can be created using the CREATE STORAGE VOLUME statement, and this method is recommended in later versions. * The internal catalog `default_catalog` in shared-data clusters uses the default storage volume for data persistence. You can assign different storage volumes for databases and tables in `default_catalog` by setting the property `storage_volume`. If not configured, the property `storage_volume` is inherited in the order of catalog, database, and table. * Currently, storage volumes can be used only for storing data in cloud-native tables. Future support will include external storage management, data loading, and backup capabilities. #### Cache[​](#cache "Direct link to Cache") ##### Cache types[​](#cache-types "Direct link to Cache types") ###### File Cache[​](#file-cache "Direct link to File Cache") File Cache was the initial caching mechanism introduced along with the shared-data cluster. It loads the cache at the segment file level. File Cache is not recommended in v3.1.7, v3.2.3, and later versions. ###### Data Cache[​](#data-cache "Direct link to Data Cache") Data Cache is supported from v3.1.7 and v3.2.3 onwards to replace File Cache in earlier versions. Data Cache loads data from remote storage in blocks (on the order of MBs) on demand, without needing to load the entire file. It is recommended in the later versions and enabled by default in v3.2.3 and later. ###### Data Cache Warmup[​](#data-cache-warmup "Direct link to Data Cache Warmup") StarRocks v3.3.0 introduces the Data Cache Warmup feature to accelerate queries in data lakes and shared-data clusters. Data Cache Warmup is an active process of populating the cache. By executing CACHE SELECT, you can proactively fetch the desired data from remote storage in advance. ##### Configurations[​](#configurations "Direct link to Configurations") * Table properties: * `datacache.enable`: Whether to enable the local disk cache. Default: `true`. * `datacache.partition_duration`: The validity duration of the cached data. * BE configurations: * `starlet_use_star_cache`: Whether to enable Data Cache. * `starlet_star_cache_disk_size_percent`: The percentage of disk capacity that Data Cache can use at most in a shared-data cluster. ##### Capabilities[​](#capabilities "Direct link to Capabilities") * Data loading generates a local cache, whose eviction is only managed by the cache capacity control mechanism instead of `partition_duration`. * StarRocks supports setting up regular tasks for Data Cache Warmup. ##### Limitations[​](#limitations "Direct link to Limitations") * StarRocks does not support multiple replicas for cached data. #### Compaction[​](#compaction "Direct link to Compaction") ##### Observability[​](#observability "Direct link to Observability") ###### Partition-level Compaction status[​](#partition-level-compaction-status "Direct link to Partition-level Compaction status") From v3.1.9 onwards, you can view the Compaction status of partitions by querying `information_schema.partitions_meta`. We recommend monitoring the following key metrics: * **AvgCS**: Average Compaction score of all tablets in the partition. * **MaxCS**: Maximum Compaction score among all tablets in the partition. ###### Compaction task status[​](#compaction-task-status "Direct link to Compaction task status") From v3.2.0 onwards, you can view the status and progress of Compaction tasks by querying `information_schema.be_cloud_native_compactions`. We recommend monitoring the following key metrics: * **PROGRESS**: Current Compaction progress (in percentage) of the tablet. * **STATUS**: The status of the compaction task. If any error occurs, detailed error messages will be returned in this field. ##### Cancelling Compaction tasks[​](#cancelling-compaction-tasks "Direct link to Cancelling Compaction tasks") You can cancel specific compaction tasks using the CANCEL COMPACTION statement. Example: ```sql CANCEL COMPACTION WHERE TXN_ID = 123; ``` > **NOTE** > > The CANCEL COMPACTION statement must be executed on the Leader FE node. ##### Manual Compaction[​](#manual-compaction "Direct link to Manual Compaction") From v3.1, StarRocks offers a SQL statement for manual Compaction. You can specify the table or partitions for compaction. For more information, refer to [Manual Compaction](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md#manual-compaction). #### Primary Key tables[​](#primary-key-tables "Direct link to Primary Key tables") The following table lists the major features of Primary Key tables and their support status in shared-data clusters: | **Feature** | **Supported Version(s)** | **Description** | | ----------------------------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------- | | Primary Key tables | v3.1.0 | | | Primary Key index persistence | v3.2.0
v3.1.3
v3.3.2 | | | Partial Update | v3.1.0 | Shared-data clusters support Partial Update in Row mode from v3.1.0 onwards and in Column mode from v3.3.1 onwards. | | Conditional Update | v3.1.0 | Currently, the condition only supports 'Greater'. | | Hybrid row-column storage | ❌ | To be supported in future releases. | #### Query performance[​](#query-performance "Direct link to Query performance") The following test compares the query performance of a shared-data cluster with Data Cache disabled, one with Data Cache enabled, one that queries the dataset in Hive, and a shared-nothing cluster. ##### Hardware Specifications[​](#hardware-specifications "Direct link to Hardware Specifications") The cluster used in the test includes one FE node and five CN/BE nodes. The hardware specifications are as follows: | **VM provider** | Alibaba Cloud ECS | | --------------------- | ------------------- | | **FE node** | 8 Core 32 GB Memory | | **CN/BE node** | 8 Core 64 GB Memory | | **Network bandwidth** | 8 Gbits/s | | **Disk** | ESSD | ##### Software version[​](#software-version "Direct link to Software version") StarRocks v3.3.0 ##### Dataset[​](#dataset "Direct link to Dataset") SSB 1TB dataset note The dataset and queries used in this comparison are from the [Star Schema Benchmark](https://docs.starrocks.io/docs/benchmarking/SSB_Benchmarking.md). ##### Test Results[​](#test-results "Direct link to Test Results") The following table shows the performance test results on thirteen queries and the sum of each cluster. The unit of query latency is milliseconds (ms). | **Query** | **Shared-data Without Data Cache** | **Shared-data With Data Cache** | **Hive Catalog Without Data Cache** | **Shared-nothing** | | --------- | ---------------------------------- | ------------------------------- | ----------------------------------- | ------------------ | | **Q01** | 2742 | 858 | 9652 | 3555 | | **Q02** | 2714 | 704 | 8638 | 3183 | | **Q03** | 1908 | 658 | 8163 | 2980 | | **Q04** | 31135 | 8582 | 34604 | 7997 | | **Q05** | 26597 | 7806 | 29183 | 6794 | | **Q06** | 21643 | 7147 | 24401 | 5602 | | **Q07** | 35271 | 15490 | 38904 | 19530 | | **Q08** | 24818 | 7368 | 27598 | 6984 | | **Q09** | 21056 | 6667 | 23587 | 5687 | | **Q10** | 2823 | 912 | 16663 | 3942 | | **Q11** | 50027 | 18947 | 52997 | 19636 | | **Q12** | 10300 | 4919 | 36146 | 8136 | | **Q13** | 7378 | 3386 | 23153 | 6380 | | **SUM** | 238412 | 83444 | 333689 | 100406 | ##### Conclusion[​](#conclusion "Direct link to Conclusion") * The query performance of the shared-data cluster with Data Cache disabled and Parallel Scan and I/O merge optimization enabled is **1.4 times** that of the cluster that queries Hive data. * The query performance of the shared-data cluster with Data Cache enabled and Parallel Scan and I/O merge optimization enabled is **1.2 times** that of the shared-nothing cluster. #### Other features to be supported[​](#other-features-to-be-supported "Direct link to Other features to be supported") * Full-text inverted index * Hybrid row-column storage * Global dictionary object * Backup and restore --- ### Deploy StarRocks with Helm [Helm](https://helm.sh/) is a package manager for Kubernetes. A [Helm Chart](https://helm.sh/docs/topics/charts/) is a Helm package and contains all of the resource definitions necessary to run an application on a Kubernetes cluster. This topic describes how to use Helm to automatically deploy a StarRocks cluster on a Kubernetes cluster. #### Before you begin[​](#before-you-begin "Direct link to Before you begin") * [Create a Kubernetes cluster](https://docs.starrocks.io/docs/deployment/sr_operator.md#create-kubernetes-cluster). * [Install Helm](https://helm.sh/docs/intro/quickstart/). #### Procedure[​](#procedure "Direct link to Procedure") 1. Add the Helm Chart Repo for StarRocks. The Helm Chart contains the definitions of the StarRocks Operator and the custom resource StarRocksCluster. 1. Add the Helm Chart Repo. ```bash helm repo add starrocks https://starrocks.github.io/starrocks-kubernetes-operator ``` 2. Update the Helm Chart Repo to the latest version. ```bash helm repo update ``` 3. View the Helm Chart Repo that you added. ```bash $ helm search repo starrocks NAME CHART VERSION APP VERSION DESCRIPTION starrocks/kube-starrocks 1.8.0 3.1-latest kube-starrocks includes two subcharts, starrock... starrocks/operator 1.8.0 1.8.0 A Helm chart for StarRocks operator starrocks/starrocks 1.8.0 3.1-latest A Helm chart for StarRocks cluster ``` 2. Use the default **[values.yaml](https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/helm-charts/charts/kube-starrocks/values.yaml)** of the Helm Chart to deploy the StarRocks Operator and StarRocks cluster, or create a YAML file to customize your deployment configurations. 1. Deployment with default configurations Run the following command to deploy the StarRocks Operator and the StarRocks cluster which consists of one FE and one BE: > Tip > > The default `values.yaml` is configured to deploy: > > * The operator pod with 1/2CPU and 0.8GB RAM > * One FE with 4GB RAM, 4 cores, and 15Gi disk > * One BE with 4GB RAM, 4 cores, and 1Ti disk > > If you do not have these resources available in your Kubernetes cluster then skip to the **Deployment with custom configurations** section and adjust the resources. ```bash $ helm install starrocks starrocks/kube-starrocks # If the following result is returned, the StarRocks Operator and StarRocks cluster are being deployed. NAME: starrocks LAST DEPLOYED: Tue Aug 15 15:12:00 2023 NAMESPACE: starrocks STATUS: deployed REVISION: 1 TEST SUITE: None ``` 3. Deployment with custom configurations * Create a YAML file, for example, **my-values.yaml**, and customize the configurations for the StarRocks Operator and StarRocks cluster in the YAML file. For the supported parameters and descriptions, see the comments in the default **[values.yaml](https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/helm-charts/charts/kube-starrocks/values.yaml)** of the Helm Chart. * Run the following command to deploy the StarRocks Operator and StarRocks cluster with the custom configurations in **my-values.yaml**. ```bash helm install -f my-values.yaml starrocks starrocks/kube-starrocks ``` Deployment takes a while. During this period, you can check the deployment status with: ```bash kubectl --namespace default get starrockscluster -l "cluster=kube-starrocks" ``` If the following result is returned, the deployment has been successfully completed. ```bash NAME PHASE FESTATUS BESTATUS CNSTATUS FEPROXYSTATUS kube-starrocks running running running ``` You can also run `kubectl get pods` to check the deployment status. If all Pods are in the `Running` state and all containers within the Pods are `READY`, the deployment has been successfully completed. ```bash kubectl get pods ``` ```bash NAME READY STATUS RESTARTS AGE kube-starrocks-be-0 1/1 Running 0 2m50s kube-starrocks-fe-0 1/1 Running 0 4m31s kube-starrocks-operator-69c5c64595-pc7fv 1/1 Running 0 4m50s ``` #### Next steps[​](#next-steps "Direct link to Next steps") * Access StarRocks cluster You can access the StarRocks cluster from inside and outside the Kubernetes cluster. For detailed instructions, see [Access StarRocks Cluster](https://docs.starrocks.io/docs/deployment/sr_operator.md#access-starrocks-cluster). * Manage StarRocks operator and StarRocks cluster * If you need to update the configurations of the StarRocks operator and StarRocks cluster, see [Helm Upgrade](https://helm.sh/docs/helm/helm_upgrade/). * If you need to uninstall the StarRocks Operator and StarRocks cluster, run the following command: ```bash helm uninstall starrocks ``` #### More information[​](#more-information "Direct link to More information") * The address of the GitHub repository: [starrocks-kubernetes-operator and kube-starrocks Helm Chart](https://github.com/StarRocks/starrocks-kubernetes-operator). * The docs in the GitHub repository provide more information, for example: * If you need to manage objects like the StarRocks cluster via the Kubernetes API, see [API reference](https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/doc/api.md). * If you need to mount persistent volumes to FE and BE pods to store FE metadata and logs, as well as BE data and logs, see [Mount Persistent Volumes by Helm Chart](https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/doc/mount_persistent_volume_howto.md#2-mounting-persistent-volumes-by-helm-chart). danger If persistent volumes are not mounted, the StarRocks Operator will use emptyDir to store FE metadata and logs, as well as BE data and logs. When containers restart, data will be lost. * If you need to set the root user password: * Manually set the root user's password after deploying the StarRocks cluster, see [Change root user password HOWTO](https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/doc/change_root_password_howto.md). * Automatically set the root user's password when deploying the StarRocks cluster, see [Initialize root user password](https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/doc/initialize_root_password_howto.md). * How to resolve the following error that occurs after a CREATE TABLE statement is executed in a StarRocks shared-data cluster. * **Error message** ```plaintext ERROR 1064 (HY000): Table replication num should be less than or equal to the number of available BE nodes. You can change this default by setting the replication_num table properties. Current alive backend is [10001]. , table=orders1, default_replication_num=3 ``` * **Solution** This may be because only one BE exists in that StarRocks shared-data cluster, which supports only one replica. However, the default number of replicas is 3. You can modify the number of replicas to 1 in PROPERTIES, such as, `PROPERTIES( "replication_num" = "1" )`. * The address of Helm Chart maintained by StarRocks on Artifact Hub: [kube-starrocks](https://artifacthub.io/packages/helm/kube-starrocks/kube-starrocks). --- ### Plan StarRocks Cluster This topic describes how to plan resources for your StarRocks cluster in production from the perspectives of node count, CPU core count, memory size, and storage size. #### Node count[​](#node-count "Direct link to Node count") StarRocks mainly consists of two types of components: FE nodes and BE/CN nodes. Each node must be deployed separately on a physical or virtual machine. ##### FE node count[​](#fe-node-count "Direct link to FE node count") FE nodes are mainly responsible for metadata management, client connection management, query planning, and query scheduling. In production, we recommend you deploy at least **THREE** Follower FE nodes in your StarRocks cluster to prevent single points of failure (SPOFs). The Leader FE will be automatically elected from these three followers. StarRocks uses the Raft protocol to manage the metadata across FE nodes. StarRocks elects a Leader FE node from all Follower FE nodes. Only the Leader FE node can write metadata. The other Follower FE nodes only update their metadata based on the logs from the Leader FE node. Each time the Leader FE node fails, StarRocks re-elects a new Leader FE node as long as more than half of the Follower FE nodes are alive. If your application generates highly concurrent query requests, you can add Observer FE nodes to your cluster. Observer FE nodes only process the query requests and do not participate in the election for the Leader FE node. ##### BE node count[​](#be-node-count "Direct link to BE node count") BE nodes are responsible for data storage and SQL execution in **shared-nothing** clusters. In production, we recommend you deploy at least **THREE** BE nodes in your StarRocks cluster to ensure high data reliability and service availability. A high-availability cluster of BEs is automatically formed when at least three BE nodes are deployed and added to your StarRocks cluster. The failure of one BE node will not affect the overall availability of the BE services. You can increase the number of BE nodes to enable your StarRocks cluster to process highly concurrent queries. ##### CN node count[​](#cn-node-count "Direct link to CN node count") CN nodes are responsible for data caching and SQL execution in **shared-data** clusters. You can increase the number of CN nodes to elastically scale compute resources in your StarRocks cluster. #### CPU and memory[​](#cpu-and-memory "Direct link to CPU and memory") Usually, the FE service does not consume a lot of CPU and memory resources. We recommend allocating 8 CPU cores and 16 GB RAM to each FE node. Unlike the FE service, the BE/CN service can be significantly CPU- and memory-intensive if your application works with highly concurrent or complex queries on a large dataset. Therefore, we recommend allocating 16 CPU cores and 64 GB RAM to each BE/CN node. #### Storage capacity[​](#storage-capacity "Direct link to Storage capacity") ##### FE storage[​](#fe-storage "Direct link to FE storage") Because FE nodes only maintain StarRocks' metadata in their storage, 100 GB of HDD storage is enough for each FE node in most scenarios. ##### BE storage[​](#be-storage "Direct link to BE storage") ###### Estimate initial storage space for BE[​](#estimate-initial-storage-space-for-be "Direct link to Estimate initial storage space for BE") The total storage space that your StarRocks cluster will need is simultaneously influenced by the size of your raw data, the data replica count, and the compression ratio of the data compression algorithm you use. With the following formula, you can estimate the total storage space you will need for all BE nodes: ```plain Total BE storage space = Raw data size * Replica count/Compression ratio Raw data size = Sum of the space taken up by all fields in a row * Row count ``` In StarRocks, data in a table is first divided into multiple partitions and then into multiple tablets. Tablets are the basic logical units of data management in StarRocks. To ensure high data reliability, you can maintain multiple replicas of each tablet and store them across different BEs. By default, StarRocks maintains three replicas. Currently, StarRocks supports four data compression algorithms, which are listed in order from higher to lower compression ratio: zlib, Zstandard (or zstd), LZ4, and Snappy. They can provide a compression ratio from 3:1 to 5:1. After determining the total storage space, you can simply divide it by the number of BE nodes in your cluster to estimate the average storage space per BE node. ###### Add extra storage as you go[​](#add-extra-storage-as-you-go "Direct link to Add extra storage as you go") If the BE storage space runs out as your raw data grows, you can supplement it by scaling your cluster vertically or horizontally, or simply scaling up your cloud storage. * Add new BE nodes to your StarRocks cluster You can add new BE nodes to your StarRocks cluster so that the data can be re-distributed evenly to more nodes. For detailed instructions, see [Scale your StarRocks cluster - Scale BE out](https://docs.starrocks.io/docs/administration/management/Scale_up_down.md). After new BE nodes are added, StarRocks automatically re-balances the data among all BE nodes. Such auto-balancing is supported on all table types. * Add extra storage volumes to your BE nodes You can also add extra storage volumes to existing BE nodes. For detailed instructions, see [Scale your StarRocks cluster - Scale BE up](https://docs.starrocks.io/docs/administration/management/Scale_up_down.md). After extra storage volumes are added, StarRocks automatically re-balances the data in all tables. * Add cloud storage If your StarRocks cluster is deployed on cloud, you can scale up your cloud storage on demand. For detailed instructions, contact your cloud provider. ##### CN storage[​](#cn-storage "Direct link to CN storage") Raw data in shared-data clusters is stored in remote storage. You can scale the storage space whenever needed. The local disk for a CN node is used to cache hot data for query acceleration. You can estimate the local disk space based on the hot data size of your daily business scenario. --- ### Post-deployment setup This topic describes tasks that you should perform after deploying StarRocks. Before getting your new StarRocks cluster into production, you must secure the initial account and set the necessary variables and properties to allow your cluster to run properly. #### Secure initial account[​](#secure-initial-account "Direct link to Secure initial account") Upon the creation of a StarRocks cluster, the initial `root` user of the cluster is generated automatically. The `root` user is granted the `root` privileges, which are the collection of all privileges within the cluster. We recommend you secure this user account and avoid using it in production to prevent misuse. StarRocks automatically assigns an empty password to the `root` user when the cluster is created. Follow these procedures to set a new password for the `root` user: 1. Connect to StarRocks via your MySQL client with the username `root` and an empty password. ```sh # Replace with the IP address (priority_networks) or FQDN # of the FE node you connect to, and replace # with the query_port (Default: 9030) you specified in fe.conf. mysql -h -P -uroot ``` 2. Reset the password of the `root` user by executing the following SQL: ```sql -- Replace with the password you want to assign to the root user. SET PASSWORD = PASSWORD('') ``` note * Keep the password properly after resetting it. If you forgot the password, see [Reset lost root password](https://docs.starrocks.io/docs/administration/user_privs/authentication/native_authentication.md#reset-lost-root-password) for detailed instructions. * After completing the post-deployment setup, you can create new users and roles to manage the privileges within your team. See [Manage user privileges](https://docs.starrocks.io/docs/administration/user_privs/authorization/User_privilege.md) for detailed instructions. #### Set necessary system variables[​](#set-necessary-system-variables "Direct link to Set necessary system variables") To allow your StarRocks cluster to work properly in production, you need to set the following system variables: ##### enable\_profile[​](#enable_profile "Direct link to enable_profile") ###### Description[​](#description "Direct link to Description") The boolean switch that controls whether to send the profile of a query for analysis. The default value is `false`, which means no profile is required. Setting this variable to `true` can affect the concurrency of StarRocks. ###### Recommended value[​](#recommended-value "Direct link to Recommended value") false * Set `enable_profile` to `false` globally: ```sql SET GLOBAL enable_profile = false; ``` ##### enable\_pipeline\_engine[​](#enable_pipeline_engine "Direct link to enable_pipeline_engine") ###### Description[​](#description-1 "Direct link to Description") The boolean switch that controls whether to enable the pipeline execution engine. `true` indicates enabled and `false` indicates the opposite. Default value: `true`. ###### Recommended value[​](#recommended-value-1 "Direct link to Recommended value") true * Set `enable_pipeline_engine` to `true` globally: ```sql SET GLOBAL enable_pipeline_engine = true; ``` ##### parallel\_fragment\_exec\_instance\_num[​](#parallel_fragment_exec_instance_num "Direct link to parallel_fragment_exec_instance_num") ###### Description[​](#description-2 "Direct link to Description") The number of instances used to scan nodes on each BE. The default value is `1`. ###### Recommended value[​](#recommended-value-2 "Direct link to Recommended value") If you have enabled the pipeline engine, you can set this variable to `1`. If you have not enabled the pipeline engine, you should set it to half the number of CPU cores. * Set `parallel_fragment_exec_instance_num` to `1` globally: ```sql SET GLOBAL parallel_fragment_exec_instance_num = 1; ``` For more information about system variables, see [System variables](https://docs.starrocks.io/docs/sql-reference/System_variable.md). #### Set user property[​](#set-user-property "Direct link to Set user property") If you have created new users in your cluster, you need to enlarge their maximum connection number (to `1000`, for example): ```sql -- Replace with the username you want to enlarge the maximum connection number for. ALTER USER '' SET PROPERTIES ("max_user_connections" = "1000"); ``` #### What to do next[​](#what-to-do-next "Direct link to What to do next") After deploying and setting up your StarRocks cluster, you can then proceed to design tables that best work for your scenarios. See [Understand StarRocks table design](https://docs.starrocks.io/docs/table_design.md) for detailed instructions on designing a table. --- ### Prepare deployment files This topic describes how to prepare StarRocks deployment files. Currently, the binary distribution packages StarRocks provides on [the StarRocks official website](https://www.starrocks.io/download/community) support deployments only on x86-based CPU. If you want to deploy StarRocks with the ARM-based CPU, you need to prepare the deployment files using the StarRocks Docker image. #### For x86-based CPU[​](#for-x86-based-cpu "Direct link to For x86-based CPU") From v3.1.14, v3.2.10, and v3.3.3, StarRocks binary distribution packages are named in the `StarRocks-{Version}-{OS}-{ARCH}.tar.gz` format, where `Version` is a number (for example, `3.3.3`) that indicates the version information of the binary distribution package, `OS` indicates the operating system (including `centos` and `ubuntu`), and `ARCH` indicates the CPU architecture (currently only `amd64` is available, which is equivalent to x86\_64). Make sure that you have chosen the correct version of the package. note In versions earlier than v3.1.14, v3.2.10, and v3.3.3, StarRocks provides binary distribution packages named in the `StarRocks-{Version}.tar.gz` format. Follow these steps to prepare deployment files for the x86-based platform: 1. Obtain the StarRocks binary distribution package directly from the [Download StarRocks](https://www.starrocks.io/download/community) page or by running the following command in your terminal: ```bash # Replace with the version of StarRocks you want to download, for example, 3.3.3, # and replace with centos or ubuntu. wget https://releases.starrocks.io/starrocks/StarRocks---amd64.tar.gz ``` 2. Extract the files in the package. ```bash # Replace with the version of StarRocks you want to download, for example, 3.3.3, # and replace with centos or ubuntu. tar -xzvf StarRocks---amd64.tar.gz ``` The package includes the following directories and files: | **Directory/File** | **Description** | | ------------------------ | -------------------------------------------- | | **apache\_hdfs\_broker** | The deployment directory of the Broker node. | | **fe** | The FE deployment directory. | | **be** | The BE deployment directory. | | **LICENSE.txt** | The StarRocks license file. | | **NOTICE.txt** | The StarRocks notice file. | 3. Dispatch the directory **fe** to all the FE instances and the directory **be** to all the BE or CN instances for [manual deployment](https://docs.starrocks.io/docs/deployment/deploy_manually.md). #### For ARM-based CPU[​](#for-arm-based-cpu "Direct link to For ARM-based CPU") ##### Prerequisites[​](#prerequisites "Direct link to Prerequisites") You must have [Docker Engine](https://docs.docker.com/engine/install/) (17.06.0 or later) installed on your machine. ##### Procedures[​](#procedures "Direct link to Procedures") From v3.1.14, v3.2.10, and v3.3.3, StarRocks provides Docker images in the `starrocks/artifacts-{OS}:{Version}` format, where `OS` indicates the operating system (including `centos7` and `ubuntu`), and `Version` is the version number (for example, `3.3.3`). Docker will automatically identify your CPU architecture and pull the corresponding image. Make sure that you have chosen the correct version of the image. note In versions earlier than v3.1.14, v3.2.10, and v3.3.3, StarRocks provides Docker images in the repositories `starrocks/artifacts-ubuntu` and `starrocks/artifacts-centos7`. 1. Download a StarRocks Docker image from [StarRocks Docker Hub](https://hub.docker.com/u/starrocks?page=1\&search=artifacts). You can choose a specific version based on the tag of the image. ```bash # Replace with centos7 or ubuntu, # and replace with the version of StarRocks you want to download, for example, 3.3.3. # e.g. docker pull starrocks/artifacts-centos7:3.3.3 or docker pull starrocks/artifacts-ubuntu:3.3.3 docker pull starrocks/artifacts-: ``` 2. Copy the StarRocks deployment files from the Docker image to your host machine by running the following command: ```bash # Replace with centos7 or ubuntu, # and replace with the version of StarRocks you want to download, for example, 3.3.3. docker run --rm starrocks/artifacts-: \ tar -cf - -C /release . | tar -xvf - ``` 3. Dispatch the deployment files to the corresponding instances for [manual deployment](https://docs.starrocks.io/docs/deployment/deploy_manually.md). --- ### StarRocks Kubernetes Operator Automate deployment and management of a StarRocks cluster on a Kubernetes cluster with the StarRocks Kubernetes Operator. note The StarRocks k8s operator was designed to be a level 2 operator. See to understand more about the capabilities of a level 2 operator. #### How it works[​](#how-it-works "Direct link to How it works") ![img](/assets/images/starrocks_operator-5c0529d834fcc0959839194ce33ec326.png) #### Before you begin[​](#before-you-begin "Direct link to Before you begin") ##### Create Kubernetes cluster[​](#create-kubernetes-cluster "Direct link to Create Kubernetes cluster") You can use the cloud-managed Kubernetes service, such as an [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/?nc1=h_ls) or [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) cluster, or a self-managed Kubernetes cluster. * Create an Amazon EKS cluster 1. Check that [the following command-line tools are installed in your environment](https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html): 1. Install and configure AWS command-line tool AWS CLI. 2. Install EKS cluster command-line tool eksctl. 3. Install the Kubernetes cluster command-line tool kubectl. 2. Use one of the following methods to create an EKS cluster: 1. [Use eksctl to quickly create an EKS cluster](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html). 2. [Manually create an EKS cluster with the AWS console and AWS CLI](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-console.html). * Create a GKE cluster Before you start to create a GKE cluster, make sure that you complete all the [prerequisites](https://docs.cloud.google.com/kubernetes-engine/docs/deploy-app-cluster#before-you-begin). Then follow the instructions provided in [Create a GKE cluster](https://docs.cloud.google.com/kubernetes-engine/docs/deploy-app-cluster#create_cluster) to create a GKE cluster. * Create a self-managed Kubernetes cluster Follow the instructions provided in [Bootstrapping clusters with kubeadm](https://kubernetes.io/docs) to create a self-managed Kubernetes cluster. You can use Minikube and Docker Desktop to create a single-node private Kubernetes cluster with minimum steps. ##### Deploy StarRocks Kubernetes Operator[​](#deploy-starrocks-kubernetes-operator "Direct link to Deploy StarRocks Kubernetes Operator") 1. Add the custom resource StarRocksCluster. ```bash kubectl apply -f https://raw.githubusercontent.com/StarRocks/starrocks-kubernetes-operator/main/deploy/starrocks.com_starrocksclusters.yaml ``` 2. Deploy the StarRocks Operator. You can choose to deploy the StarRocks Operator by using a default configuration file or a custom configuration file. 1. Deploy the StarRocks Operator by using a default configuration file. ```bash kubectl apply -f https://raw.githubusercontent.com/StarRocks/starrocks-kubernetes-operator/main/deploy/operator.yaml ``` The StarRocks Operator is deployed to the namespace `starrocks` and manages all StarRocks clusters under all namespaces. 2. Deploy the StarRocks Operator by using a custom configuration file. * Download the configuration file **operator.yaml**, which is used to deploy the StarRocks Operator. ```bash curl -O https://raw.githubusercontent.com/StarRocks/starrocks-kubernetes-operator/main/deploy/operator.yaml ``` * Modify the configuration file **operator.yaml** to suit your needs. * Deploy the StarRocks Operator. ```bash kubectl apply -f operator.yaml ``` 3. Check the running status of the StarRocks Operator. If the pod is in the `Running` state and all containers inside the pod are `READY`, the StarRocks Operator is running as expected. ```bash $ kubectl -n starrocks get pods NAME READY STATUS RESTARTS AGE starrocks-controller-65bb8679-jkbtg 1/1 Running 0 5m6s ``` > **NOTE** > > If you customize the namespace in which the StarRocks Operator is located, you need to replace`starrocks` with the name of your customized namespace. #### Deploy StarRocks Cluster[​](#deploy-starrocks-cluster "Direct link to Deploy StarRocks Cluster") You can directly use the [sample configuration files](https://github.com/StarRocks/starrocks-kubernetes-operator/tree/main/examples/starrocks) provided by StarRocks to deploy a StarRocks cluster (an object instantiated by using the custom resource StarRocks Cluster). For example, you can use **starrocks-fe-and-be.yaml** to deploy a StarRocks cluster that contains three FE nodes and three BE nodes. ```bash kubectl apply -f https://raw.githubusercontent.com/StarRocks/starrocks-kubernetes-operator/main/examples/starrocks/starrocks-fe-and-be.yaml ``` The following table describes a few important fields in the **starrocks-fe-and-be.yaml** file. | **Field** | **Description** | | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Kind | The resource type of the object. The value must be `StarRocksCluster`. | | Metadata | Metadata, in which the following sub-fields are nested:- `name`: the name of the object. Each object name uniquely identifies an object of the same resource type.
- `namespace`: the namespace to which the object belongs. | | Spec | The expected status of the object. Valid values are `starRocksFeSpec`, `starRocksBeSpec`, and `starRocksCnSpec`. | You can also deploy the StarRocks cluster by using a modified configuration file. For supported fields and detailed descriptions, see [api.md](https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/doc/api.md). Deploying the StarRocks cluster takes a while. During this period, you can use the command `kubectl -n starrocks get pods` to check the starting status of the StarRocks cluster. If all the pods are in the `Running` state and all containers inside the pods are `READY`, the StarRocks cluster is running as expected. > **NOTE** > > If you customize the namespace in which the StarRocks cluster is located, you need to replace `starrocks` with the name of your customized namespace. ```bash $ kubectl -n starrocks get pods NAME READY STATUS RESTARTS AGE starrocks-controller-65bb8679-jkbtg 1/1 Running 0 22h starrockscluster-sample-be-0 1/1 Running 0 23h starrockscluster-sample-be-1 1/1 Running 0 23h starrockscluster-sample-be-2 1/1 Running 0 22h starrockscluster-sample-fe-0 1/1 Running 0 21h starrockscluster-sample-fe-1 1/1 Running 0 21h starrockscluster-sample-fe-2 1/1 Running 0 22h ``` tip If some pods cannot start after a long period of time, you can use `kubectl logs -n starrocks ` to view the log information or use `kubectl -n starrocks describe pod ` to view the event information to locate the problem. #### Manage StarRocks Cluster[​](#manage-starrocks-cluster "Direct link to Manage StarRocks Cluster") ##### Access StarRocks Cluster[​](#access-starrocks-cluster "Direct link to Access StarRocks Cluster") The components of the StarRocks cluster can be accessed through their associated Services, such as the FE Service. For detailed descriptions of Services and their access addresses, see [api.md](https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/doc/api.md) and [Services](https://kubernetes.io/docs/concepts/services-networking/service/). note * Only the FE Service is deployed by default. If you need to deploy the BE Service and CN Service, you need to configure `starRocksBeSpec` and `starRocksCnSpec` in the StarRocks cluster configuration file. * The name of a Service is `--service` by default, for example, `starrockscluster-sample-fe-service`. You can also specify the Service name in the spec of each component. ###### Access StarRocks Cluster from within Kubernetes Cluster[​](#access-starrocks-cluster-from-within-kubernetes-cluster "Direct link to Access StarRocks Cluster from within Kubernetes Cluster") From within the Kubernetes cluster, the StarRocks cluster can be accessed through the FE Service's ClusterIP. 1. Obtain the internal virtual IP address `CLUSTER-IP` and port `PORT(S)` of the FE Service. ```bash $ kubectl -n starrocks get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE be-domain-search ClusterIP None 9050/TCP 23m fe-domain-search ClusterIP None 9030/TCP 25m starrockscluster-sample-fe-service ClusterIP 10.100.162.xxx 8030/TCP,9020/TCP,9030/TCP,9010/TCP 25m ``` 2. Access the StarRocks cluster by using the MySQL client from within the Kubernetes cluster. ```bash mysql -h 10.100.162.xxx -P 9030 -uroot ``` ###### Access StarRocks Cluster from outside Kubernetes Cluster[​](#access-starrocks-cluster-from-outside-kubernetes-cluster "Direct link to Access StarRocks Cluster from outside Kubernetes Cluster") From outside the Kubernetes cluster, you can access the StarRocks cluster through the FE Service's LoadBalancer or NodePort. This topic uses LoadBalancer as an example: 1. Run the command `kubectl -n starrocks edit src starrockscluster-sample` to update the StarRocks cluster configuration file, and change the Service type of `starRocksFeSpec` to `LoadBalancer`. ```yaml starRocksFeSpec: image: starrocks/fe-ubuntu:3.0-latest replicas: 3 requests: cpu: 4 memory: 16Gi service: type: LoadBalancer # specified as LoadBalancer ``` 2. Obtain the IP address `EXTERNAL-IP` and port `PORT(S)` that the FE Service exposes to the outside. ```bash $ kubectl -n starrocks get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE be-domain-search ClusterIP None 9050/TCP 127m fe-domain-search ClusterIP None 9030/TCP 129m starrockscluster-sample-fe-service LoadBalancer 10.100.162.xxx a7509284bf3784983a596c6eec7fc212-618xxxxxx.us-west-2.elb.amazonaws.com 8030:30629/TCP,9020:32544/TCP,9030:32244/TCP,9010:32024/TCP 129m ClusterIP None 9030/TCP 23h ``` 3. Log in to your machine host and access the StarRocks cluster by using the MySQL client. ```bash mysql -h a7509284bf3784983a596c6eec7fc212-618xxxxxx.us-west-2.elb.amazonaws.com -P9030 -uroot ``` ##### Upgrade StarRocks Cluster[​](#upgrade-starrocks-cluster "Direct link to Upgrade StarRocks Cluster") ###### Upgrade BE nodes[​](#upgrade-be-nodes "Direct link to Upgrade BE nodes") Run the following command to specify a new BE image file, such as `starrocks/be-ubuntu:latest`: ```bash kubectl -n starrocks patch starrockscluster starrockscluster-sample --type='merge' -p '{"spec":{"starRocksBeSpec":{"image":"starrocks/be-ubuntu:latest"}}}' ``` ###### Upgrade FE nodes[​](#upgrade-fe-nodes "Direct link to Upgrade FE nodes") Run the following command to specify a new FE image file, such as `starrocks/fe-ubuntu:latest`: ```bash kubectl -n starrocks patch starrockscluster starrockscluster-sample --type='merge' -p '{"spec":{"starRocksFeSpec":{"image":"starrocks/fe-ubuntu:latest"}}}' ``` The upgrade process lasts for a while. You can run the command `kubectl -n starrocks get pods` to view the upgrade progress. ##### Scale StarRocks cluster[​](#scale-starrocks-cluster "Direct link to Scale StarRocks cluster") ###### Scale out BE cluster[​](#scale-out-be-cluster "Direct link to Scale out BE cluster") Run the following command to scale the BE cluster to 9 nodes: ```bash kubectl -n starrocks patch starrockscluster starrockscluster-sample --type='merge' -p '{"spec":{"starRocksBeSpec":{"replicas":9}}}' ``` ##### Scale in BE cluster[​](#scale-in-be-cluster "Direct link to Scale in BE cluster") When scaling in BE nodes, you need to scale them one at a time, and wait for the tablets on the BEs to be re-distributed before proceeding. If there are tables with single replicas, taking a BE node offline may cause data loss if the tablets fail to be redistributed. Execute the following command to scale in a cluster with 10 BE nodes to 9. ```bash kubectl -n starrocks patch starrockscluster starrockscluster-sample --type='merge' -p '{"spec":{"starRocksBeSpec":{"replicas":9}}}' ``` After scaling in, you must manually drop the nodes whose `alive` status is `false`. The redistribution of tablets will take some time. You can check the progress by executing `SHOW PROC '/statistic';`. ###### Scale out FE cluster[​](#scale-out-fe-cluster "Direct link to Scale out FE cluster") Run the following command to scale out the FE cluster to 4 nodes: ```bash kubectl -n starrocks patch starrockscluster starrockscluster-sample --type='merge' -p '{"spec":{"starRocksFeSpec":{"replicas":4}}}' ``` The scaling process lasts for a while. You can use the command `kubectl -n starrocks get pods` to view the scaling progress. ##### Automatic scaling for CN cluster[​](#automatic-scaling-for-cn-cluster "Direct link to Automatic scaling for CN cluster") Run the command `kubectl -n starrocks edit src starrockscluster-sample` to configure the automatic scaling policy for the CN cluster. You can specify the resource metrics for CNs as the average CPU utilization, average memory usage, elastic scaling threshold, upper elastic scaling limit, and lower elastic scaling limit. The upper elastic scaling limit and lower elastic scaling limit specify the maximum number and minimum number of CNs allowed for elastic scaling. note If the automatic scaling policy for the CN cluster is configured, delete the `replicas` field from the `starRocksCnSpec` in the StarRocks cluster configuration file. Kubernetes also supports using `behavior` to customize scaling behaviors according to business scenarios, helping you achieve rapid or slow scaling or disable scaling. For more information about automatic scaling policies, see [Horizontal Pod Scaling](https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/). The following is a [template](https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/examples/starrocks/deploy_a_starrocks_cluster_with_cn.yaml) provided by StarRocks to help you configure automatic scaling policies: ```yaml starRocksCnSpec: image: starrocks/cn-ubuntu:latest limits: cpu: 16 memory: 64Gi requests: cpu: 16 memory: 64Gi autoScalingPolicy: # Automatic scaling policy of the CN cluster. maxReplicas: 10 # The maximum number of CNs is set to 10. minReplicas: 1 # The minimum number of CNs is set to 1. # operator creates an HPA resource based on the following field. # see https://kubernetes.io/docs/concepts/workloads/autoscaling/horizontal-pod-autoscale/ for more information. hpaPolicy: metrics: # Resource metrics - type: Resource resource: name: memory # The average memory usage of CNs is specified as a resource metric. target: # The elastic scaling threshold is 60%. # When the average memory utilization of CNs exceeds 60%, the number of CNs increases for scale-out. # When the average memory utilization of CNs is below 60%, the number of CNs decreases for scale-in. averageUtilization: 60 type: Utilization - type: Resource resource: name: cpu # The average CPU utilization of CNs is specified as a resource metric. target: # The elastic scaling threshold is 60%. # When the average CPU utilization of CNs exceeds 60%, the number of CNs increases for scale-out. # When the average CPU utilization of CNs is below 60%, the number of CNs decreases for scale-in. averageUtilization: 60 type: Utilization behavior: # The scaling behavior is customized according to business scenarios, helping you achieve rapid or slow scaling or disable scaling. scaleUp: policies: - type: Pods value: 1 periodSeconds: 10 scaleDown: selectPolicy: Disabled ``` The following table describes a few important fields: * The upper and lower elastic scaling limit. ```yaml maxReplicas: 10 # The maximum number of CNs is set to 10. minReplicas: 1 # The minimum number of CNs is set to 1. ``` * The elastic scaling threshold. ```yaml # For example, the average CPU utilization of CNs is specified as a resource metric. # The elastic scaling threshold is 60%. # When the average CPU utilization of CNs exceeds 60%, the number of CNs increases for scale-out. # When the average CPU utilization of CNs is below 60%, the number of CNs decreases for scale-in. - type: Resource resource: name: cpu target: averageUtilization: 60 ``` #### FAQ[​](#faq "Direct link to FAQ") **Issue description:** When a custom resource StarRocksCluster is installed using `kubectl apply -f xxx`, an error is returned `The CustomResourceDefinition 'starrocksclusters.starrocks.com' is invalid: metadata.annotations: Too long: must have at most 262144 bytes`. **Cause analysis:** Whenever `kubectl apply -f xxx` is used to create or update resources, a metadata annotation `kubectl.kubernetes.io/last-applied-configuration` is added. This metadata annotation is in JSON format and records the *last-applied-configuration*. `kubectl apply -f xxx`" is suitable for most cases, but in rare situations , such as when the configuration file for the custom resource is too large, it may cause the size of the metadata annotation to exceed the limit. **Solution:** If you install the custom resource StarRocksCluster for the first time, it is recommended to use `kubectl create -f xxx`. If the custom resource StarRocksCluster is already installed in the environment, and you need to update its configuration, it is recommended to use `kubectl replace -f xxx`. --- ### Upgrade StarRocks This topic describes how to upgrade your StarRocks cluster. #### Important information[​](#important-information "Direct link to Important information") important Before upgrading StarRocks you should: * Read the [release notes](https://docs.starrocks.io/releasenotes/release-3.5/) for the StarRocks version that you are upgrading to, and all versions between the current version and the target version and: * Make notes of any behavior changes within StarRocks * Make notes of any changes with integrations between StarRocks and external systems used for importing, exporting, visualization, etc * Verify the [deployment prerequisites](https://docs.starrocks.io/docs/deployment/deployment_prerequisites.md) for the target version. For example, StarRocks 3.5.x requires JDK 17 and StarRocks 3.4.x on Ubuntu requires JDK 11. #### Overview[​](#overview "Direct link to Overview") Review the information in this section before upgrading, and perform any recommended actions. ##### StarRocks versions[​](#starrocks-versions "Direct link to StarRocks versions") The version of StarRocks is represented by three numbers in the form **Major.Minor.Patch**, for example, `2.5.4`. The first number represents the major version of StarRocks, the second number represents the minor version, and the third number represents the patch version. > **CAUTION** > > Please note that you cannot upgrade an existing shared-nothing cluster to a shared-data cluster, or vice versa. You must deploy a new shared-data cluster. ##### Upgrade paths[​](#upgrade-paths "Direct link to Upgrade paths") * **For patch version upgrade** You can upgrade your StarRocks cluster across patch versions, for example, from v2.2.6 directly to v2.2.11. * **For minor version upgrade** From StarRocks v2.0 onwards, you can upgrade a StarRocks cluster across minor versions, for example, from v2.2.x directly to v2.5.x. However, for compatibility and safety reasons, we strongly recommend you upgrade your StarRocks cluster **consecutively from one minor version to another**. For example, to upgrade a StarRocks v2.2 cluster to v2.5, you need to upgrade it in the following order: v2.2.x --> v2.3.x --> v2.4.x --> v2.5.x. * **For major version upgrade** To upgrade your StarRocks cluster to v3.0, you must first upgrade it to v2.5. > **CAUTION** > > Suppose you need to perform consecutive minor version upgrades, for example, 2.4->2.5->3.0->3.1->3.2, or you have downgraded your cluster after a failed upgrade and you want to upgrade the cluster again, for example, 2.5->3.0->2.5->3.0. To prevent metadata upgrade failure for some Follower FEs, perform the following steps between two consecutive upgrades or after the downgrade before the second trial of upgrade: > > 1. Run [ALTER SYSTEM CREATE IMAGE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md) to create a new image. > 2. Wait for the new image to be synchronized to all Follower FEs. > > You can check whether the image file has been synchronized by viewing the log file **fe.log** of the Leader FE. A record of log like "push image.\* from subdir \[] to other nodes. totally xx nodes, push successful xx nodes" suggests that the image file has been successfully synchronized. ##### Upgrade procedure[​](#upgrade-procedure "Direct link to Upgrade procedure") StarRocks supports **rolling upgrades**, which allow you to upgrade your cluster without stopping the service. By design, BEs and CNs are backward compatible with the FEs. Therefore, you need to **upgrade BEs and CNs first and then FEs** to allow your cluster to run properly while being upgraded. Upgrading them in an inverted order may lead to incompatibility between FEs and BEs/CNs, and thereby cause the service to crash. For FE nodes, you must first upgrade all Follower FE nodes before upgrading the Leader FE node. #### Before you begin[​](#before-you-begin "Direct link to Before you begin") During preparation, you must perform the compatibility configuration if you are up for a minor or major version upgrade. You also need to perform the upgrade availability test on one of the FEs and BEs before upgrading all nodes in the cluster. ##### Perform compatibility configuration[​](#perform-compatibility-configuration "Direct link to Perform compatibility configuration") If you want to upgrade your StarRocks cluster to a later minor or major version, you must perform the compatibility configuration. In addition to the universal compatibility configuration, detailed configurations vary depending on the version of the StarRocks cluster you upgrade from. * **Universal compatibility configuration** Before upgrading your StarRocks cluster, you must disable tablet clone. You can skip this step if you have disabled the balancer. ```sql ADMIN SET FRONTEND CONFIG ("tablet_sched_max_scheduling_tablets" = "0"); ADMIN SET FRONTEND CONFIG ("tablet_sched_max_balancing_tablets" = "0"); ADMIN SET FRONTEND CONFIG ("disable_balance"="true"); ADMIN SET FRONTEND CONFIG ("disable_colocate_balance"="true"); ``` After the upgrade, and the status of all BE nodes is `Alive`, you can re-enable tablet clone. ```sql ADMIN SET FRONTEND CONFIG ("tablet_sched_max_scheduling_tablets" = "10000"); ADMIN SET FRONTEND CONFIG ("tablet_sched_max_balancing_tablets" = "500"); ADMIN SET FRONTEND CONFIG ("disable_balance"="false"); ADMIN SET FRONTEND CONFIG ("disable_colocate_balance"="false"); ``` * **If you upgrade from v2.0 to later versions** Before upgrading your StarRocks v2.0 cluster, you must set the following BE configuration and system variable. 1. If you have modified the BE configuration item `vector_chunk_size`, you must set it to `4096` before upgrading. Because it is a static parameter, you must modify it in the BE configuration file **be.conf** and restart the node to allow the modification to take effect. 2. Set the system variable `batch_size` to less than or equal to `4096` globally. ```sql SET GLOBAL batch_size = 4096; ``` #### Upgrade BE[​](#upgrade-be "Direct link to Upgrade BE") Having passed the upgrade availability test, you can first upgrade the BE nodes in the cluster. 1. Navigate to the working directory of the BE node and stop the node. ```bash # Replace with the deployment directory of the BE node. cd /be ./bin/stop_be.sh ``` 2. Replace the original deployment files under **bin** and **lib** with the ones of the new version. ```bash mv lib lib.bak mv bin bin.bak cp -r /tmp/StarRocks-x.x.x/be/lib . cp -r /tmp/StarRocks-x.x.x/be/bin . # If a custom function (UDF) was used in the old version, you need to copy the old version's UDF directory to the new lib directory. cp -r lib.bak/udf lib/ ``` 3. Start the BE node. ```bash ./bin/start_be.sh --daemon ``` 4. Check if the BE node is started successfully. ```bash ps aux | grep starrocks_be ``` 5. Repeat the above procedures to upgrade other BE nodes. #### Upgrade CN[​](#upgrade-cn "Direct link to Upgrade CN") 1. Navigate to the working directory of the CN node and stop the node gracefully. ```bash # Replace with the deployment directory of the CN node. cd /be ./bin/stop_cn.sh --graceful ``` 2. Replace the original deployment files under **bin** and **lib** with the ones of the new version. ```bash mv lib lib.bak mv bin bin.bak cp -r /tmp/StarRocks-x.x.x/be/lib . cp -r /tmp/StarRocks-x.x.x/be/bin . # If a custom function (UDF) was used in the old version, you need to copy the old version's UDF directory to the new lib directory. cp -r lib.bak/udf lib/ ``` 3. Start the CN node. ```bash ./bin/start_cn.sh --daemon ``` 4. Check if the CN node is started successfully. ```bash ps aux | grep starrocks_be ``` 5. Repeat the above procedures to upgrade other CN nodes. #### Upgrade FE[​](#upgrade-fe "Direct link to Upgrade FE") After upgrading all BE and CN nodes, you can then upgrade the FE nodes. You must first upgrade the Follower FE nodes and then the Leader FE node. 1. Navigate to the working directory of the FE node and stop the node. ```bash # Replace with the deployment directory of the FE node. cd /fe ./bin/stop_fe.sh ``` 2. Replace the original deployment files under **bin**, **lib**, and **spark-dpp** with the ones of the new version. ```bash mv lib lib.bak mv bin bin.bak mv spark-dpp spark-dpp.bak cp -r /tmp/StarRocks-x.x.x/fe/lib . cp -r /tmp/StarRocks-x.x.x/fe/bin . cp -r /tmp/StarRocks-x.x.x/fe/spark-dpp . ``` 3. Start the FE node. ```bash ./bin/start_fe.sh --daemon ``` 4. Check if the FE node is started successfully. ```bash ps aux | grep StarRocksFE ``` 5. Repeat the above procedures to upgrade other Follower FE nodes, and finally the Leader FE node. --- ## Developers ### Compile StarRocks with Docker This topic describes how to compile StarRocks using Docker. #### Overview[​](#overview "Direct link to Overview") StarRocks provides development environment images for Ubuntu (22.04 and 24.04), CentOS 7.9, and Rocky Linux 9. With the image, you can launch a Docker container and compile StarRocks in the container. note Starting from v4.2, the development environment changes as follows: * CentOS 7 reached its End-of-Life on June 30, 2024, so its build distribution is discontinued and replaced by Rocky Linux 9. CentOS 7 images remain available only for v4.1 and earlier. * The Ubuntu development environment is upgraded from Ubuntu 22.04 (v4.1 and earlier) to Ubuntu 24.04 (v4.2 and later). The image name `starrocks/dev-env-ubuntu` is unchanged. ##### StarRocks version and DEV ENV image[​](#starrocks-version-and-dev-env-image "Direct link to StarRocks version and DEV ENV image") Different branches of StarRocks correspond to different development environment images provided on [StarRocks Docker Hub](https://hub.docker.com/u/starrocks). * For Ubuntu (22.04 for v4.1 and earlier, 24.04 for v4.2 and later): | **Branch name** | **Image name** | | --------------- | --------------------------------------- | | main | starrocks/dev-env-ubuntu:latest | | branch-4.1 | starrocks/dev-env-ubuntu:4.1-latest | | branch-4.0 | starrocks/dev-env-ubuntu:4.0-latest | | branch-3.5 | starrocks/dev-env-ubuntu:3.5-latest | * For CentOS 7.9 (v4.1 and earlier; discontinued from v4.2): | **Branch name** | **Image name** | | --------------- | ------------------------------------ | | branch-4.1 | starrocks/dev-env-centos7:4.1-latest | | branch-4.0 | starrocks/dev-env-centos7:4.0-latest | | branch-3.5 | starrocks/dev-env-centos7:3.5-latest | * For Rocky Linux 9 (v4.2 and later): | **Branch name** | **Image name** | | --------------- | --------------------------------------- | | main | starrocks/dev-env-rocky9:latest | #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before compiling StarRocks, make sure the following requirements are satisfied: * **Hardware** Your machine must have at least 8 GB RAM. * **Software** * Your machine must be running on Ubuntu 22.04 or 24.04, CentOS 7.9, or Rocky Linux 9. * You must have Docker installed on your machine and version v20.10.10 at least. #### Step 1: Download the image[​](#step-1-download-the-image "Direct link to Step 1: Download the image") Download the development environment image by running the following command: ```bash # Replace with the name of the image that you want to download, # for example, `starrocks/dev-env-ubuntu:latest`. # Make sure you have choose the correct image for your OS. docker pull ``` Docker automatically identifies the CPU architecture of your machine and pulls the corresponding image that suits your machine. The `linux/amd64` images are for the x86-based CPUs, and `linux/arm64` images are for the ARM-based CPUs. #### Step 2: Compile StarRocks in a Docker container[​](#step-2-compile-starrocks-in-a-docker-container "Direct link to Step 2: Compile StarRocks in a Docker container") You can launch the development environment Docker container with or without the local host path mounted. We recommend you launch the container with the local host path mounted, so that you can avoid re-downloading the Java dependency during the next compilation, and you do not need to manually copy the binary files from the container to your local host. * **Launch the container with the local host path mounted**: 1. Clone the StarRocks source code to your local host. ```bash git clone https://github.com/StarRocks/starrocks.git ``` 2. Launch the container. ```bash # Replace with the parent directory of the StarRocks source code directory. # Replace with the name of the branch that corresponds to the image name. # Replace with the name of the image that you downloaded. docker run -it -v /.m2:/root/.m2 \ -v /starrocks:/root/starrocks \ --name -d ``` 3. Launch a bash shell inside the container you have launched. ```bash # Replace with the name of the branch that corresponds to the image name. docker exec -it /bin/bash ``` 4. Compile StarRocks in the container. ```bash cd /root/starrocks && ./build.sh ``` * **Launch the container without the local host path mounted**: 1. Launch the container. ```bash # Replace with the name of the branch that corresponds to the image name. # Replace with the name of the image that you downloaded. docker run -it --name -d ``` 2. Launch a bash shell inside the container. ```bash # Replace with the name of the branch that corresponds to the image name. docker exec -it /bin/bash ``` 3. Clone the StarRocks source code to the container. ```bash git clone https://github.com/StarRocks/starrocks.git ``` 4. Compile StarRocks in the container. ```bash cd starrocks && ./build.sh ``` #### Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") Q: The StarRocks BE building fails, and the following error message has been returned: ```bash g++: fatal error: Killed signal terminated program cc1plus compilation terminated. ``` What should I do? A: This error message indicates a lack of memory in the Docker container. You need to allocate at least 8 GB of memory resources to the container. --- ### Compile StarRocks on Ubuntu This topic describes how to compile StarRocks on the Ubuntu operating system. StarRocks supports compilation on both x86\_64 and AArch64 architectures. note Build StarRocks v4.1 and earlier on Ubuntu 22.04, and StarRocks v4.2 and later on Ubuntu 24.04. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") ##### Install Dependencies[​](#install-dependencies "Direct link to Install Dependencies") Run the following commands to install necessary dependencies: ```bash sudo apt update ``` ```bash sudo apt install build-essential automake bison byacc ccache flex libiberty-dev libtool maven zip python3 python-is-python3 bzip2 -y ``` ##### Install Compiler[​](#install-compiler "Direct link to Install Compiler") If you are using Ubuntu 22.04 or later, run the following command to install the tools and compilers: ```bash sudo apt install cmake gcc g++ openjdk-17-jdk -y ``` If you are using an Ubuntu version earlier than 22.04, run the following commands to check the versions of tools and compilers: 1. Check GCC/G++ versions: ```bash gcc --version g++ --version ``` GCC/G++ versions must be 10.3 or later. If you are using earlier versions, [click here to install GCC/G++](https://gcc.gnu.org/releases.html). 2. Check JDK version: ```bash java --version ``` OpenJDK version must be 17 or later. If you are using an earlier version, [click here to install OpenJDK](https://openjdk.org/install). 3. Check CMake version: ```bash cmake --version ``` CMake version must be 3.20.1 or later. If you are using an earlier version, [click here to install CMake](https://cmake.org/download). #### Compile StarRocks[​](#compile-starrocks "Direct link to Compile StarRocks") ##### Download Source Code[​](#download-source-code "Direct link to Download Source Code") Run the following command to clone the StarRocks repository and navigate into the directory: ```bash git clone https://github.com/StarRocks/starrocks.git cd starrocks ``` ##### Build StarRocks[​](#build-starrocks "Direct link to Build StarRocks") Run the following command to start the compilation: ```bash ./build.sh ``` The default compilation parallelism is equal to **CPU core count/4**. Assuming you have 32 CPU cores, the default parallelism is 8. If you want to adjust the parallelism, you can specify the number of CPU cores to be used for compilation via `-j` in the command line. The following example uses 24 CPU cores for compilation: ```bash ./build.sh -j 24 ``` #### FAQ[​](#faq "Direct link to FAQ") Q-1: Building `aws_cpp_sdk` fails on Ubuntu 20.04 with the error "Error: undefined reference to pthread\_create". How can I resolve this? A: This error occurs due to a lower version of CMake. Please upgrade CMake to version 3.20.1 or above. Q-2: Building StarRocks fails on Ubuntu 24.04 or GCC 12+ with strict warning errors. How can I resolve this? A: To prevent build failures in third-party libraries, export the following flag before building: ```bash export DISABLE_WARNING_AS_ERROR=1 ./build.sh ``` --- ### How to build StarRocks In general, you can build StarRocks by just executing ```text ./build.sh ``` This command will check if all the thirdpary dependencies are ready at first. If all dependencies are ready, it will build StarRocks `Backend` and `Frontend`. After this command executes succefully, the generated binary will be in `output` directory. #### build FE/BE separately[​](#build-febe-separately "Direct link to build FE/BE separately") You don't need to build both FE and BE each time, you can build them separately. For example, you can only build BE by ```text ./build.sh --be ``` and, only build FE by ```text ./build.sh --fe ``` ### How to run unit test Unit tests of BE and FE are separted. In general, you can run BE test by ```text ./run-be-ut.sh ``` run FE test by ```text ./run-fe-ut.sh ``` #### How to run BE UT in command line[​](#how-to-run-be-ut-in-command-line "Direct link to How to run BE UT in command line") Now, BE UT needs some dependency to run, and `./run-be-ut.sh` helps it. But it is not flexible enough. When you want to run UT in the command-line, you can execute ```text UDF_RUNTIME_DIR=./ STARROCKS_HOME=./ LD_LIBRARY_PATH=/usr/lib/jvm/java-18-openjdk-amd64/lib/server ./be/ut_build_ASAN/test/starrocks_test ``` StarRocks Backend UT is built on top of google-test, so you can pass filter to run some of the UT, For example, you want to test only MapColumn related tests, you can execute ```text UDF_RUNTIME_DIR=./ STARROCKS_HOME=./ LD_LIBRARY_PATH=/usr/lib/jvm/java-18-openjdk-amd64/lib/server ./be/ut_build_ASAN/test/starrocks_test --gtest_filter="*MapColumn*" ``` ### Build options #### build with clang[​](#build-with-clang "Direct link to build with clang") You can build StarRocks by `clang` too ```text CC=clang CXX=clang++ ./build.sh --be ``` Then you can see the following similar message in the build message ```text -- compiler Clang version 14.0.0 ``` #### build with different linker[​](#build-with-different-linker "Direct link to build with different linker") The default linker is slow, developer can specify different linker to speed up linking. For example, you can use `lld`, the LLVM-based linker. You need to install `lld` firstly. ```text sudo apt install lld ``` Then you set the environment variable STARROCKS\_LINKER with the linker you want to use. For example: ```text STARROCKS_LINKER=lld ./build.sh --be ``` #### build different type[​](#build-different-type "Direct link to build different type") You can build StarRocks with different types with different BUILD\_TYPE variable, the default BUILD\_TYPE is `RELEASE`. For example, you can build StarRocks with `ASAN` type by ```text BUILD_TYPE=ASAN ./build.sh --be ``` --- ### Protobuf Guides #### Never use required[​](#never-use-required "Direct link to Never use required") As the project involving, any fields may become optional. But if it is defined as required, it can not be removed. So `required` should not be used. #### Never change the ordinal[​](#never-change-the-ordinal "Direct link to Never change the ordinal") To be back compatible, the ordinal of the field SHOULD NOT be changed. ### Naming #### file name[​](#file-name "Direct link to file name") The names of messages are all lowercase, with underscores between words. Files should end in `.proto`. ```text my_message.proto // Good mymessage.proto // Bad my_message.pb // Bad ``` #### Message Name[​](#message-name "Direct link to Message Name") Message names start with a capital letter and have a capital letter for each new word, with no underscores, and with `PB` as postfix: MyMessagePB ```protobuf message MyMessagePB // Good message MyMessage // Bad message My_Message_PB // Bad message myMessagePB // Bad ``` #### field name[​](#field-name "Direct link to field name") The names of messages are all lowercase, with underscores between words. ```text optional int64 my_field = 3; // Good optional int64 myField = 3; // Bad ``` --- ### StarRocks Restful API Standard #### API Format[​](#api-format "Direct link to API Format") 1. The API format follows the pattern: `/api/{version}/{target-object-access-path}/{action}`. 2. `{version}` is denoted as `v{number}`, such as v1, v2, v3, v4, etc. 3. `{target-object-access-path}` is organized in a hierarchical manner, which will be explained in detail later. 4. `{action}` is optional, API implementors should utilize the HTTP METHOD to convey the operation's meaning as much as possible. Only when the HTTP methods' semantics cannot be fulfilled should the action be used. For example, if there is no HTTP method available to rename an object. #### Definition of Target Object Access Path[​](#definition-of-target-object-access-path "Direct link to Definition of Target Object Access Path") 1. Target objects accessed by the REST API need to be categorized and organized into a hierarchical access path. The access path format is as follows: ```text /primary_categories/primary_object/secondary_categories/secondary_object/.../categories/object ``` Taking catalog, database, table, column as examples: ```text /catalogs: Represents all catalogs. /catalogs/hive: Represents the specific catalog object named "hive" under the catalog category. /catalogs/hive/databases: Represents all databases in the "hive" catalog. /catalogs/hive/databases/tpch_100g: Represents the database named "tpch_100g" in the "hive" catalog. /catalogs/hive/databases/tpch_100g/tables: Represents all tables in the "tpch_100g" database. /catalogs/hive/databases/tpch_100g/tables/lineitem: Represents the tpch_100g.lineitem table. /catalogs/hive/databases/tpch_100g/tables/lineitem/columns: Represents all columns in the tpch_100g.lineitem table. /catalogs/hive/databases/tpch_100g/tables/lineitem/columns/l_orderkey: Represents the specific column l_orderkey in the tpch_100g.lineitem table. ``` 2. Categories are named using snake-case, and the last word is in plural form.all the words are in lowercase, and multiple words are connected by underscores (\_). Specific objects are named using their actual names. The hierarchical relationship of the target objects needs to be clearly defined. #### Selection of HTTP Method[​](#selection-of-http-method "Direct link to Selection of HTTP Method") 1. GET: Use the GET method to show a single object and list all objects of a certain category. The GET method's access to objects is read-only and does not provide a request body. ```text # list all of the tables in database ssb_100g GET /api/v2/catalogs/default/databases/ssb_100g/tables # show the table ssb_100g.lineorder GET /api/v2/catalogs/default/databases/ssb_100g/tables/lineorder ``` 2. POST: Used to create objects. Parameters are passed through the request body. It is not idempotent. If the object already exists, the repeated creation will fail and return an error message. ```text POST /api/v2/catalogs/default/databases/ssb_100g/tables/create -d@create_customer.sql ``` 3. PUT: Used to create objects. Parameters are passed through the request body. It is idempotent. If the object already exists, it will return success. PUT method is the CREATE IF NOT EXISTS version of the POST method. ```text PUT /api/v2/databases/ssb_100g/tables/create -d@create_customer.sql ``` 4. DELETE: Used to delete objects. It does not provide a request body. If the object to be deleted does not exist, it will return success. DELETE method has the DROP IF EXISTS semantics. ```text DELETE /api/v2/catalogs/default/databases/ssb_100g/tables/customer ``` 5. PATCH: Used to update objects. It provides a request body, which only contains the partial information that needs to be modified. ```text PATCH /api/v2/databases/ssb_100g/tables/customer -d '{"unique_key_constraints": ["c_custkey"]}' ``` #### Authentication and Authorization[​](#authentication-and-authorization "Direct link to Authentication and Authorization") 1. Authentication and authorization information is passed in the HTTP Request Header. #### HTTP Status Codes[​](#http-status-codes "Direct link to HTTP Status Codes") 1. HTTP status codes are returned by the REST API to indicate the success or failure of an operation. 2. The status codes (2xx) for success operation as follows: * 200 OK: Indicates that the request has been successfully completed. It is used for viewing/listing/deleting/updating objects and querying the status of pending tasks. * 201 Created: Indicates that the object has been successfully created. It is used for PUT/POST methods. The response body must include the object URI for subsequent viewing/listing/deleting/updating. * 202 Accepted: Indicates that the task submission is successful and the task is in a pending state. The response body must include the task URI for subsequent cancellation, deletion, and polling of task status. 3. The error codes (4xx) indicate client errors. Users need to adjust and modify the HTTP request and retry. * 400 Bad Request: Invalid request parameters. * 401 Unauthorized: Missing authentication information, illegal authentication information, authentication failure. * 403 Forbidden: Authentication succeeded, but the user's operation failed the authorization check. No access permission. * 404 Not Found: API URI encoding error. It does not belong to the registered REST API. * 405 Method Not Allowed: Incorrect HTTP Method used. * 406 Not Acceptable: The response format does not match the media type specified in the Accept header. * 415 Not Acceptable: The media type of the request content does not match the media type specified in the Content-Type header. 4. The error codes (5xx) indicate server errors. Users do not need to modify the request and can retry later. * 500 Internal Server Error: Internal server error, similar to Unknown error. * 503 Service Unavailable: The service is temporarily unavailable. For example, the user's access frequency is too high and has reached the rate limit; or the service is currently unable to provide service due to internal status, such as when creating a table with 3 replicas, but only 2 BEs are available; all Tablet replicas involved in a user's query are unavailable. #### HTTP Response Format[​](#http-response-format "Direct link to HTTP Response Format") 1. When the API returns an HTTP code code of 200/201/202, the HTTP response is not empty. The API returns results in JSON format, including top-level fields "code", "message", and "result". All JSON fields are named using camel-case. 2. In a successful API response, the "code" is "0", the "message" is "OK", and the "result" contains the actual results. ```json { "code":"0", "message": "OK", "result": {....} } ``` 3. In a failed API response, the "code" is not "0", the "message" is a simple error message, and the "result" can contain detailed error information, such as error stack traces. ```json { "code":"1", "message": "Analyze error", "result": {....} } ``` #### Parameter Passing[​](#parameter-passing "Direct link to Parameter Passing") 1. API parameters are passed in the precedence order of path, request body, query parameters, and header. Choose the appropriate method for parameter passing. 2. Path parameters: Required parameters that represent the object's hierarchical relationship are placed in the path parameters. ```text /api/v2/warehouses/{warehouseName}/backends/{backendId} /api/v2/warehouses/ware0/backends/10027 ``` 3. Request body: Parameters are passed using application/json. Parameters can be of required or optional types. 4. Query parameters: Using query parameters and request body parameters at the same time is not allowed. For the same API, choose either one. If the number of parameters excluding header parameters and path parameters is not more than 2, query parameters can be used; otherwise, use the request body to pass parameters. 5. HEADER parameters: Headers should be used to pass HTTP standard parameters such as Content-type and Accept are placed in the header, implementors should not abuse http headers to pass customized parameters. When using headers to pass parameters for user extensions, the header name should be in the format `x-starrocks-{name}`, where the name can contain multiple English words, and each word is in lowercase and concatenated by hyphens (-). --- ### Thrift Guides #### Never use required[​](#never-use-required "Direct link to Never use required") As the project involving, any fields may become optional. But if it is defined as required, it can not be removed. So `required` should not be used. #### Never change the ordinal[​](#never-change-the-ordinal "Direct link to Never change the ordinal") To be back compatible, the ordinal of the field SHOULD NOT be changed. ### Naming #### file name[​](#file-name "Direct link to file name") The names of messages are all lowercase, with underscores between words. Files should end in `.thrift`. ```text my_struct.thrift // Good MyStruct.thrift // Bad my_struct.proto // Bad ``` #### struct name[​](#struct-name "Direct link to struct name") Struct names start with a capital letter `T` and have a capital letter for each new word, with no underscores: TMyStruct ```text struct TMyStruct; // Good struct MyStruct; // Bad struct TMy_Struct; // Bad struct TmyStruct; // Bad ``` #### field name[​](#field-name "Direct link to field name") The names of struct members are all lowercase, with underscores between words. ```text 1: optional i64 my_field; // Good 1: optional i64 myField; // Bad ``` --- ### Use the debuginfo file for debugging #### Change description[​](#change-description "Direct link to Change description") From v2.5 onwards, the debuginfo file of BE is stripped from the StarRocks installation package to reduce the size and space usage of the installation package. You can see two packages at [StarRocks website](https://www.starrocks.io/download/community). ![debuginfo](/assets/images/debug_info-5791bd74c1be19fa504931d9c8491749.png) In this figure, you can click `Get Debug Symbol files` to download the debuginfo package. `StarRocks-2.5.10.tar.gz` is the installation package and you can click **Download** to download this package. This change does not affect your download behavior or use of StarRocks. You can download only the installation package for cluster deployment and upgrade. The debuginfo package is only for developers to debug programs using GDB. #### Precautions[​](#precautions "Direct link to Precautions") GDB 12.1 or later is recommended for debugging. #### How to use the debuginfo file[​](#how-to-use-the-debuginfo-file "Direct link to How to use the debuginfo file") 1. Download and decompress the debuginfo package. ```sql wget https://releases.starrocks.io/starrocks/StarRocks-.debuginfo.tar.gz tar -xzvf StarRocks-.debuginfo.tar.gz ``` > **NOTE** > > Replace `` with the version number of the StarRocks installation package you want to download. 2. Load the debuginfo file when you perform GDB debugging. * **Method 1** ```shell objcopy --add-gnu-debuglink=starrocks_be.debug starrocks_be ``` This operation associates the debug info file with your executable file. * **Method 2** ```shell gdb -s starrocks_be.debug -e starrocks_be -c `core_file` ``` The debuginfo file works well with perf and pstack. You can directly use perf and pstack without additional operations. --- ### Setup development environment with Docker This directory provides Docker-based build tools for StarRocks that use the official `starrocks/dev-env-ubuntu:latest` development environment image. This ensures consistent builds across different host systems by using a standardized Ubuntu environment with all required toolchains and dependencies pre-installed. #### 🚀 Quick Start[​](#-quick-start "Direct link to 🚀 Quick Start") ##### Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Docker installed and running * At least 8GB RAM available for Docker * At least 20GB free disk space ##### Simple Commands[​](#simple-commands "Direct link to Simple Commands") ```bash # Open development shell ./docker-dev.sh shell # Build Frontend only ./docker-dev.sh build-fe # Build Backend only ./docker-dev.sh build-be # Build everything ./docker-dev.sh build-all # Clean build everything ./docker-dev.sh clean-build # Run Frontend tests ./docker-dev.sh test-fe ``` #### 📋 Available Tools[​](#-available-tools "Direct link to 📋 Available Tools") ##### 1. `build-in-docker.sh` - Full-Featured Build Script[​](#1-build-in-dockersh---full-featured-build-script "Direct link to 1-build-in-dockersh---full-featured-build-script") The main build script that automatically passes through all `build.sh` options: ```bash # Basic usage ./build-in-docker.sh # Build all (FE + BE) ./build-in-docker.sh --fe # Build Frontend only ./build-in-docker.sh --be # Build Backend only ./build-in-docker.sh --fe --be --clean # Clean and build both # Advanced options ./build-in-docker.sh --be --with-gcov # Build BE with code coverage ./build-in-docker.sh --fe --disable-java-check-style # Skip checkstyle ./build-in-docker.sh --be -j 8 # Build with 8 parallel jobs # Development ./build-in-docker.sh --shell # Interactive shell ./build-in-docker.sh --test # Build and run tests # Custom image ./build-in-docker.sh --image starrocks/dev-env-ubuntu:latest --fe # Future build.sh options work automatically ./build-in-docker.sh --be --new-future-option ``` ##### 2. `docker-dev.sh` - Simple Wrapper[​](#2-docker-devsh---simple-wrapper "Direct link to 2-docker-devsh---simple-wrapper") Quick commands for common tasks: ```bash ./docker-dev.sh shell # Development shell ./docker-dev.sh build-fe # Build Frontend ./docker-dev.sh build-be # Build Backend ./docker-dev.sh build-all # Build everything ./docker-dev.sh clean-build # Clean and build all ./docker-dev.sh test-fe # Run FE tests ./docker-dev.sh test-be # Run BE tests ./docker-dev.sh test-all # Run all tests # Pass through any build.sh options ./docker-dev.sh build --be --with-gcov ./docker-dev.sh build --fe --new-option ``` ##### 3. `docker-compose.dev.yml` - Docker Compose[​](#3-docker-composedevyml---docker-compose "Direct link to 3-docker-composedevyml---docker-compose") For persistent development environments: ```bash # Start development shell docker-compose -f docker-compose.dev.yml run --rm starrocks-dev # Build Frontend docker-compose -f docker-compose.dev.yml run --rm build-fe # Build Backend docker-compose -f docker-compose.dev.yml run --rm build-be # Run tests docker-compose -f docker-compose.dev.yml run --rm test-fe docker-compose -f docker-compose.dev.yml run --rm test-be # Clean up docker-compose -f docker-compose.dev.yml down -v ``` #### 🔧 Configuration[​](#-configuration "Direct link to 🔧 Configuration") ##### Volume Mounts[​](#volume-mounts "Direct link to Volume Mounts") The Docker scripts automatically mount: * **Source code**: `$(pwd):/workspace` - Your local repository * **Maven cache**: `~/.m2:/tmp/.m2` - Maven dependencies cache for faster builds **Note**: The Maven cache is shared between your host system and the Docker container, so dependencies downloaded during builds are persisted and reused across build sessions. ##### Environment Variables[​](#environment-variables "Direct link to Environment Variables") ```bash # Use different Docker image export STARROCKS_DEV_ENV_IMAGE=starrocks/dev-env-ubuntu:latest # Additional Docker options export DOCKER_BUILD_OPTS="--memory=16g --cpus=8" # Set user ID for file permissions export UID=$(id -u) export GID=$(id -g) ``` ##### Custom Build Options[​](#custom-build-options "Direct link to Custom Build Options") All original `build.sh` options are supported: ```bash # Backend build types BUILD_TYPE=Debug ./build-in-docker.sh --be # Debug build BUILD_TYPE=Release ./build-in-docker.sh --be # Release build (default) BUILD_TYPE=Asan ./build-in-docker.sh --be # AddressSanitizer build # Feature flags ./build-in-docker.sh --be --enable-shared-data # Enable shared data ./build-in-docker.sh --be --with-gcov # Code coverage ./build-in-docker.sh --be --with-bench # Benchmarks ./build-in-docker.sh --be --without-avx2 # Disable AVX2 ``` #### 📁 Output and Artifacts[​](#-output-and-artifacts "Direct link to 📁 Output and Artifacts") Build artifacts are created in the `output/` directory: ```text output/ ├── fe/ # Frontend artifacts ├── be/ # Backend artifacts └── java-extensions/ # Java extensions ``` The Docker container mounts your local repository and Maven cache, so all build outputs are available on your host system and dependencies are cached for faster subsequent builds. **Multi-User Support**: Container names include username and user ID to prevent conflicts on shared development machines (e.g., `starrocks-build-username-1001-1234567890`). **Automatic Extensibility**: All unrecognized options are automatically passed through to `build.sh`, so new build options work without updating the Docker scripts. #### 🐛 Troubleshooting[​](#-troubleshooting "Direct link to 🐛 Troubleshooting") ##### Common Issues[​](#common-issues "Direct link to Common Issues") 1. **Permission Issues** ```bash # Fix file permissions sudo chown -R $(id -u):$(id -g) output/ # Or run with correct user export UID=$(id -u) GID=$(id -g) ./build-in-docker.sh --fe ``` 2. **Out of Memory** ```bash # Increase Docker memory limit or reduce parallel jobs ./build-in-docker.sh --be -j 2 ``` 3. **Docker Image Not Found** ```bash # Pull the image manually docker pull starrocks/dev-env-ubuntu:latest ``` 4. **Build Failures** ```bash # Clean build ./build-in-docker.sh --clean --fe --be # Check logs in interactive shell ./build-in-docker.sh --shell ``` ##### Debug Mode[​](#debug-mode "Direct link to Debug Mode") For debugging build issues: ```bash # Open shell and run commands manually ./build-in-docker.sh --shell # Inside container: ./build.sh --fe --clean ./run-fe-ut.sh ``` #### 🔍 Verification[​](#-verification "Direct link to 🔍 Verification") Test that everything works: ```bash # 1. Test Docker setup docker --version docker info # 2. Test image availability docker pull starrocks/dev-env-ubuntu:latest # 3. Test build scripts ./docker-dev.sh shell # Inside container: exit # 4. Test simple build ./docker-dev.sh build-fe ``` #### 📊 Performance Tips[​](#-performance-tips "Direct link to 📊 Performance Tips") 1. **Use parallel builds**: `./build-in-docker.sh --be -j $(nproc)` 2. **Maven cache**: Automatically mounted from `~/.m2` for faster dependency resolution 3. **Memory allocation**: Increase Docker memory limit for faster builds 4. **SSD storage**: Use SSD for Docker storage driver 5. **Persistent volumes**: Use Docker Compose for additional cache persistence #### 🤝 Integration with IDEs[​](#-integration-with-ides "Direct link to 🤝 Integration with IDEs") ##### VS Code with Dev Containers[​](#vs-code-with-dev-containers "Direct link to VS Code with Dev Containers") Create `.devcontainer/devcontainer.json`: ```json { "name": "StarRocks Dev Environment", "image": "starrocks/dev-env-ubuntu:latest", "workspaceMount": "source=${localWorkspaceFolder},target=/workspace,type=bind", "workspaceFolder": "/workspace", "customizations": { "vscode": { "extensions": [ "ms-vscode.cpptools", "redhat.java" ] } } } ``` ##### IntelliJ IDEA[​](#intellij-idea "Direct link to IntelliJ IDEA") Use the Docker integration to run builds and tests within the container environment. #### 📚 Additional Resources[​](#-additional-resources "Direct link to 📚 Additional Resources") * [Original build.sh](https://github.com/StarRocks/starrocks/blob/main/build.sh) for reference #### 🆘 Support[​](#-support "Direct link to 🆘 Support") If you encounter issues: 1. Check the troubleshooting section above 2. Verify Docker setup and image availability 3. Try clean builds with `--clean` flag 4. Open an issue with build logs and system information --- ### Setup IDE for developing StarRocks ### Setup IDE for developing StarRocks Some people want to become StarRocks contributor, but are troubled by the development environment, so here I write a tutorial about it. What is a perfect development toolchain? * Support one click to compile FE and BE. * Support code jump in Clion and IDEA. * All variables in the IDE can be analyzed normally without red lines. * Clion can enable its analysis function normally. * Support FE and BE debug. #### Prepare[​](#prepare "Direct link to Prepare") I use a MacBook(M1) for local coding and a remote server for compiling & testing StarRocks. (Remote server uses Ubuntu 22, **at least need 16GB RAM**). The overall idea is to write code on the MacBook, then automatically synchronize the code to the server through the IDE, and use the server to compile and develop StarRocks. ##### MacBook Setup[​](#macbook-setup "Direct link to MacBook Setup") ###### Thrift 0.20[​](#thrift-020 "Direct link to Thrift 0.20") ```bash brew install cartman-kai/thrift/thrift@0.20 ``` You can check whether Thrift is installed successfully with the following command: ```bash $ thrift -version Thrift version 0.20.0 ``` ###### Protobuf[​](#protobuf "Direct link to Protobuf") Just use the latest version v3 directly, because the latest version of Protobuf is compatible with the v2 version of the Protobuf protocol in StarRocks. ```bash brew install protobuf ``` ###### Maven[​](#maven "Direct link to Maven") ```bash brew install maven ``` ###### OpenJDK 17[​](#openjdk-17 "Direct link to OpenJDK 17") ```bash brew install openjdk@17 ``` ###### Python3[​](#python3 "Direct link to Python3") MacOS comes with it, no installation is needed. ###### Setup system env[​](#setup-system-env "Direct link to Setup system env") ```bash export JAVA_HOME=xxxxx export PYTHON=/usr/bin/python3 ``` ##### Ubuntu server setup[​](#ubuntu-server-setup "Direct link to Ubuntu server setup") ###### Clone StarRocks code[​](#clone-starrocks-code "Direct link to Clone StarRocks code") `git clone https://github.com/StarRocks/starrocks.git` ###### Install required tools for compilation[​](#install-required-tools-for-compilation "Direct link to Install required tools for compilation") ```bash sudo apt update ``` ```bash sudo apt install gcc g++ maven openjdk-17-jdk python3 python-is-python3 unzip cmake bzip2 ccache byacc ccache flex automake libtool bison libiberty-dev build-essential ninja-build curl ``` Setup `JAVA_HOME` env ```bash export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 ``` ###### Do a compilation of StarRocks[​](#do-a-compilation-of-starrocks "Direct link to Do a compilation of StarRocks") ```bash cd starrocks/ ./build.sh ``` The first time compile needs to compile thirdparty, it will require some time. **You have to use gcc for the first compilation, currently, thirdparty can't compile success in clang.** #### IDE Setup[​](#ide-setup "Direct link to IDE Setup") ##### FE[​](#fe "Direct link to FE") FE development is simple because you can compile it in MacOS directly. Just enter `fe` folder and run the command `mvn install -DskipTests`. Then you can use IDEA to open `fe` folder directly, everything is ok. ###### Local debug[​](#local-debug "Direct link to Local debug") The same as other Java applications. You might face OOM error while compiling StarRocks on Intellij IDEA, you can go to Build, Execution, Deployment -> Compiler -> Increase Heap size to a larger value. ###### Remote debug[​](#remote-debug "Direct link to Remote debug") In Ubuntu server, run with `./start_fe.sh --debug`, then use IDEA remote debug to connect it. The default port is 5005, you can change it in `start_fe.sh` scripts. Debug java parameter: `-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005` is just copied from IDEA. ![IDE](/assets/images/ide-1-98b198df0ef5da5f566e18eba4c5cf5f.png) ##### BE[​](#be "Direct link to BE") It is recommended to run `mvn install -DskipTests` first in `fe` to generate the FE-side Java thrift/proto sources. For BE, run the CMake configure/build flow once. The BE thrift/protobuf headers are materialized by CMake during configure/build into the active build directory (for example `be/build_Release/gensrc/gen_cpp`), so you no longer need `cd gensrc && make` for those files. If you need the shared script outputs first, run `make -C gensrc script`. Use Clion to open `be` folder. Enter `Settings`, add `Toolchains`. Add a remote server first, then setup Build Tool, C and C++ Compiler separately. ![IDE](/assets/images/ide-2-d70ce01d1c2227a2f753da205414c5e4.png) In `Settings` / `Deployment`. Change folder `mappings`. ![IDE](/assets/images/ide-3-7f7db478d4179890a26040f538cc6e53.png) In `Settings` / `Cmake`. Change Toolchain to be the remote toolchain just added. Add the following environment variables: ```bash JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 STARROCKS_GCC_HOME=/usr/ STARROCKS_THIRDPARTY=/root/starrocks/thirdparty ``` Notice: Be careful not to check `Include system environment variables`. ![IDE](/assets/images/ide-4-44ca169b84d329083c3bd624627fd200.png) ![IDE](/assets/images/ide-5-b84ccafdc6ed800da98838a727ab1bcf.png) From here on, all setup is complete. After Clion and the remote server are synchronized for a while, the code jump will work normally. ###### Debug[​](#debug "Direct link to Debug") BE debug is a little difficult, you have to use gdb in your remote server. Of course, you can use gdb server + Clion remote gdb, but I don't recommend it, it's too stuck. We need to change `start_backend.sh` script from: ```bash if [ ${RUN_BE} -eq 1 ]; then echo "start time: "$(date) >> $LOG_DIR/be.out if [ ${RUN_DAEMON} -eq 1 ]; then nohup ${STARROCKS_HOME}/lib/starrocks_be "$@" >> $LOG_DIR/be.out 2>&1 > $LOG_DIR/be.out 2>&1 > $LOG_DIR/be.out if [ ${RUN_DAEMON} -eq 1 ]; then nohup ${STARROCKS_HOME}/lib/starrocks_be "$@" >> $LOG_DIR/be.out 2>&1 If you face the error report when debugging for lakehouse, just add `handle SIGSEGV nostop noprint pass` in `~/.gdbinit`. ###### LLVM[​](#llvm "Direct link to LLVM") Of course, you can use LLVM tools to development be. Ubuntu LLVM installtion refer to: Then use the command: `CC=clang-15 CXX=clang++-15 ./build.sh` to compile be. But the premise is that your thirdparty has been compiled with gcc. #### Last[​](#last "Direct link to Last") Feel free to contribute codes to StarRocks. 🫵 --- ### Setup StarRocks FE development environment on IDEA This tutorial is based on macOS and has been tested on Apple Chip(M1, M2). Even if you are not using macOS, you can also refer to this tutorial. #### Requirements[​](#requirements "Direct link to Requirements") ##### Thrift 0.13[​](#thrift-013 "Direct link to Thrift 0.13") There is no 0.13 version of Thrift in the official brew repository; one of our committers created a version in their repo to install. ```bash brew install alberttwong/thrift/thrift@0.13 ``` After installing Thrift successfully, you can check by executing following command: ```bash $ thrift -version Thrift version 0.13.0 ``` ##### Protobuf[​](#protobuf "Direct link to Protobuf") Just use the latest version v3, because the latest version of Protobuf is compatible with the v2 version of the Protobuf used in StarRocks. ```bash brew install protobuf ``` ##### Maven[​](#maven "Direct link to Maven") ```text brew install maven ``` ##### Openjdk 1.8 or 11[​](#openjdk-18-or-11 "Direct link to Openjdk 1.8 or 11") ```bash brew install openjdk@11 ``` ##### Python3[​](#python3 "Direct link to Python3") MacOS is already installed by default. Everyone's Thrift and Protobuf installation directories may be different, you can use the brew list command to inspect: ```bash brew list thrift@0.13.0 brew list protobuf ``` #### Configure the StarRocks[​](#configure-the-starrocks "Direct link to Configure the StarRocks") ##### Download the StarRocks[​](#download-the-starrocks "Direct link to Download the StarRocks") ```text git clone https://github.com/StarRocks/starrocks.git ``` ##### Setup thirdparty directory[​](#setup-thirdparty-directory "Direct link to Setup thirdparty directory") Create `installed/bin` directory in `thirdparty`. ```bash cd starrocks && mkdir -p thirdparty/installed/bin ``` Then create soft link for Thrift and Protobuf respectively. ```bash ln -s /opt/homebrew/bin/thrift thirdparty/installed/bin/thrift ln -s /opt/homebrew/bin/protoc thirdparty/installed/bin/protoc ``` ##### Setting environment variables[​](#setting-environment-variables "Direct link to Setting environment variables") ```bash export JAVA_HOME="/opt/homebrew/Cellar/openjdk@11/11.0.15" # Caution: The jdk version may be different in you desktop export PYTHON=/usr/bin/python3 export STARROCKS_THIRDPARTY=$(pwd)/thirdparty # Caution: Make sure you are in the starrocks directory ``` #### Generate source code[​](#generate-source-code "Direct link to Generate source code") Many source files in FE need to be generated manually, otherwise IDEA will report an error due to missing files. Execute the following command to automatically generate: ```bash make -C gensrc script ./build.sh --be --configure-only cmake --build be/build_Release --target be_proto_codegen be_thrift_codegen ``` The generated BE thrift/protobuf C++ files will appear under the active build directory, for example `be/build_Release/gensrc/gen_cpp`. #### Compile FE[​](#compile-fe "Direct link to Compile FE") Enter `fe` directory and use Maven to compile: ```bash cd fe mvn install -DskipTests ``` #### Open StarRocks in IDEA[​](#open-starrocks-in-idea "Direct link to Open StarRocks in IDEA") 1. Open the `StarRocks` directory in IDEA. 2. Add Coding style setting To standardize the coding style, you should import the `fe/starrocks_intellij_style.xml` code style file in IDEA. ![image-20220701193938856](/assets/images/IDEA-2-8dd4f3fe0da44fb9d1c6f4f035b0a432.png) #### Run StarRocks FE in MacOS[​](#run-starrocks-fe-in-macos "Direct link to Run StarRocks FE in MacOS") Use IDEA to open the `fe` directory. If you execute the Main function directly in `StarRocksFE.java`, some errors will be reported. You only need to do some simple settings to run it smoothly. **NOTICE:** `StarRocksFE.java` is in the `fe/fe-core/src/main/java/com/starrocks` directory. 1. Copy the conf, bin and webroot directories from the StarRocks directory to `fe` directory: ```bash cp -r conf fe/conf cp -r bin fe/bin cp -r webroot fe/webroot ``` 2. Enter the `fe` directory and create the log and meta folders under the `fe` directory: ```bash cd fe mkdir log mkdir meta ``` 3. Set the environment variable, as shown in the following figure: ![image-20220701193938856](/assets/images/IDEA-1-10bdc0d1013b0b63c1ee394608e3091a.png) ```bash export PID_DIR=/Users/smith/Code/starrocks/fe/bin export STARROCKS_HOME=/Users/smith/Code/starrocks/fe export LOG_DIR=/Users/smith/Code/starrocks/fe/log ``` 4. Modify the priority\_networks in `fe/conf/fe.conf` to `127.0.0.1/24` to prevent FE from using the current computer's LAN IP and cause the port fail to bind. 5. Then you have run StarRocks FE successfully. #### DEBUG StarRocks FE in MacOS[​](#debug-starrocks-fe-in-macos "Direct link to DEBUG StarRocks FE in MacOS") If you started the FE with the debug option, you can then attach the IDEA debugger to the FE process. ```text ./start_fe.sh --debug ``` See . --- ### Contribute to StarRocks Contributing to StarRocks is cordially welcome from everyone. Contributing to StarRocks is not limited to contributing code. Below, we list different approaches to contributing to our community. | Report a bug | You can [file an issue](https://github.com/StarRocks/starrocks/issues/new/choose) to report a bug with StarRocks. You can also click `Feedback` in the upper-right corner of the page you are reading in the Documentation Site to report a bug. | | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Contribute code** | You can contribute your code by fixing a bug or implementing a feature. | | **Contribute test case** | You can contribute your test cases. | | **Help review code** | If you are an active contributor or committer of StarRocks, you can help us review the pull requests (PRs). | | **Contribute documentation** | StarRocks community maintains a tremendous amount of documentation both in Chinese and English. You can contribute documentation changes by fixing a documentation bug or proposing a new piece of content. | | **Help StarRocks users** | You can help newcomers who meet difficulties in our community. | | **Spread the word about StarRocks** | You can author an article or give a talk about us to help spread our technology to the world. | > **NOTE** > > To contribute documentation, remember to **sign off** your commit using `git commit -s`. Otherwise, the Developer Certificate of Origin (DCO) check will fail and the PR may be blocked. In addition, prefix `[Doc]` to your PR title and select the **Doc** check box. ![Doc title](/assets/images/doctitle-814494a3b89c3e9762496f5bc4fbcc72.png) #### Community resources[​](#community-resources "Direct link to Community resources") The best place to get a wide variety of help about StarRocks is via StarRocks's [Slack Channel](https://docs.starrocks.io/join/) For contribution-related discussions, please go to the **#contributing-to-starrocks** channel. You can also report issues and problems, or suggest new features, on [GitHub](https://github.com/StarRocks/starrocks/). #### Code of conduct[​](#code-of-conduct "Direct link to Code of conduct") Our community strictly adheres to the [code of conduct](https://github.com/StarRocks/starrocks/blob/main/CODE_OF_CONDUCT.md). #### Community roles[​](#community-roles "Direct link to Community roles") ![Community roles](/assets/images/contri-1-3648053cec929acf5bebd3710cb44476.png) Everyone is encouraged to participate in the StarRocks project. Anyone can make an impact by simply being involved in the discussions about new features, project roadmap, architecture, and even reporting issues you are facing. The roles listed below are a few possible ways to get involved in the community, it also defines what is expected from each role. #### User group[​](#user-group "Direct link to User group") ##### Participants[​](#participants "Direct link to Participants") Participants are actively involved in the community and work to make StarRocks better for everyone. As a participant, you can submit issue reports on GitHub, contribute test cases, translate/modify documentation, help answer user questions in forums or communities, participate in events, share your experience with StarRocks, star StarRocks on GitHub, and more. Expectations and responsibilities: * Follow the community's [code of conduct](https://github.com/StarRocks/starrocks/blob/main/CODE_OF_CONDUCT.md). * Be involved in discussions and community events. * Share feedback with the community so everyone else knows what is/isn’t working. * Suggest improvements. ##### Champions[​](#champions "Direct link to Champions") StarRocks Community Champions are a group of passionate community evangelists and pioneers who are well-versed in StarRocks technology. They enjoy sharing the latest developments and products in the community, and their enthusiasm drives the progress and development of the community. ###### Benefits[​](#benefits "Direct link to Benefits") Product * Early access to new products/features. * Participate in exclusive meetings with the project core team. Community * Listed as a StarRocks champion on the website. * Exclusive prizes and badges. Events * Free access to StarRocks' events. * Expense reimbursement for travel and accommodations for events. * VIP seats and VIP events. ###### Requirements[​](#requirements "Direct link to Requirements") 1. Have published at least 3 StarRocks-related original and quality technical content (including articles, tutorials, videos, etc.). 2. Participate in at least 2 online/offline technical sharing sessions as a speaker. 3. Help answer user questions on Slack and other channels. ###### Other Requirements[​](#other-requirements "Direct link to Other Requirements") 1. StarRocks Champions must abide by the StarRocks Community [Code of Conduct.](https://github.com/StarRocks/starrocks/blob/main/CODE_OF_CONDUCT.md) 2. StarRocks Champions are eligible for one year, and must reapply each year based on their contributions in the past year. #### Developer Group[​](#developer-group "Direct link to Developer Group") ##### Contributor[​](#contributor "Direct link to Contributor") Everyone who contributes can become a StarRocks contributor. The members will provide mentorship and guidance when new contributors need assistance. ###### How to become a Contributor?[​](#how-to-become-a-contributor "Direct link to How to become a Contributor?") * 1 merged PR in any StarRocks' public repos. As a Contributor, we expect you to * Actively participate in StarRocks' project development. * Participate in community events (meetups, hackathons, etc.). * Learn and help others learn StarRocks-related technologies. ###### Privileges[​](#privileges "Direct link to Privileges") * Be listed as a StarRocks contributor. * Be awarded a StarRocks Contributor e-certificate. ##### Active Contributor[​](#active-contributor "Direct link to Active Contributor") Active contributors are contributors who have made outstanding contributions and sustained commitment to StarRocks. They actively participate in the community by contributing code, improving docs, and helping others. ###### How to become an Active Contributor?[​](#how-to-become-an-active-contributor "Direct link to How to become an Active Contributor?") * Have 5 merged PRs or fixed major bugs. * Participate in more than 5 code reviews. * Actively participate in community events such as online/offline meetups and community discussions. ###### Responsibilities and privileges[​](#responsibilities-and-privileges "Direct link to Responsibilities and privileges") * Join the community meeting and discussion. * Mentor and guide new contributors. * Be listed as a StarRocks Active Contributor. * Be awarded a StarRocks Active Contributor e-certificate. ##### Committer[​](#committer "Direct link to Committer") Committers are promoted from Active Contributors. They have the authority to merge PRs into master branches and are responsible for the planning and maintenance of StarRocks. They also are active members in sharing their knowledge with the community. ###### How to become a Committer?[​](#how-to-become-a-committer "Direct link to How to become a Committer?") * Have a deep understanding of StarRocks' principles and future plans. * Have the ability to deal with various issues that arise in the project promptly. * Lead a major development, write and revise related documents. * Receive at least two PMC nominations and pass voting. ###### Responsibilities and privileges[​](#responsibilities-and-privileges-1 "Direct link to Responsibilities and privileges") * Mentor and guide other memberships in the community. * Ensure continued health of subproject. * Be granted write access to StarRocks repos (to be specified). * Be listed as a StarRocks Committer. * Be awarded a StarRocks Committer e-certificate. ##### PMC[​](#pmc "Direct link to PMC") PMCs are promoted from Committers. They have the authority to merge merge PRs into master branches and are responsible for the planning and maintenance of StarRocks. They also are active members in sharing their knowledge with the community. ###### How to become a PMC?[​](#how-to-become-a-pmc "Direct link to How to become a PMC?") * In-depth understanding of StarRocks principles and a clear understanding of StarRocks' future plans. * Have the ability to deal with project issues promptly. * Lead project development and iterations, and steer the overall direction of the project. * Receive at least two PMC nominations and pass voting. ###### Responsibilities and privileges[​](#responsibilities-and-privileges-2 "Direct link to Responsibilities and privileges") * Mentor and guide other memberships in the community. * Ensure continued health of the project, such as code quality and test coverage. * Make and approve technical design decisions. * Define milestones and releases. * Vote and promote new committers and PMCs. * Be listed as a StarRocks PMC. * Be awarded a StarRocks PMC e-certificate. #### Contribution process[​](#contribution-process "Direct link to Contribution process") If you don't know how to get started, this is the process we suggest for contributions. This process is designed to help reduce your learning curve and get your pull requests merged more efficiently! 1. Sign the [Contributor License Agreement (CLA)](https://cla-assistant.io/StarRocks/starrocks). 2. Start a discussion by creating a Github [issue](https://github.com/StarRocks/starrocks/issues), or asking on [Slack channel](https://docs.starrocks.io/join/) unless the change is trivial). Before getting your hands on codes, you should comment in the issue body, and inform the maintainer to assign you the issue that you wish to solve. It is recommended to share your plan on how to solve this problem in the issue body as well. * This step helps you identify possible collaborators and reviewers. * Will the change conflict with another change in progress? If so, work with others to minimize impact. * Is this change major? If so, work with others to break the change into smaller steps. 3. Implement the change. * If the change is major, split it into smaller PRs. * Include tests and documentation as necessary. 4. Create a Github [pull request](https://github.com/StarRocks/starrocks/pulls): In StarRocks community, we follow the fork-and-merge GitHub workflow when contributing code. * Create a fork of StarRocks in your GitHub account. * Clone this forked repository to your computer. * Check out a new branch based on the branch you expect to contribute to. * Commit your code changes to the new branch. * Push the branch with code changes to GitHub. * Create a PR to submit your code changes. It is recommended to submit ONE commit in ONE PR. You can follow the [PR templated](https://github.com/StarRocks/starrocks/blob/main/.github/PULL_REQUEST_TEMPLATE.md) when submitting a PR. * Make sure the pull request passes the tests in CI. 5. Review is required by at least 2 reviewers: * For StarRocks project, we require at least 2 lgtm from reviewers (committers) to merge the pull request. * This normally happens within a few days, but may take longer if the change is major, complex, or if a critical reviewer is unavailable. (feel free to ping the reviewer on the pull request). 6. Maintainers merge the pull request after the final changes are accepted. --- ### Jemalloc Heap Profiling This topic describes how to enable and visualize the Jemalloc Heap Profile for StarRocks. note * Enabling Jemalloc Heap Profiling may have an impact on StarRocks' performance. * This solution is available only for StarRocks v3.1.6 and later. #### Enable Jemalloc Heap Profile[​](#enable-jemalloc-heap-profile "Direct link to Enable Jemalloc Heap Profile") Syntax: ```sql ADMIN EXECUTE ON 'System.print(HeapProf.getInstance().enable_prof())' ``` `be_id`: The ID of BE/CN node. You can get the ID by running SHOW BACKENDS or SHOW COMPUTE NODES. Example: ```sql mysql> admin execute on 10001 'System.print(HeapProf.getInstance().enable_prof())'; +----------------------+ | result | +----------------------+ | instance of HeapProf | +----------------------+ 1 row in set (0.00 sec) ``` You can check whether Jemalloc Heap Profile is enabled or not by using the following syntax: ```sql ADMIN EXECUTE ON 'System.print(HeapProf.getInstance().has_enable())' ``` Example: ```sql mysql> admin execute on 10001 'System.print(HeapProf.getInstance().has_enable())'; +--------+ | result | +--------+ | true | +--------+ 1 row in set (0.01 sec) ``` You can disable Jemalloc Heap Profile by using the following syntax: ```sql ADMIN EXECUTE ON 'System.print(HeapProf.getInstance().disable_prof())' ``` Example: ```sql mysql> admin execute on 10001 'System.print(HeapProf.getInstance().disable_prof())'; +----------------------+ | result | +----------------------+ | instance of HeapProf | +----------------------+ 1 row in set (0.00 sec) ``` #### Collect Jemalloc Heap Profile[​](#collect-jemalloc-heap-profile "Direct link to Collect Jemalloc Heap Profile") Syntax: ```sql ADMIN EXECUTE ON 'System.print(HeapProf.getInstance().dump_dot_snapshot())' ``` Example: ```sql mysql> admin execute on 10001 'System.print(HeapProf.getInstance().dump_dot_snapshot())'; +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | result | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | digraph "/home/disk/opt/env/default/be/lib/starrocks_be; 1.0 MB" { | | node [width=0.375,height=0.25]; | | Legend [shape=box,fontsize=24,shape=plaintext,label="/home/disk/opt/env/default/be/lib/starrocks_be\lTotal MB: 1.0\lFocusing on: 1.0\lDropped nodes with <= 0.0 abs(MB)\lDropped edges with <= 0.0 MB\l"]; | | N1 [label="brpc\nInputMessenger\nOnNewMessages\n0.0 (0.0%)\rof 1.0 (100.0%)\r",shape=box,fontsize=8.0]; | | N2 [label="brpc\nSocket\nProcessEvent\n0.0 (0.0%)\rof 1.0 (100.0%)\r",shape=box,fontsize=8.0]; | | N3 [label="bthread\nTaskGroup\ntask_runner\n0.0 (0.0%)\rof 1.0 (100.0%)\r",shape=box,fontsize=8.0]; | | N4 [label="bthread_make_fcontext\n0.0 (0.0%)\rof 1.0 (100.0%)\r",shape=box,fontsize=8.0]; | | N5 [label="brpc\nInputMessenger\nCutInputMessage\n0.0 (0.0%)\rof 0.5 (50.1%)\r",shape=box,fontsize=8.0]; | | N6 [label="brpc\npolicy\nParseRpcMessage\n0.5 (50.1%)\r",shape=box,fontsize=43.4]; | | N7 [label="brpc\nProcessInputMessage\n0.0 (0.0%)\rof 0.5 (49.9%)\r",shape=box,fontsize=8.0]; | | N8 [label="brpc\npolicy\nProcessRpcRequest\n0.0 (0.0%)\rof 0.5 (49.9%)\r",shape=box,fontsize=8.0]; | | N9 [label="starrocks\nPInternalServiceImplBase\nexecute_command\n0.0 (0.0%)\rof 0.5 (49.9%)\r",shape=box,fontsize=8.0]; | | N10 [label="starrocks\nStorageEngineRef\nbind\n0.0 (0.0%)\rof 0.5 (49.9%)\r",shape=box,fontsize=8.0]; | | N11 [label="starrocks\nexecute_command\n0.0 (0.0%)\rof 0.5 (49.9%)\r",shape=box,fontsize=8.0]; | | N12 [label="starrocks\nexecute_script\n0.0 (0.0%)\rof 0.5 (49.9%)\r",shape=box,fontsize=8.0]; | | N13 [label="std\nmake_unique\n0.5 (49.9%)\r",shape=box,fontsize=43.3]; | | N2 -> N1 [label=1.0, weight=16398, style="setlinewidth(2.000000)"]; | | N3 -> N2 [label=1.0, weight=16398, style="setlinewidth(2.000000)"]; | | N4 -> N3 [label=1.0, weight=16398, style="setlinewidth(2.000000)"]; | | N1 -> N5 [label=0.5, weight=10102, style="setlinewidth(2.000000)"]; | | N5 -> N6 [label=0.5, weight=10102, style="setlinewidth(2.000000)"]; | | N9 -> N11 [label=0.5, weight=10086, style="setlinewidth(2.000000)"]; | | N12 -> N10 [label=0.5, weight=10086, style="setlinewidth(2.000000)"]; | | N11 -> N12 [label=0.5, weight=10086, style="setlinewidth(2.000000)"]; | | N1 -> N7 [label=0.5, weight=10086, style="setlinewidth(2.000000)"]; | | N7 -> N8 [label=0.5, weight=10086, style="setlinewidth(2.000000)"]; | | N10 -> N13 [label=0.5, weight=10086, style="setlinewidth(2.000000)"]; | | N8 -> N9 [label=0.5, weight=10086, style="setlinewidth(2.000000)"]; | | } | +-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 29 rows in set (30.22 sec) ``` #### Visualize Jemalloc Heap Profile[​](#visualize-jemalloc-heap-profile "Direct link to Visualize Jemalloc Heap Profile") Copy the profile text you collected in the last step, and paste it to [GraphvizOnline](https://dreampuf.github.io/GraphvizOnline/). You can then download the visualized Heap Profile. Example: ![Example - Visualized Heap Profile](/assets/images/visualized_heap_profile-886621b6e11bd0a830b24f807c7409ea.png) --- ### Query Trace Profile This topic introduces how to obtain and analyze query trace profiles. A query trace profile records the debug information for a specified query statement, including time costs, variables & values, and logs. Such information is categorized into several modules, allowing you to debug and identify the performance bottlenecks from different aspects. This feature is supported from v3.2.0 onwards. #### Syntax[​](#syntax "Direct link to Syntax") You can use the following syntax to obtain the trace profile of a query: ```sql TRACE { TIMES | VALUES | LOGS | ALL } [ ] ``` * `TIMES`: Traces the time costs of events in each stage of the specified query. * `VALUES`: Traces the variables and their values of the specified query. * `LOGS`: Traces the log records of the specified query. * `ALL`: Lists all the `TIMES`, `VALUES`, and `LOGS` information in chronological order. * ``: The module you want to trace information from. Valid values: * `BASE`: The base module. * `MV`: The materialized view module. * `OPTIMIZER`: The optimizer module. * `SCHEDULE`: The schedule module. * `EXTERNAL`: The external table-related module. If no module is specified, `BASE` is used. * ``: The query statement whose query trace profile you want to obtain. #### Use cases[​](#use-cases "Direct link to Use cases") ##### Trace the time costs of a query[​](#trace-the-time-costs-of-a-query "Direct link to Trace the time costs of a query") The following example traces the time costs of a query's optimizer module. ```plain MySQL > TRACE TIMES OPTIMIZER SELECT * FROM t1 JOIN t2 ON t1.v1 = t2.v1; +---------------------------------------------------------------------+ | Explain String | +---------------------------------------------------------------------+ | 2ms|-- Total[1] 15ms | | 2ms| -- Analyzer[1] 1ms | | 4ms| -- Transformer[1] 1ms | | 6ms| -- Optimizer[1] 11ms | | 6ms| -- preprocessMvs[1] 0 | | 6ms| -- RuleBaseOptimize[1] 3ms | | 6ms| -- RewriteTreeTask[41] 2ms | | 7ms| -- PushDownJoinOnClauseRule[1] 0 | | 7ms| -- PushDownPredicateProjectRule[2] 0 | | 7ms| -- PushDownPredicateScanRule[2] 0 | | 8ms| -- MergeTwoProjectRule[3] 0 | | 8ms| -- PushDownJoinOnExpressionToChildProject[1] 0 | | 8ms| -- PruneProjectColumnsRule[6] 0 | | 8ms| -- PruneJoinColumnsRule[2] 0 | | 8ms| -- PruneScanColumnRule[4] 0 | | 9ms| -- PruneSubfieldRule[2] 0 | | 9ms| -- PruneProjectRule[6] 0 | | 9ms| -- PartitionPruneRule[2] 0 | | 9ms| -- DistributionPruneRule[2] 0 | | 9ms| -- MergeProjectWithChildRule[3] 0 | | 10ms| -- CostBaseOptimize[1] 6ms | | 10ms| -- OptimizeGroupTask[6] 0 | | 10ms| -- OptimizeExpressionTask[9] 0 | | 10ms| -- ExploreGroupTask[4] 0 | | 10ms| -- DeriveStatsTask[9] 3ms | | 13ms| -- ApplyRuleTask[16] 0 | | 13ms| -- OnlyScanRule[2] 0 | | 14ms| -- HashJoinImplementationRule[2] 0 | | 14ms| -- EnforceAndCostTask[12] 1ms | | 14ms| -- OlapScanImplementationRule[2] 0 | | 15ms| -- OnlyJoinRule[2] 0 | | 15ms| -- JoinCommutativityRule[1] 0 | | 16ms| -- PhysicalRewrite[1] 0 | | 17ms| -- PlanValidate[1] 0 | | 17ms| -- InputDependenciesChecker[1] 0 | | 17ms| -- TypeChecker[1] 0 | | 17ms| -- CTEUniqueChecker[1] 0 | | 17ms| -- ExecPlanBuild[1] 0 | | Tracer Cost: 273us | +---------------------------------------------------------------------+ 39 rows in set (0.029 sec) ``` In the **Explain String** returned by the TRACE TIMES statement, each row (except the last row) corresponds to an event in the specified module (stage) of the query. The last row `Tracer Cost` records the time cost of the tracing process. Take `| 4ms| -- Transformer[1] 1ms` as an example: * The left column records the time point, in the lifecycle of the query, when the event was first executed. * In the right column, following the consecutive hyphens is the name of the event, for example, `Transformer`. * Following the event name, the number in the brackets (`[1]`) indicates the number of times the event was executed. * The last part of this column is the overall time cost of the event, for example, `1ms`. * The records of events are indented based on the depth of the method stack. That is to say, in this example, the first execution of `Transformer` always happens within `Total`. ##### Trace the variables of a query[​](#trace-the-variables-of-a-query "Direct link to Trace the variables of a query") The following example traces the variables & values of a query's MV module. ```plain MySQL > TRACE VALUES MV SELECT t1.v2, sum(t1.v3) FROM t1 JOIN t0 ON t1.v1 = t0.v1 GROUP BY t1.v2; +----------------------------+ | Explain String | +----------------------------+ | 32ms| mv2: Rewrite Succeed | | Tracer Cost: 66us | +----------------------------+ 2 rows in set (0.045 sec) ``` The structure of the **Explain String** returned by the TRACE VALUES statement is similar to that of the TRACE TIMES statement, except that the right column records the variables and settings of the event in the specified module. The above example records that the materialized view is successfully used to rewrite the query. ##### Trace the logs of a query[​](#trace-the-logs-of-a-query "Direct link to Trace the logs of a query") The following example traces the logs of a query's MV module. ```plain MySQL > TRACE LOGS MV SELECT v2, sum(v3) FROM t1 GROUP BY v2; +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Explain String | +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | 3ms| [MV TRACE] [PREPARE cac571e8-47f9-11ee-abfb-2e95bcb5f199][mv2] [SYNC=false] Prepare MV mv2 success | | 3ms| [MV TRACE] [PREPARE cac571e8-47f9-11ee-abfb-2e95bcb5f199][GLOBAL] [SYNC=false] RelatedMVs: [mv2], CandidateMVs: [mv2] | | 4ms| [MV TRACE] [PREPARE cac571e8-47f9-11ee-abfb-2e95bcb5f199][GLOBAL] [SYNC=true] There are no related mvs for the query plan | | 35ms| [MV TRACE] [REWRITE cac571e8-47f9-11ee-abfb-2e95bcb5f199 TF_MV_AGGREGATE_SCAN_RULE mv2] Rewrite ViewDelta failed: cannot compensate query by using PK/FK constraints | | 35ms| [MV TRACE] [REWRITE cac571e8-47f9-11ee-abfb-2e95bcb5f199 TF_MV_ONLY_SCAN_RULE mv2] MV is not applicable: mv expression is not valid | | 43ms| Query cannot be rewritten, please check the trace logs or `set enable_mv_optimizer_trace_log=on` to find more infos. | | Tracer Cost: 400us | +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 7 rows in set (0.056 sec) ``` Alternatively, you can print these logs in the FE log file **fe.log** by setting the variable `trace_log_mode` as follows: ```sql SET trace_log_mode='file'; ``` The default value of `trace_log_mode` is `command`, indicating that the logs are returned as the **Explain String** as shown above. If you set its value to `file`, the logs are printed in the FE log file **fe.log** with the class name being `FileLogTracer`. After you set `trace_log_mode` to `file`, no logs will be returned when you execute the TRACE LOGS statement. Example: ```plain MySQL > TRACE LOGS OPTIMIZER SELECT v1 FROM t1 ; +---------------------+ | Explain String | +---------------------+ | Tracer Cost: 3422us | +---------------------+ 1 row in set (0.023 sec) ``` The log will be printed in **fe.log**. ![img](/assets/images/query_trace_profile-bcdf3ff971f4c20f7f9d591c97b2c5c8.png) --- ### Trace ##### Background[​](#background "Direct link to Background")  A Distributed Trace, more commonly known as a Trace, records the paths taken by requests (made by an application or end-user) as they propagate through multi-service architectures, like microservice and serverless applications. Without tracing, it is challenging to pinpoint the cause of performance problems in a distributed system. It improves the visibility of our application or system’s health and lets us debug behavior that is difficult to reproduce locally. Tracing is essential for distributed systems, which commonly have nondeterministic problems or are too complicated to reproduce locally.  Tracing makes debugging and understanding distributed systems less daunting by breaking down what happens within a request as it flows through a distributed system. A Trace is made of one or more Spans. The first Span represents the Root Span. Each Root Span represents a request from start to finish. The Spans underneath the parent provide a more in-depth context of what occurs during a request (or what steps make up a request). Many Observability back-ends visualize Traces as waterfall diagrams that may look something like this picture. ![trace\_pic1](/assets/images/trace_pic1-1d5e52ca0f5dc47f1b716e1106fdc267.png)  Waterfall diagrams show the parent-child relationship between a Root Span and its child Spans. When a Span encapsulates another Span, this also represents a nested relationship.  Recently SR added a tracing framework. It leverages opentelemetry and jaeger to trace distributed events in the system. * Opentelemetry is an instrumentation/tracing SDK. Developers can use it to instrument code and emit tracing data to an observability backend. It supports many languages. We use java and CPP SDK in SR. * Currently, Jaeger is used as the observability backend. ##### Basic Usage[​](#basic-usage "Direct link to Basic Usage") Steps to enable tracing in SR: 1. Install [Jaeger](https://www.jaegertracing.io/docs/1.76/getting-started/) The guide above uses docker. For simplicity, you can also just download [binary package](https://github.com/jaegertracing/jaeger/releases) and run locally. ```text decster@decster-MS-7C94:~/soft/jaeger-1.31.0-linux-amd64$ ll total 215836 drwxr-xr-x 2 decster decster 4096 02-05 05:01:30 ./ drwxrwxr-x 28 decster decster 4096 05-18 18:24:07 ../ -rwxr-xr-x 1 decster decster 19323884 02-05 05:01:31 example-hotrod* -rwxr-xr-x 1 decster decster 23430444 02-05 05:01:29 jaeger-agent* -rwxr-xr-x 1 decster decster 51694774 02-05 05:01:29 jaeger-all-in-one* -rwxr-xr-x 1 decster decster 41273869 02-05 05:01:30 jaeger-collector* -rwxr-xr-x 1 decster decster 37576660 02-05 05:01:30 jaeger-ingester* -rwxr-xr-x 1 decster decster 47698843 02-05 05:01:30 jaeger-query* decster@decster-MS-7C94:~/soft/jaeger-1.31.0-linux-amd64$ ./jaeger-all-in-one ``` 2. Config FE\&FE to enable tracing. Currently, opentelemetry java & cpp sdk use different protocols, java uses grpc proto, while cpp uses thrift\&UDP, so the endpoint ports are different. ```text fe.conf # Enable jaeger tracing by setting jaeger_grpc_endpoint # jaeger_grpc_endpoint = http://localhost:14250 be.conf # Enable jaeger tracing by setting jaeger_endpoint # jaeger_endpoint = localhost:6831 ``` 3. Open jaeger web UI, usually in `http://localhost:16686/search` 4. Do some data ingestion (streamload/insert into) and search TXN traces on web UI ![trace\_pic2.png](/assets/images/trace_pic2-7667cb8c15ecdfce6a0eb93a365a96e5.png)(trace\_pic2.png) ![trace\_pic3.png](/assets/images/trace_pic3-31201861601577398a87e72701f9f3fa.png)(trace\_pic3.png) ##### Adding traces[​](#adding-traces "Direct link to Adding traces") * To add trace, first get familiar with basic concepts like tracer, span, trace propagation read the [observability primer](https://opentelemetry.io/docs/concepts/observability-primer/) * Read utility class and it's usages in SR: TraceManager.java(java) `common/tracer.h/cpp (cpp)`, it's current usage(like write txn(load/insert/update/delete) trace, and its propagation to BE). * Add your own trace --- ### Version Release Guide Version naming is detailed in the [versioning](https://docs.starrocks.io/docs/introduction/versioning.md) documentation. Read that page first to understand **major**, **minor**, and **patch** versioning. #### Release Plan[​](#release-plan "Direct link to Release Plan") * Release a minor version approximately every four months. * Maintain the three latest minor versions (**minor** is the second number in the dot separated version, for example, in 3.4.2 **4** is the minor version). With a minor version released every four months, a minor version should be expected to be supported for at most one year. * Release a patch version within 2-3 weeks for the minor version in maintenance. #### Pull Request Type[​](#pull-request-type "Direct link to Pull Request Type") Every pull request in StarRocks should be titled with a type, including **feature**, **enhancement**, and **bugfix**. ##### Feature[​](#feature "Direct link to Feature") * Definition: A feature is a new capability or functionality that did not previously exist in the database. It adds new behavior or significantly extends the existing functionality. * Examples: * Adding a new type of data structure (e.g., a new table type or index type). * Implementing a new query language feature (e.g., a new SQL function or operator). * Introducing a new API endpoint or interface for interacting with the database. ##### Enhancement[​](#enhancement "Direct link to Enhancement") * Definition: An enhancement is an improvement to an existing feature or functionality. It does not introduce entirely new behavior but makes existing features better, faster, or more efficient. * Examples: * Optimizing the performance of a query execution plan. * Improving the user interface of the database management tool. * Enhancing the security features by adding more granular access controls. ##### Bugfix[​](#bugfix "Direct link to Bugfix") * Definition: A bugfix is a correction of an error or flaw in the existing code. It addresses issues that prevent the database from functioning correctly or as intended. * Examples: * Fixing a crash that occurs under certain query conditions. * Correcting an incorrect result returned by a query. * Resolving a memory leak or resource management issue. #### Cherry-pick Rule[​](#cherry-pick-rule "Direct link to Cherry-pick Rule") We define some status for a minor version to assist cherry-pick management. You can find the version status in the `.github/.status` file. For example, at the time this document was published, StarRocks version 3.4 is in the `feature-freeze` state and version 3.3 is `bugfix-only`. To verify this: ```bash git switch branch-3.3 cat .github/.status ``` ```bash bugfix-only ``` 1. `open`: All types of pull requests can be merged, including feature, enhancement, and bugfix. 2. `feature-freeze`: Only enhancement and bugfix pull requests can be merged. 3. `bugfix-only`: Only bugfix pull requests can be merged. 4. `code-freeze`: No pull requests can be merged except critical CVE fixes. The minor version status will change with some baseline triggers, as shown below, and it can also be changed in advance if necessary. 1. When the minor version branch is created, it becomes `open` and stays `open` until it is released. 2. When the minor version is released, it becomes `feature-freeze`. 3. When the next minor version is released, the previous minor version becomes `bugfix-only`. 4. The minor version stays as `bugfix-only` until three more minor versions are released, and then becomes `code-freeze`. ##### Example[​](#example "Direct link to Example") * branch-5.1 is created, this branch is in the `open` state until it passes through release candidate and is publicly released. * Once version 5.1 is publicly released it enters the `feature-freeze` state. * Once version 5.2 is publicly released 5.1 switches to `bugfix-only`. * When versions 5.1, 5.2, 5.3, and 5.4 are all released: * 5.4 is in the `feature-freeze` state * 5.3 is in the `bugfix-only` state * 5.2 is also in the `bugfix-only` state * 5.1 is in the `code-freeze` state --- ## Faq ### Datalake FAQ ### Data lake FAQ This topic describes some commonly asked questions (FAQ) about data lake and provides solutions to these issues. Some metrics mentioned in this topic can be obtained only from the profiles of the SQL queries. To obtain the profiles of SQL queries, you must specify `set enable_profile=true`. #### Slow HDFS DataNodes[​](#slow-hdfs-datanodes "Direct link to Slow HDFS DataNodes") ##### Issue description[​](#issue-description "Direct link to Issue description") When you access the data files stored in your HDFS cluster, you may find a huge difference between the values of the `__MAX_OF_FSIOTime` and `__MIN_OF_FSIOTime` metrics from the profiles of the SQL queries you run. This indicates that some DataNodes in the HDFS cluster are slow. The following example is a typical profile that indicates a slow HDFS DataNode issue: ```plaintext - InputStream: 0 - AppIOBytesRead: 22.72 GB - __MAX_OF_AppIOBytesRead: 187.99 MB - __MIN_OF_AppIOBytesRead: 64.00 KB - AppIOCounter: 964.862K (964862) - __MAX_OF_AppIOCounter: 7.795K (7795) - __MIN_OF_AppIOCounter: 1 - AppIOTime: 1s372ms - __MAX_OF_AppIOTime: 4s358ms - __MIN_OF_AppIOTime: 1.539ms - FSBytesRead: 15.40 GB - __MAX_OF_FSBytesRead: 127.41 MB - __MIN_OF_FSBytesRead: 64.00 KB - FSIOCounter: 1.637K (1637) - __MAX_OF_FSIOCounter: 12 - __MIN_OF_FSIOCounter: 1 - FSIOTime: 9s357ms - __MAX_OF_FSIOTime: 60s335ms - __MIN_OF_FSIOTime: 1.536ms ``` ##### Solution[​](#solution "Direct link to Solution") You can use one of the following solutions to resolve this issue: * **\[Recommended]** Enable the [data cache](https://docs.starrocks.io/docs/data_source/data_cache.md) feature, which eliminates the impact of slow HDFS DataNodes on queries by automatically caching the data from external storage systems to the BEs or CNs of your StarRocks cluster. * **\[Recommended]** Shorten the timeout duration between the HDFS client and DataNode. This solution is suitable when Data Cache cannot help resolve the slow HDFS DataNode issue. * Enable the [Hedged Read](https://hadoop.apache.org/docs/r2.8.3/hadoop-project-dist/hadoop-common/release/2.4.0/RELEASENOTES.2.4.0.html) feature. With this feature enabled, if a read from a block is slow, StarRocks starts up a new read, which runs in parallel to the original read, to read against a different block replica. Whenever one of the two reads returns, the other read is cancelled. **The Hedged Read feature can help accelerate reads, but it also significantly increases heap memory consumption on Java virtual machines (JVMs). Therefore, if your physical machines provide a small memory capacity, we recommend that you do not enable the Hedged Read feature.** ###### \[Recommended] Data Cache[​](#recommended-data-cache "Direct link to [Recommended] Data Cache") See [Data Cache](https://docs.starrocks.io/docs/data_source/data_cache.md). ###### \[Recommended] Shorten timeout duration between HDFS client and DataNode[​](#recommended-shorten-timeout-duration-between-hdfs-client-and-datanode "Direct link to [Recommended] Shorten timeout duration between HDFS client and DataNode") Configure the `dfs.client.socket-timeout` property in the `hdfs-site.xml` file to shorten the timeout duration between the HDFS client and DataNode. (The default timeout duration is 60s, which is a bit long.) As such, when StarRocks encounters a slow DataNode, the connection request from it can time out within a very short period of time and then be forwarded to another DataNode. The following example sets a 5-second timeout duration: ```xml dfs.client.socket-timeout 5000 ``` ###### Hedged Read[​](#hedged-read "Direct link to Hedged Read") Use the following parameters (supported from v3.0 onwards) in the BE or CN configuration file `be.conf` to enable and configure the Hedged Read feature in your HDFS cluster. | Parameter | Default value | Description | | --------------------------------------------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | hdfs\_client\_enable\_hedged\_read | false | Specifies whether to enable the hedged read feature. | | hdfs\_client\_hedged\_read\_threadpool\_size | 128 | Specifies the size of the Hedged Read thread pool on your HDFS client. The thread pool size limits the number of threads to dedicate to the running of hedged reads in your HDFS client. This parameter is equivalent to the `dfs.client.hedged.read.threadpool.size` parameter in the `hdfs-site.xml` file of your HDFS cluster. | | hdfs\_client\_hedged\_read\_threshold\_millis | 2500 | Specifies the number of milliseconds to wait before starting up a hedged read. For example, you have set this parameter to `30`. In this situation, if a read from a block has not returned within 30 milliseconds, your HDFS client immediately starts up a hedged read against a different block replica. This parameter is equivalent to the `dfs.client.hedged.read.threshold.millis` parameter in the `hdfs-site.xml` file of your HDFS cluster. | If the value of any of the following metrics in your query profiles exceeds `0`, the Hedged Read feature is enabled. | Metric | Description | | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | TotalHedgedReadOps | The number of hedged reads that are started up. | | TotalHedgedReadOpsInCurThread | The number of times that StarRocks has to start up a hedged read in the current thread instead of in a new thread because the Hedged Read thread pool has reached its maximum size specified by the `hdfs_client_hedged_read_threadpool_size` parameter. | | TotalHedgedReadOpsWin | The number of times that a hedged read beats its original read. | #### How do I resolve the error “ERROR 1064 (HY000): Type mismatches on column \[is\_refund], JDBC result type is Integer, please set the type to one of tinyint,smallint,int,bigint” when querying a table in the Hive Catalog?[​](#how-do-i-resolve-the-error-error-1064-hy000-type-mismatches-on-column-is_refund-jdbc-result-type-is-integer-please-set-the-type-to-one-of-tinyintsmallintintbigint-when-querying-a-table-in-the-hive-catalog "Direct link to How do I resolve the error “ERROR 1064 (HY000): Type mismatches on column [is_refund], JDBC result type is Integer, please set the type to one of tinyint,smallint,int,bigint” when querying a table in the Hive Catalog?") This issue is caused by an incorrect JDBC connection configuration. Add the parameter `tinyInt1isBit=false` to your JDBC URI to prevent this issue: ```sql "jdbc_uri" = "jdbc:mysql://xxx:3306?database=yl_spmibill&tinyInt1isBit=false" ``` #### Why can’t I query the latest updated data in the Iceberg Catalog (even after refresh or catalog rebuild), and how should I troubleshoot this?[​](#why-cant-i-query-the-latest-updated-data-in-the-iceberg-catalog-even-after-refresh-or-catalog-rebuild-and-how-should-i-troubleshoot-this "Direct link to Why can’t I query the latest updated data in the Iceberg Catalog (even after refresh or catalog rebuild), and how should I troubleshoot this?") First check whether the issue is caused by Data Cache being enabled. Follow these steps to verify: 1. Compare the scanned data files between StarRocks and Spark: * In StarRocks: `select file_path, spec_id from db.table_name$files;` * In Spark: `select file_path, spec_id from db.table_name.files;` 2. If the results are consistent, continue troubleshooting by disabling Data Cache and querying again to see whether the issue persists. Root cause: To update the Iceberg table data is to overwrite old files, which corrupts Iceberg’s historical data. The correct behavior is to generate new file names when writing updates. StarRocks Data Cache uses the file name, file size, and modification time to determine whether cached data is valid. Since Iceberg does not overwrite files and the modification time is always 0, StarRocks incorrectly treats the files as unchanged and reads from cache, resulting in outdated query results. #### FE constantly crashes after queries against tables in external catalog integrated with JuiceFS. How can I solve this?[​](#fe-constantly-crashes-after-queries-against-tables-in-external-catalog-integrated-with-juicefs-how-can-i-solve-this "Direct link to FE constantly crashes after queries against tables in external catalog integrated with JuiceFS. How can I solve this?") To solve this, restart FE after adding the following configuration items to **fe.conf**: ```properties proc_profile_mem_enable=false proc_profile_cpu_enable=false ``` --- ### Deployment This topic provides answers to some frequently asked questions about deployment. #### How do I bind a fixed IP address with the `priority_networks` parameter in the `fe.conf` file?[​](#how-do-i-bind-a-fixed-ip-address-with-the-priority_networks-parameter-in-the-feconf-file "Direct link to how-do-i-bind-a-fixed-ip-address-with-the-priority_networks-parameter-in-the-feconf-file") ##### Problem description[​](#problem-description "Direct link to Problem description") For example, if you have two IP addresses: 192.168.108.23 and 192.168.108.43. You might provide IP addresses as follows: * If you specify the addresses as 192.168.108.23/24, StarRocks will recognize them as 192.168.108.43. * If you specify the addresses as 192.168.108.23/32, StarRocks will recognize them as 127.0.0.1. ##### Solution[​](#solution "Direct link to Solution") There are the following two ways to solve this problem: * Do not add "32" at the end of an IP address or change "32" to "28". * You can also upgrade to StarRocks 2.1 or later. #### Why does the error "StarRocks BE http service did not start correctly, exiting" occur when I start a backend (BE) after installation?[​](#why-does-the-error-starrocks-be-http-service-did-not-start-correctly-exiting-occur-when-i-start-a-backend-be-after-installation "Direct link to Why does the error \"StarRocks BE http service did not start correctly, exiting\" occur when I start a backend (BE) after installation?") When installing a BE, the system reports a startup error: StarRocks Be http service did not start correctly, exiting. This error occurs because the web services port of the BE is occupied. Try to modify the ports in the `be.conf` file and restart the BE. #### What do I do when the error occurs: ERROR 1064 (HY000): Could not initialize class com.starrocks.rpc.BackendServiceProxy?[​](#what-do-i-do-when-the-error-occurs-error-1064-hy000-could-not-initialize-class-comstarrocksrpcbackendserviceproxy "Direct link to What do I do when the error occurs: ERROR 1064 (HY000): Could not initialize class com.starrocks.rpc.BackendServiceProxy?") This error occurs when you run programs in Java Runtime Environment (JRE). To solve this problem, replace JRE with Java Development Kit (JDK). We recommend that you use Oracle's JDK 17 or later. #### Can FE and BE configuration items be modified and then take effect without restarting the cluster?[​](#can-fe-and-be-configuration-items-be-modified-and-then-take-effect-without-restarting-the-cluster "Direct link to Can FE and BE configuration items be modified and then take effect without restarting the cluster?") Yes. Perform the following steps to complete the modifications for an FE and a BE configuration item: * FE: You can complete the modification for an FE in one of the following ways: * SQL ```plaintext ADMIN SET FRONTEND CONFIG ("key" = "value"); ``` Example: ```plaintext ADMIN SET FRONTEND CONFIG ("enable_statistic_collect" = "false"); ``` * Shell ```plaintext curl --location-trusted -u username:password \ http://: ``` Example: ```plaintext curl --location-trusted -u : \ http://192.168.110.101:8030/api/_set_config?enable_statistic_collect=true ``` * BE: You can complete the modification for a BE in the following way: ```plaintext curl -XPOST -u username:password \ http://:/api/update_config?key=value ``` > Note: Make sure that the user has permission to log in remotely. If not, you can grant the permission to the user in the following way: ```plaintext CREATE USER 'test'@'%' IDENTIFIED BY '123456'; GRANT SELECT ON . TO 'test'@'%'; ``` #### Why does the error "Fe type:unknown ,is ready :false." occur when I start an FE during the cluster restart?[​](#why-does-the-error-fe-type-is-ready--occur-when-i-start-an-fe-during-the-cluster-restart "Direct link to why-does-the-error-fe-type-is-ready--occur-when-i-start-an-fe-during-the-cluster-restart") Check if the leader FE is running. If not, restart the FE nodes in your cluster one by one. #### Why does the error "failed to get service info err." occur when I deploy the cluster?[​](#why-does-the-error-failed-to-get-service-info-err-occur-when-i-deploy-the-cluster "Direct link to Why does the error \"failed to get service info err.\" occur when I deploy the cluster?") Check if OpenSSH Daemon (sshd) is enabled. If not, run the `/etc/init.d/sshd`` status` command to enable it. #### Why does the error "Fail to get master client from `cache. ``host= port=0 code=THRIFT_RPC_ERROR`" occur when I start a BE?[​](#why-does-the-error-fail-to-get-master-client-from-cache-host-port0-codethrift_rpc_error-occur-when-i-start-a-be "Direct link to why-does-the-error-fail-to-get-master-client-from-cache-host-port0-codethrift_rpc_error-occur-when-i-start-a-be") Run the `netstat -anp |grep port` command to check whether the ports in the `be.conf` file are occupied. If so, replace the occupied port with a free port and then restart the BE. #### Why does the FE node log on the diagnostics page of StarRocks Manager display "Search log failed." for a newly deployed FE node that is running properly?[​](#why-does-the-fe-node-log-on-the-diagnostics-page-of-starrocks-manager-display-search-log-failed-for-a-newly-deployed-fe-node-that-is-running-properly "Direct link to Why does the FE node log on the diagnostics page of StarRocks Manager display \"Search log failed.\" for a newly deployed FE node that is running properly?") By default, StarRocks Manager obtains the path configuration of the newly deployed FE within 30 seconds. This error occurs when the FE starts slowly or does not respond within 30 seconds due to other reasons. Check the log of Manager Web via the path: `/starrocks-manager-xxx/center/log/webcenter/log/web/``drms.INFO`(you can customize the path). Then find that whether the message "Failed to update FE configurations" display in the log. If so, restart the corresponding FE to obtain the new path configuration. #### Why does the error "exceeds max permissable delta:5000ms." occur when I start an FE?[​](#why-does-the-error-exceeds-max-permissable-delta5000ms-occur-when-i-start-an-fe "Direct link to Why does the error \"exceeds max permissable delta:5000ms.\" occur when I start an FE?") This error occurs when the time difference between two machines is more than 5s. To solve this problem, align the time of these two machines. #### How do I set the `storage_root_path` parameter if there are multiple disks in a BE for data storage?[​](#how-do-i-set-the-storage_root_path-parameter-if-there-are-multiple-disks-in-a-be-for-data-storage "Direct link to how-do-i-set-the-storage_root_path-parameter-if-there-are-multiple-disks-in-a-be-for-data-storage") Configure the `storage_root_path` parameter in the `be.conf` file and separate values of this parameter with `;`. For example: `storage_root_path=/the/path/to/storage1;/the/path/to/storage2;/the/path/to/storage3;` #### Why does the error "invalid cluster id: 209721925." occur after an FE is added to my cluster?[​](#why-does-the-error-invalid-cluster-id-209721925-occur-after-an-fe-is-added-to-my-cluster "Direct link to Why does the error \"invalid cluster id: 209721925.\" occur after an FE is added to my cluster?") If you do not add the `--helper` option for this FE when starting your cluster for the first time, the metadata between two machines is inconsistent, thus this error occurs. To solve this problem, you need to clear all metadata under the meta directory and then add an FE with the `--helper` option. #### Why Alive is `false` when an FE is running and prints log `transfer: follower`?[​](#why-alive-is-false-when-an-fe-is-running-and-prints-log-transfer-follower "Direct link to why-alive-is-false-when-an-fe-is-running-and-prints-log-transfer-follower") This issue occurs when more than half of memory of Java Virtual Machine (JVM) is used and no checkpoint is marked. In general, a checkpoint will be marked after the system accumulates 50,000 pieces of log. We recommend that you modify the JVM's parameters of each FE and restarting these FEs when they are not heavily loaded. #### Query error: “could not initialize class com.starrocks.rpc.BackendServiceProxy”. How do I resolve this?[​](#query-error-could-not-initialize-class-comstarrocksrpcbackendserviceproxy-how-do-i-resolve-this "Direct link to Query error: “could not initialize class com.starrocks.rpc.BackendServiceProxy”. How do I resolve this?") * Verify that the environment variable `$JAVA_HOME` points to the correct JDK path. * Ensure all nodes use the same JDK version. All nodes must use the identical JDK version. #### What are the MySQL version requirements for installing StarRocks?[​](#what-are-the-mysql-version-requirements-for-installing-starrocks "Direct link to What are the MySQL version requirements for installing StarRocks?") MySQL 5.7 or later is recommended to connect to StarRocks. #### If FE and BE are deployed on the same machine, how can I separate them?[​](#if-fe-and-be-are-deployed-on-the-same-machine-how-can-i-separate-them "Direct link to If FE and BE are deployed on the same machine, how can I separate them?") It is recommended to scale out a BE node first. After the cluster finishes balancing, you can then scale in the original BE node. #### What should I do when FE fails to start with the error “Replica exceeds max permissible delta:5000ms”?[​](#what-should-i-do-when-fe-fails-to-start-with-the-error-replica-exceeds-max-permissible-delta5000ms "Direct link to What should I do when FE fails to start with the error “Replica exceeds max permissible delta:5000ms”?") The clocks between FE nodes are not synchronized. The time difference between FE nodes must be less than 5 seconds. #### If I have five physical machines in a production environment, what is the recommended StarRocks deployment?[​](#if-i-have-five-physical-machines-in-a-production-environment-what-is-the-recommended-starrocks-deployment "Direct link to If I have five physical machines in a production environment, what is the recommended StarRocks deployment?") A recommended deployment is 3 FE nodes and 5 BE nodes. #### Should I use the root user to install StarRocks?[​](#should-i-use-the-root-user-to-install-starrocks "Direct link to Should I use the root user to install StarRocks?") It is not recommended to use the root user because it has excessive privileges. Create a dedicated user for installing StarRocks. #### Can I install the MySQL client on any machine?[​](#can-i-install-the-mysql-client-on-any-machine "Direct link to Can I install the MySQL client on any machine?") Yes. It is simply a client tool and does not need to run on the same machine as StarRocks. Make sure the MySQL client can access the cluster. #### Is there a limit to the number of tablets on a BE node? For example, for a server with 64 GB RAM and 16 cores, what is the reasonable range of tablet count?[​](#is-there-a-limit-to-the-number-of-tablets-on-a-be-node-for-example-for-a-server-with-64-gb-ram-and-16-cores-what-is-the-reasonable-range-of-tablet-count "Direct link to Is there a limit to the number of tablets on a BE node? For example, for a server with 64 GB RAM and 16 cores, what is the reasonable range of tablet count?") There is no strict limit for the tablet number. However, for the tablet size, it is recommended to keep each tablet around 1 GB. Proper partitioning and bucketing strategies will help improve query performance. Tablet size planning is important. #### When starting BE, I see the error “error while loading shared libraries: libjvm.so: cannot open shared object file: No such file or directory”. And after manually creating the directory `lib/starrocks_be`, I get a permission denied error. What should I do?[​](#when-starting-be-i-see-the-error-error-while-loading-shared-libraries-libjvmso-cannot-open-shared-object-file-no-such-file-or-directory-and-after-manually-creating-the-directory-libstarrocks_be-i-get-a-permission-denied-error-what-should-i-do "Direct link to when-starting-be-i-see-the-error-error-while-loading-shared-libraries-libjvmso-cannot-open-shared-object-file-no-such-file-or-directory-and-after-manually-creating-the-directory-libstarrocks_be-i-get-a-permission-denied-error-what-should-i-do") There is an issue with the JDK installation. Please reinstall and properly configure your JDK environment. #### Does StarRocks support running on AMD AVX2? Will mixing Intel and AMD servers cause problems?[​](#does-starrocks-support-running-on-amd-avx2-will-mixing-intel-and-amd-servers-cause-problems "Direct link to Does StarRocks support running on AMD AVX2? Will mixing Intel and AMD servers cause problems?") StarRocks can run on AMD. Mixing Intel and AMD servers is not recommended because hardware heterogeneity may cause issues. It is suggested to fully test StarRocks on AMD before migrating. --- ### query_dump interface This topic describes how to use the query\_dump interface to obtain the details of an SQL query and its related information. If you encounter any of the following issues when executing SQL queries with StarRocks, you can use query\_dump to obtain the SQL details and send the information to StarRocks technical support for troubleshooting: * `Unknown Error` is returned when you execute an SQL query or EXPLAIN. * An error message or exception is returned when you execute an SQL query. * Executing an SQL query is not as efficient as expected, or the execution plan can be optimized (for example, partitions can be pruned or Join order can be adjusted). #### Function overview[​](#function-overview "Direct link to Function overview") The query\_dump interface returns the information that FE relies on when executing the SQL, including: * Query statement * Table creation statement * Session variables * Number of BEs * Statistics information (Min, Max values in a column) * Exception information (abnormal stack) * Explain costs info To ensure data privacy, we desensitize the meta information such as database names, table names, and column names. We also use the desensitized metadata to rewrite the query statements. Meta information desensitization is enabled by default. If an exception occurs during the desensitization process, the original information is used. If desensitization needs to be bypassed, you can add "mock=false" in the HTTP URI. #### Syntax[​](#syntax "Direct link to Syntax") HTTP Post ```shell fe_host:fe_http_port/api/query_dump?db=${database}&mock=${value} post_data=${Query} ``` ```shell wget --user=${username} --password=${password} --post-file ${query_file} "http://${fe_host}:${fe_http_port}/api/query_dump?db=${database}&mock={value}" -O ${dump_file} ``` Parameter description: * query\_file: the file containing the query * dump\_file: the output file * db: the database where the SQL query is executed. The `db` parameter is optional if the query includes `use db`. Otherwise, it must be specified. * mock: whether to enable or disable desensitization #### Examples[​](#examples "Direct link to Examples") ##### Disable desensitization[​](#disable-desensitization "Direct link to Disable desensitization") Command: ```shell wget --user=root --password=123 --post-file query_file "http://127.0.0.1:8030/api/query_dump?db=tpch&mock=false" -O dump_file ``` Return data: Data is returned in JSON format. ```json { "statement": "select\n l_returnflag,\n l_linestatus,\n sum(l_quantity) as sum_qty,\n sum(l_extendedprice) as sum_base_price,\n sum(l_extendedprice * (1 - l_discount)) as sum_disc_price,\n sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) as sum_charge,\n avg(l_quantity) as avg_qty,\n avg(l_extendedprice) as avg_price,\n avg(l_discount) as avg_disc,\n count(*) as count_order\nfrom\n lineitem\nwhere\n l_shipdate <= date '1998-12-01'\ngroup by\n l_returnflag,\n l_linestatus\norder by\n l_returnflag,\n l_linestatus ;\n\n", "table_meta": { "tpch.lineitem": "CREATE TABLE `lineitem` (\n `L_ORDERKEY` int(11) NOT NULL COMMENT \"\",\n `L_PARTKEY` int(11) NOT NULL COMMENT \"\",\n `L_SUPPKEY` int(11) NOT NULL COMMENT \"\",\n `L_LINENUMBER` int(11) NOT NULL COMMENT \"\",\n `L_QUANTITY` double NOT NULL COMMENT \"\",\n `L_EXTENDEDPRICE` double NOT NULL COMMENT \"\",\n `L_DISCOUNT` double NOT NULL COMMENT \"\",\n `L_TAX` double NOT NULL COMMENT \"\",\n `L_RETURNFLAG` char(1) NOT NULL COMMENT \"\",\n `L_LINESTATUS` char(1) NOT NULL COMMENT \"\",\n `L_SHIPDATE` date NOT NULL COMMENT \"\",\n `L_COMMITDATE` date NOT NULL COMMENT \"\",\n `L_RECEIPTDATE` date NOT NULL COMMENT \"\",\n `L_SHIPINSTRUCT` char(25) NOT NULL COMMENT \"\",\n `L_SHIPMODE` char(10) NOT NULL COMMENT \"\",\n `L_COMMENT` varchar(44) NOT NULL COMMENT \"\",\n `PAD` char(1) NOT NULL COMMENT \"\"\n) ENGINE=OLAP \nDUPLICATE KEY(`L_ORDERKEY`)\nCOMMENT \"OLAP\"\nDISTRIBUTED BY HASH(`L_ORDERKEY`) BUCKETS 20 \nPROPERTIES (\n\"replication_num\" = \"1\",\n\"in_memory\" = \"false\",\n\"enable_persistent_index\" = \"true\",\n\"replicated_storage\" = \"true\",\n\"compression\" = \"LZ4\"\n);" }, "table_row_count": { "tpch.lineitem": { "lineitem": 3 } }, "column_statistics": { "tpch.lineitem": { "L_TAX": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE", "L_SHIPDATE": "[1.6094304E9, 1.6094304E9, 0.0, 4.0, 1.0] ESTIMATE", "L_EXTENDEDPRICE": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE", "L_DISCOUNT": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE", "L_RETURNFLAG": "[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE", "L_LINESTATUS": "[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE", "L_QUANTITY": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE" } }, "explain_info": "PLAN FRAGMENT 0(F02)\n Output Exprs:9: L_RETURNFLAG | 10: L_LINESTATUS | 20: sum | 21: sum | 22: sum | 23: sum | 24: avg | 25: avg | 26: avg | 27: count\n Input Partition: UNPARTITIONED\n RESULT SINK\n\n 6:MERGING-EXCHANGE\n distribution type: GATHER\n cardinality: 1\n column statistics: \n * L_RETURNFLAG-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n * L_LINESTATUS-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n\nPLAN FRAGMENT 1(F01)\n\n Input Partition: HASH_PARTITIONED: 9: L_RETURNFLAG, 10: L_LINESTATUS\n OutPut Partition: UNPARTITIONED\n OutPut Exchange Id: 06\n\n 5:SORT\n | order by: [9, VARCHAR, false] ASC, [10, VARCHAR, false] ASC\n | offset: 0\n | cardinality: 1\n | column statistics: \n | * L_RETURNFLAG-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * L_LINESTATUS-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | \n 4:AGGREGATE (merge finalize)\n | aggregate: sum[([20: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], sum[([21: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], sum[([22: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], sum[([23: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], avg[([24: avg, VARBINARY, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], avg[([25: avg, VARBINARY, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], avg[([26: avg, VARBINARY, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], count[([27: count, BIGINT, false]); args: ; result: BIGINT; args nullable: true; result nullable: false]\n | group by: [9: L_RETURNFLAG, VARCHAR, false], [10: L_LINESTATUS, VARCHAR, false]\n | cardinality: 1\n | column statistics: \n | * L_RETURNFLAG-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * L_LINESTATUS-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | \n 3:EXCHANGE\n distribution type: SHUFFLE\n partition exprs: [9: L_RETURNFLAG, VARCHAR, false], [10: L_LINESTATUS, VARCHAR, false]\n cardinality: 1\n\nPLAN FRAGMENT 2(F00)\n\n Input Partition: RANDOM\n OutPut Partition: HASH_PARTITIONED: 9: L_RETURNFLAG, 10: L_LINESTATUS\n OutPut Exchange Id: 03\n\n 2:AGGREGATE (update serialize)\n | STREAMING\n | aggregate: sum[([5: L_QUANTITY, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], sum[([6: L_EXTENDEDPRICE, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], sum[([18: expr, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], sum[([19: expr, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], avg[([5: L_QUANTITY, DOUBLE, false]); args: DOUBLE; result: VARBINARY; args nullable: false; result nullable: true], avg[([6: L_EXTENDEDPRICE, DOUBLE, false]); args: DOUBLE; result: VARBINARY; args nullable: false; result nullable: true], avg[([7: L_DISCOUNT, DOUBLE, false]); args: DOUBLE; result: VARBINARY; args nullable: false; result nullable: true], count[(*); args: ; result: BIGINT; args nullable: false; result nullable: false]\n | group by: [9: L_RETURNFLAG, VARCHAR, false], [10: L_LINESTATUS, VARCHAR, false]\n | cardinality: 1\n | column statistics: \n | * L_RETURNFLAG-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * L_LINESTATUS-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | \n 1:Project\n | output columns:\n | 5 <-> [5: L_QUANTITY, DOUBLE, false]\n | 6 <-> [6: L_EXTENDEDPRICE, DOUBLE, false]\n | 7 <-> [7: L_DISCOUNT, DOUBLE, false]\n | 9 <-> [9: L_RETURNFLAG, CHAR, false]\n | 10 <-> [10: L_LINESTATUS, CHAR, false]\n | 18 <-> [29: multiply, DOUBLE, false]\n | 19 <-> [29: multiply, DOUBLE, false] * 1.0 + [8: L_TAX, DOUBLE, false]\n | common expressions:\n | 28 <-> 1.0 - [7: L_DISCOUNT, DOUBLE, false]\n | 29 <-> [6: L_EXTENDEDPRICE, DOUBLE, false] * [28: subtract, DOUBLE, false]\n | cardinality: 1\n | column statistics: \n | * L_QUANTITY-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * L_EXTENDEDPRICE-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * L_DISCOUNT-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * L_RETURNFLAG-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * L_LINESTATUS-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | \n 0:OlapScanNode\n table: lineitem, rollup: lineitem\n preAggregation: on\n Predicates: [11: L_SHIPDATE, DATE, false] <= '1998-12-01'\n partitionsRatio=1/1, tabletsRatio=20/20\n tabletList=45030,45032,45034,45036,45038,45040,45042,45044,45046,45048 ...\n actualRows=3, avgRowSize=54.0\n cardinality: 1\n column statistics: \n * L_QUANTITY-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * L_EXTENDEDPRICE-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * L_DISCOUNT-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * L_TAX-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * L_RETURNFLAG-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n * L_LINESTATUS-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n * L_SHIPDATE-->[NaN, NaN, 0.0, 4.0, 1.0] ESTIMATE\n * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n", "session_variables": "{\"partial_update_mode\":\"auto\",\"cbo_cte_reuse\":true,\"character_set_connection\":\"utf8\",\"cbo_use_correlated_join_estimate\":true,\"enable_insert_strict\":true,\"enable_connector_adaptive_io_tasks\":true,\"tx_isolation\":\"REPEATABLE-READ\",\"enable_hive_metadata_cache_with_insert\":false,\"cbo_cte_reuse_rate_v2\":1.15,\"character_set_results\":\"utf8\",\"enable_count_star_optimization\":true,\"query_excluding_mv_names\":\"\",\"enable_rewrite_simple_agg_to_meta_scan\":false,\"enable_adaptive_sink_dop\":true,\"consistent_hash_virtual_number\":32,\"enable_profile\":false,\"load_mem_limit\":0,\"sql_safe_updates\":0,\"runtime_filter_early_return_selectivity\":0.05,\"enable_local_shuffle_agg\":true,\"disable_function_fold_constants\":false,\"select_ratio_threshold\":0.15,\"query_delivery_timeout\":300,\"collation_database\":\"utf8_general_ci\",\"spill_mem_table_size\":104857600,\"cbo_use_lock_db\":false,\"new_planner_agg_stage\":0,\"use_compute_nodes\":-1,\"collation_connection\":\"utf8_general_ci\",\"resource_group\":\"\",\"profile_limit_fold\":true,\"spill_operator_max_bytes\":1048576000,\"cbo_max_reorder_node_use_dp\":10,\"enable_hive_column_stats\":true,\"enable_groupby_use_output_alias\":false,\"forward_to_leader\":false,\"count_distinct_column_buckets\":1024,\"query_cache_agg_cardinality_limit\":5000000,\"cboPushDownAggregateMode_v1\":-1,\"window_partition_mode\":1,\"enable_tablet_internal_parallel_v2\":true,\"interpolate_passthrough\":true,\"enable_incremental_mv\":false,\"SQL_AUTO_IS_NULL\":false,\"event_scheduler\":\"OFF\",\"max_pipeline_dop\":64,\"broadcast_right_table_scale_factor\":10,\"materialized_view_rewrite_mode\":\"DEFAULT\",\"enable_simplify_case_when\":true,\"runtime_join_filter_push_down_limit\":1024000,\"big_query_log_cpu_second_threshold\":480,\"div_precision_increment\":4,\"runtime_adaptive_dop_max_block_rows_per_driver_seq\":16384,\"log_rejected_record_num\":0,\"cbo_push_down_distinct_below_window\":true,\"sql_mode_v2\":32,\"prefer_cte_rewrite\":false,\"hdfs_backend_selector_scan_range_shuffle\":false,\"pipeline_profile_level\":1,\"parallel_fragment_exec_instance_num\":1,\"max_scan_key_num\":-1,\"net_read_timeout\":60,\"streaming_preaggregation_mode\":\"auto\",\"hive_partition_stats_sample_size\":3000,\"enable_mv_planner\":false,\"enable_collect_table_level_scan_stats\":true,\"profile_timeout\":2,\"cbo_push_down_aggregate\":\"global\",\"spill_encode_level\":7,\"enable_query_dump\":false,\"global_runtime_filter_build_max_size\":67108864,\"enable_rewrite_sum_by_associative_rule\":true,\"query_cache_hot_partition_num\":3,\"enable_prune_complex_types\":true,\"query_cache_type\":0,\"max_parallel_scan_instance_num\":-1,\"query_cache_entry_max_rows\":409600,\"enable_mv_optimizer_trace_log\":false,\"connector_io_tasks_per_scan_operator\":16,\"enable_materialized_view_union_rewrite\":true,\"sql_quote_show_create\":true,\"scan_or_to_union_threshold\":50000000,\"enable_exchange_pass_through\":true,\"runtime_profile_report_interval\":10,\"query_cache_entry_max_bytes\":4194304,\"enable_exchange_perf\":false,\"workgroup_id\":0,\"enable_rewrite_groupingsets_to_union_all\":false,\"transmission_compression_type\":\"NO_COMPRESSION\",\"interactive_timeout\":3600,\"use_page_cache\":true,\"big_query_log_scan_bytes_threshold\":10737418240,\"collation_server\":\"utf8_general_ci\",\"tablet_internal_parallel_mode\":\"auto\",\"enable_pipeline\":true,\"spill_mode\":\"auto\",\"enable_query_debug_trace\":false,\"enable_show_all_variables\":false,\"full_sort_max_buffered_bytes\":16777216,\"wait_timeout\":28800,\"transmission_encode_level\":7,\"query_including_mv_names\":\"\",\"transaction_isolation\":\"REPEATABLE-READ\",\"enable_global_runtime_filter\":true,\"enable_load_profile\":false,\"enable_plan_validation\":true,\"load_transmission_compression_type\":\"NO_COMPRESSION\",\"cbo_enable_low_cardinality_optimize\":true,\"scan_use_query_mem_ratio\":0.3,\"new_planner_optimize_timeout\":3000,\"enable_outer_join_reorder\":true,\"force_schedule_local\":false,\"hudi_mor_force_jni_reader\":false,\"cbo_enable_greedy_join_reorder\":true,\"range_pruner_max_predicate\":100,\"enable_rbo_table_prune\":false,\"spillable_operator_mask\":-1,\"rpc_http_min_size\":2147482624,\"cbo_debug_alive_backend_number\":0,\"global_runtime_filter_probe_min_size\":102400,\"scan_or_to_union_limit\":4,\"enable_cbo_table_prune\":false,\"enable_parallel_merge\":true,\"nested_mv_rewrite_max_level\":3,\"net_write_timeout\":60,\"cbo_prune_shuffle_column_rate\":0.1,\"spill_revocable_max_bytes\":0,\"hash_join_push_down_right_table\":true,\"pipeline_sink_dop\":0,\"broadcast_row_limit\":15000000,\"enable_populate_block_cache\":true,\"exec_mem_limit\":2147483648,\"enable_sort_aggregate\":false,\"query_cache_force_populate\":false,\"runtime_filter_on_exchange_node\":false,\"disable_join_reorder\":false,\"enable_rule_based_materialized_view_rewrite\":true,\"connector_scan_use_query_mem_ratio\":0.3,\"net_buffer_length\":16384,\"cbo_prune_subfield\":true,\"full_sort_max_buffered_rows\":1024000,\"query_timeout\":300,\"connector_io_tasks_slow_io_latency_ms\":50,\"cbo_max_reorder_node\":50,\"enable_distinct_column_bucketization\":false,\"enable_big_query_log\":true,\"analyze_mv\":\"sample\",\"runtime_filter_scan_wait_time\":20,\"enable_sync_materialized_view_rewrite\":true,\"prefer_compute_node\":false,\"enable_strict_type\":false,\"group_concat_max_len\":65535,\"parse_tokens_limit\":3500000,\"chunk_size\":4096,\"global_runtime_filter_probe_min_selectivity\":0.5,\"query_mem_limit\":0,\"enable_filter_unused_columns_in_scan_stage\":true,\"enable_scan_block_cache\":false,\"enable_materialized_view_single_table_view_delta_rewrite\":false,\"auto_increment_increment\":1,\"sql_dialect\":\"StarRocks\",\"big_query_log_scan_rows_threshold\":1000000000,\"character_set_client\":\"utf8\",\"autocommit\":true,\"enable_column_expr_predicate\":true,\"enable_runtime_adaptive_dop\":false,\"cbo_cte_max_limit\":10,\"storage_engine\":\"olap\",\"enable_optimizer_trace_log\":false,\"spill_operator_min_bytes\":52428800,\"cbo_enable_dp_join_reorder\":true,\"tx_visible_wait_timeout\":10,\"enable_materialized_view_view_delta_rewrite\":true,\"cbo_max_reorder_node_use_exhaustive\":4,\"enable_sql_digest\":false,\"spill_mem_table_num\":2,\"enable_spill\":false,\"pipeline_dop\":0,\"single_node_exec_plan\":false,\"full_sort_late_materialization_v2\":true,\"join_implementation_mode_v2\":\"auto\",\"sql_select_limit\":9223372036854775807,\"enable_materialized_view_rewrite\":true,\"statistic_collect_parallel\":1,\"hdfs_backend_selector_hash_algorithm\":\"consistent\",\"disable_colocate_join\":false,\"max_pushdown_conditions_per_column\":-1,\"default_table_compression\":\"lz4_frame\",\"runtime_adaptive_dop_max_output_amplification_factor\":0,\"innodb_read_only\":true,\"spill_mem_limit_threshold\":0.8,\"cbo_reorder_threshold_use_exhaustive\":6,\"enable_predicate_reorder\":false,\"enable_query_cache\":false,\"max_allowed_packet\":33554432,\"time_zone\":\"Asia/Shanghai\",\"enable_multicolumn_global_runtime_filter\":false,\"character_set_server\":\"utf8\",\"cbo_use_nth_exec_plan\":0,\"io_tasks_per_scan_operator\":4,\"parallel_exchange_instance_num\":-1,\"enable_shared_scan\":false,\"allow_default_partition\":false}", "be_number": 1, "be_core_stat": { "numOfHardwareCoresPerBe": "{\"10004\":104}", "cachedAvgNumOfHardwareCores": 104 }, "exception": [], "version": "main_querydump", "commit_version": "0c4d8c8d3e" } ``` ##### Enable desensitization (default)[​](#enable-desensitization-default "Direct link to Enable desensitization (default)") Command: ```shell wget --user=root --password=123 --post-file query_file "http://127.0.0.1:8030/api/query_dump?db=tpch -O dump_file ``` Return data: The desensitized data is returned in JSON format. ```json { "statement": "SELECT tbl_mock_001.mock_012, tbl_mock_001.mock_007, sum(tbl_mock_001.mock_010) AS mock_019, sum(tbl_mock_001.mock_005) AS mock_020, sum(tbl_mock_001.mock_005 * (1 - tbl_mock_001.mock_004)) AS mock_021, sum((tbl_mock_001.mock_005 * (1 - tbl_mock_001.mock_004)) * (1 + tbl_mock_001.mock_017)) AS mock_022, avg(tbl_mock_001.mock_010) AS mock_023, avg(tbl_mock_001.mock_005) AS mock_024, avg(tbl_mock_001.mock_004) AS mock_025, count(*) AS mock_026\nFROM db_mock_000.tbl_mock_001\nWHERE tbl_mock_001.mock_013 <= '1998-12-01'\nGROUP BY tbl_mock_001.mock_012, tbl_mock_001.mock_007 ORDER BY tbl_mock_001.mock_012 ASC, tbl_mock_001.mock_007 ASC ", "table_meta": { "db_mock_000.tbl_mock_001": "CREATE TABLE db_mock_000.tbl_mock_001 (\nmock_008 int(11) NOT NULL ,\nmock_009 int(11) NOT NULL ,\nmock_016 int(11) NOT NULL ,\nmock_006 int(11) NOT NULL ,\nmock_010 double NOT NULL ,\nmock_005 double NOT NULL ,\nmock_004 double NOT NULL ,\nmock_017 double NOT NULL ,\nmock_012 char(1) NOT NULL ,\nmock_007 char(1) NOT NULL ,\nmock_013 date NOT NULL ,\nmock_003 date NOT NULL ,\nmock_011 date NOT NULL ,\nmock_014 char(25) NOT NULL ,\nmock_015 char(10) NOT NULL ,\nmock_002 varchar(44) NOT NULL ,\nmock_018 char(1) NOT NULL \n) ENGINE= OLAP \nDUPLICATE KEY(mock_008)\nDISTRIBUTED BY HASH(mock_008) BUCKETS 20 \nPROPERTIES (\n\"replication_num\" = \"1\"\n);" }, "table_row_count": { "db_mock_000.tbl_mock_001": { "tbl_mock_001": 3 } }, "column_statistics": { "db_mock_000.tbl_mock_001": { "mock_017": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE", "mock_013": "[1.6094304E9, 1.6094304E9, 0.0, 4.0, 1.0] ESTIMATE", "mock_005": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE", "mock_004": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE", "mock_012": "[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE", "mock_007": "[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE", "mock_010": "[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE" } }, "explain_info": "PLAN FRAGMENT 0(F02)\n Output Exprs:9: mock_012 | 10: mock_007 | 20: sum | 21: sum | 22: sum | 23: sum | 24: avg | 25: avg | 26: avg | 27: count\n Input Partition: UNPARTITIONED\n RESULT SINK\n\n 6:MERGING-EXCHANGE\n distribution type: GATHER\n cardinality: 1\n column statistics: \n * mock_012-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n * mock_007-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n\nPLAN FRAGMENT 1(F01)\n\n Input Partition: HASH_PARTITIONED: 9: mock_012, 10: mock_007\n OutPut Partition: UNPARTITIONED\n OutPut Exchange id: 06\n\n 5:SORT\n | order by: [9, VARCHAR, false] ASC, [10, VARCHAR, false] ASC\n | offset: 0\n | cardinality: 1\n | column statistics: \n | * mock_012-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * mock_007-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | \n 4:AGGREGATE (merge finalize)\n | aggregate: sum[([20: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], sum[([21: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], sum[([22: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], sum[([23: sum, DOUBLE, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], avg[([24: avg, VARBINARY, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], avg[([25: avg, VARBINARY, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], avg[([26: avg, VARBINARY, true]); args: DOUBLE; result: DOUBLE; args nullable: true; result nullable: true], count[([27: count, BIGINT, false]); args: ; result: BIGINT; args nullable: true; result nullable: false]\n | group by: [9: mock_012, VARCHAR, false], [10: mock_007, VARCHAR, false]\n | cardinality: 1\n | column statistics: \n | * mock_012-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * mock_007-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | \n 3:EXCHANGE\n distribution type: SHUFFLE\n partition exprs: [9: mock_012, VARCHAR, false], [10: mock_007, VARCHAR, false]\n cardinality: 1\n\nPLAN FRAGMENT 2(F00)\n\n Input Partition: RANDOM\n OutPut Partition: HASH_PARTITIONED: 9: mock_012, 10: mock_007\n OutPut Exchange id: 03\n\n 2:AGGREGATE (update serialize)\n | STREAMING\n | aggregate: sum[([5: mock_010, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], sum[([6: mock_005, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], sum[([18: expr, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], sum[([19: expr, DOUBLE, false]); args: DOUBLE; result: DOUBLE; args nullable: false; result nullable: true], avg[([5: mock_010, DOUBLE, false]); args: DOUBLE; result: VARBINARY; args nullable: false; result nullable: true], avg[([6: mock_005, DOUBLE, false]); args: DOUBLE; result: VARBINARY; args nullable: false; result nullable: true], avg[([7: mock_004, DOUBLE, false]); args: DOUBLE; result: VARBINARY; args nullable: false; result nullable: true], count[(*); args: ; result: BIGINT; args nullable: false; result nullable: false]\n | group by: [9: mock_012, VARCHAR, false], [10: mock_007, VARCHAR, false]\n | cardinality: 1\n | column statistics: \n | * mock_012-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * mock_007-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * sum-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * avg-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * count-->[0.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | \n 1:Project\n | output columns:\n | 5 <-> [5: mock_010, DOUBLE, false]\n | 6 <-> [6: mock_005, DOUBLE, false]\n | 7 <-> [7: mock_004, DOUBLE, false]\n | 9 <-> [9: mock_012, CHAR, false]\n | 10 <-> [10: mock_007, CHAR, false]\n | 18 <-> [29: multiply, DOUBLE, false]\n | 19 <-> [29: multiply, DOUBLE, false] * 1.0 + [8: mock_017, DOUBLE, false]\n | common expressions:\n | 28 <-> 1.0 - [7: mock_004, DOUBLE, false]\n | 29 <-> [6: mock_005, DOUBLE, false] * [28: subtract, DOUBLE, false]\n | cardinality: 1\n | column statistics: \n | * mock_010-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * mock_005-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * mock_004-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n | * mock_012-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * mock_007-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n | * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n | \n 0:OlapScanNode\n table: mock_001, rollup: mock_001\n preAggregation: on\n Predicates: [11: mock_013, DATE, false] <= '1998-12-01'\n partitionsRatio=1/1, tabletsRatio=20/20\n tabletList=45030,45032,45034,45036,45038,45040,45042,45044,45046,45048 ...\n actualRows=3, avgRowSize=54.0\n cardinality: 1\n column statistics: \n * mock_010-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * mock_005-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * mock_004-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * mock_017-->[1.0, 1.0, 0.0, 8.0, 1.0] ESTIMATE\n * mock_012-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n * mock_007-->[-Infinity, Infinity, 0.0, 1.0, 1.0] ESTIMATE\n * mock_013-->[NaN, NaN, 0.0, 4.0, 1.0] ESTIMATE\n * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n * expr-->[0.0, 0.0, 0.0, 8.0, 1.0] ESTIMATE\n", "session_variables": "{\"partial_update_mode\":\"auto\",\"cbo_cte_reuse\":true,\"character_set_connection\":\"utf8\",\"cbo_use_correlated_join_estimate\":true,\"enable_insert_strict\":true,\"enable_connector_adaptive_io_tasks\":true,\"tx_isolation\":\"REPEATABLE-READ\",\"enable_hive_metadata_cache_with_insert\":false,\"cbo_cte_reuse_rate_v2\":1.15,\"character_set_results\":\"utf8\",\"enable_count_star_optimization\":true,\"query_excluding_mv_names\":\"\",\"enable_rewrite_simple_agg_to_meta_scan\":false,\"enable_adaptive_sink_dop\":true,\"consistent_hash_virtual_number\":32,\"enable_profile\":false,\"load_mem_limit\":0,\"sql_safe_updates\":0,\"runtime_filter_early_return_selectivity\":0.05,\"enable_local_shuffle_agg\":true,\"disable_function_fold_constants\":false,\"select_ratio_threshold\":0.15,\"query_delivery_timeout\":300,\"collation_database\":\"utf8_general_ci\",\"spill_mem_table_size\":104857600,\"cbo_use_lock_db\":false,\"new_planner_agg_stage\":0,\"use_compute_nodes\":-1,\"collation_connection\":\"utf8_general_ci\",\"resource_group\":\"\",\"profile_limit_fold\":true,\"spill_operator_max_bytes\":1048576000,\"cbo_max_reorder_node_use_dp\":10,\"enable_hive_column_stats\":true,\"enable_groupby_use_output_alias\":false,\"forward_to_leader\":false,\"count_distinct_column_buckets\":1024,\"query_cache_agg_cardinality_limit\":5000000,\"cboPushDownAggregateMode_v1\":-1,\"window_partition_mode\":1,\"enable_tablet_internal_parallel_v2\":true,\"interpolate_passthrough\":true,\"enable_incremental_mv\":false,\"SQL_AUTO_IS_NULL\":false,\"event_scheduler\":\"OFF\",\"max_pipeline_dop\":64,\"broadcast_right_table_scale_factor\":10,\"materialized_view_rewrite_mode\":\"DEFAULT\",\"enable_simplify_case_when\":true,\"runtime_join_filter_push_down_limit\":1024000,\"big_query_log_cpu_second_threshold\":480,\"div_precision_increment\":4,\"runtime_adaptive_dop_max_block_rows_per_driver_seq\":16384,\"log_rejected_record_num\":0,\"cbo_push_down_distinct_below_window\":true,\"sql_mode_v2\":32,\"prefer_cte_rewrite\":false,\"hdfs_backend_selector_scan_range_shuffle\":false,\"pipeline_profile_level\":1,\"parallel_fragment_exec_instance_num\":1,\"max_scan_key_num\":-1,\"net_read_timeout\":60,\"streaming_preaggregation_mode\":\"auto\",\"hive_partition_stats_sample_size\":3000,\"enable_mv_planner\":false,\"enable_collect_table_level_scan_stats\":true,\"profile_timeout\":2,\"cbo_push_down_aggregate\":\"global\",\"spill_encode_level\":7,\"enable_query_dump\":false,\"global_runtime_filter_build_max_size\":67108864,\"enable_rewrite_sum_by_associative_rule\":true,\"query_cache_hot_partition_num\":3,\"enable_prune_complex_types\":true,\"query_cache_type\":0,\"max_parallel_scan_instance_num\":-1,\"query_cache_entry_max_rows\":409600,\"enable_mv_optimizer_trace_log\":false,\"connector_io_tasks_per_scan_operator\":16,\"enable_materialized_view_union_rewrite\":true,\"sql_quote_show_create\":true,\"scan_or_to_union_threshold\":50000000,\"enable_exchange_pass_through\":true,\"runtime_profile_report_interval\":10,\"query_cache_entry_max_bytes\":4194304,\"enable_exchange_perf\":false,\"workgroup_id\":0,\"enable_rewrite_groupingsets_to_union_all\":false,\"transmission_compression_type\":\"NO_COMPRESSION\",\"interactive_timeout\":3600,\"use_page_cache\":true,\"big_query_log_scan_bytes_threshold\":10737418240,\"collation_server\":\"utf8_general_ci\",\"tablet_internal_parallel_mode\":\"auto\",\"enable_pipeline\":true,\"spill_mode\":\"auto\",\"enable_query_debug_trace\":false,\"enable_show_all_variables\":false,\"full_sort_max_buffered_bytes\":16777216,\"wait_timeout\":28800,\"transmission_encode_level\":7,\"query_including_mv_names\":\"\",\"transaction_isolation\":\"REPEATABLE-READ\",\"enable_global_runtime_filter\":true,\"enable_load_profile\":false,\"enable_plan_validation\":true,\"load_transmission_compression_type\":\"NO_COMPRESSION\",\"cbo_enable_low_cardinality_optimize\":true,\"scan_use_query_mem_ratio\":0.3,\"new_planner_optimize_timeout\":3000,\"enable_outer_join_reorder\":true,\"force_schedule_local\":false,\"hudi_mor_force_jni_reader\":false,\"cbo_enable_greedy_join_reorder\":true,\"range_pruner_max_predicate\":100,\"enable_rbo_table_prune\":false,\"spillable_operator_mask\":-1,\"rpc_http_min_size\":2147482624,\"cbo_debug_alive_backend_number\":0,\"global_runtime_filter_probe_min_size\":102400,\"scan_or_to_union_limit\":4,\"enable_cbo_table_prune\":false,\"enable_parallel_merge\":true,\"nested_mv_rewrite_max_level\":3,\"net_write_timeout\":60,\"cbo_prune_shuffle_column_rate\":0.1,\"spill_revocable_max_bytes\":0,\"hash_join_push_down_right_table\":true,\"pipeline_sink_dop\":0,\"broadcast_row_limit\":15000000,\"enable_populate_block_cache\":true,\"exec_mem_limit\":2147483648,\"enable_sort_aggregate\":false,\"query_cache_force_populate\":false,\"runtime_filter_on_exchange_node\":false,\"disable_join_reorder\":false,\"enable_rule_based_materialized_view_rewrite\":true,\"connector_scan_use_query_mem_ratio\":0.3,\"net_buffer_length\":16384,\"cbo_prune_subfield\":true,\"full_sort_max_buffered_rows\":1024000,\"query_timeout\":300,\"connector_io_tasks_slow_io_latency_ms\":50,\"cbo_max_reorder_node\":50,\"enable_distinct_column_bucketization\":false,\"enable_big_query_log\":true,\"analyze_mv\":\"sample\",\"runtime_filter_scan_wait_time\":20,\"enable_sync_materialized_view_rewrite\":true,\"prefer_compute_node\":false,\"enable_strict_type\":false,\"group_concat_max_len\":65535,\"parse_tokens_limit\":3500000,\"chunk_size\":4096,\"global_runtime_filter_probe_min_selectivity\":0.5,\"query_mem_limit\":0,\"enable_filter_unused_columns_in_scan_stage\":true,\"enable_scan_block_cache\":false,\"enable_materialized_view_single_table_view_delta_rewrite\":false,\"auto_increment_increment\":1,\"sql_dialect\":\"StarRocks\",\"big_query_log_scan_rows_threshold\":1000000000,\"character_set_client\":\"utf8\",\"autocommit\":true,\"enable_column_expr_predicate\":true,\"enable_runtime_adaptive_dop\":false,\"cbo_cte_max_limit\":10,\"storage_engine\":\"olap\",\"enable_optimizer_trace_log\":false,\"spill_operator_min_bytes\":52428800,\"cbo_enable_dp_join_reorder\":true,\"tx_visible_wait_timeout\":10,\"enable_materialized_view_view_delta_rewrite\":true,\"cbo_max_reorder_node_use_exhaustive\":4,\"enable_sql_digest\":false,\"spill_mem_table_num\":2,\"enable_spill\":false,\"pipeline_dop\":0,\"single_node_exec_plan\":false,\"full_sort_late_materialization_v2\":true,\"join_implementation_mode_v2\":\"auto\",\"sql_select_limit\":9223372036854775807,\"enable_materialized_view_rewrite\":true,\"statistic_collect_parallel\":1,\"hdfs_backend_selector_hash_algorithm\":\"consistent\",\"disable_colocate_join\":false,\"max_pushdown_conditions_per_column\":-1,\"default_table_compression\":\"lz4_frame\",\"runtime_adaptive_dop_max_output_amplification_factor\":0,\"innodb_read_only\":true,\"spill_mem_limit_threshold\":0.8,\"cbo_reorder_threshold_use_exhaustive\":6,\"enable_predicate_reorder\":false,\"enable_query_cache\":false,\"max_allowed_packet\":33554432,\"time_zone\":\"Asia/Shanghai\",\"enable_multicolumn_global_runtime_filter\":false,\"character_set_server\":\"utf8\",\"cbo_use_nth_exec_plan\":0,\"io_tasks_per_scan_operator\":4,\"parallel_exchange_instance_num\":-1,\"enable_shared_scan\":false,\"allow_default_partition\":false}", "be_number": 1, "be_core_stat": { "numOfHardwareCoresPerBe": "{\"10004\":104}", "cachedAvgNumOfHardwareCores": 104 }, "exception": [], "version": "main_querydump", "commit_version": "0c4d8c8d3e" } ``` --- ### Data Export #### Alibaba cloud OSS backup and restore[​](#alibaba-cloud-oss-backup-and-restore "Direct link to Alibaba cloud OSS backup and restore") StarRocks supports backing up data to alicloud OSS / AWS S3 (or object storage compatible with S3 protocol). Suppose there are two StarRocks clusters, namely DB1 cluster and DB2 cluster. We need to back up the data in DB1 to alicloud OSS and then restore it to DB2 when necessary. The general process of backup and recovery is as follows: ##### Create a cloud repository[​](#create-a-cloud-repository "Direct link to Create a cloud repository") Execute SQL in DB1 and DB2 respectively: ```sql CREATE REPOSITORY `repository name` WITH BROKER `broker_name` ON LOCATION "oss://bucket name/path" PROPERTIES ( "fs.oss.accessKeyId" = "xxx", "fs.oss.accessKeySecret" = "yyy", "fs.oss.endpoint" = "oss-cn-beijing.aliyuncs.com" ); ``` a. Both DB1 and DB2 need to be created, and the created REPOSITORY name should be the same. View the repository: ```sql SHOW REPOSITORIES; ``` b. broker\_ name needs to fill in the broker name in a cluster. View BrokerName: ```sql SHOW BROKER; ``` c. The path after fs.oss.endpoint does not need to have a bucket name. ##### Backup data table[​](#backup-data-table "Direct link to Backup data table") BACKUP the tables to be backed up to the cloud repository in DB1. Execute SQL in DB1: ```sql BACKUP SNAPSHOT [db_name].{snapshot_name} TO `repository_name` ON ( `table_name` [PARTITION (`p1`, ...)], ... ) PROPERTIES ("key"="value", ...); ``` ```plain PROPERTIES currently supports the following properties: "type" = "full": indicates that this is a full update (default). "timeout" = "3600": task timeout. The default is one day. The unit is seconds. ``` StarRocks does not support full database backup at present. We need to specify the tables or partitions to be backed up ON (...), and these tables or partitions will be backed up in parallel. View the backup tasks in progress (note that only one backup task can be performed at the same time): ```sql SHOW BACKUP FROM db_name; ``` After the backup is completed, you can check whether the backup data in the OSS already exists (unnecessary backups need to be deleted in the OSS): ```sql SHOW SNAPSHOT ON OSS repository name; ``` ##### Data restore[​](#data-restore "Direct link to Data restore") For data restore in DB2, there is no need to create a table structure to be restored in DB2. It will be created automatically during the Restore operation. Perform restore SQL: ```sql RESTORE SNAPSHOT [db_name].{snapshot_name} FROMrepository_name`` ON ( 'table_name' [PARTITION ('p1', ...)] [AS 'tbl_alias'], ... ) PROPERTIES ("key"="value", ...); ``` View the restore progress: ```sql SHOW RESTORE; ``` --- ### Troubleshooting FE Heap Memory This topic describes how to troubleshoot FE heap memory issues in StarRocks. #### Sudden surge of heap memory allocation[​](#sudden-surge-of-heap-memory-allocation "Direct link to Sudden surge of heap memory allocation") You can identify issues that cause sudden heap memory allocation surge by checking memory profiles (or memory allocate profiles) in StarRocks. From v3.3.6 onwards, StarRocks supports printing memory profiles in `.tgz`-formatted compressed HTML files under directory **fe/log/proc\_profile**. ![FE Memory FAQ - 1](/assets/images/fe_mem_faq_1-cc79083e05ac1d8a54b98b4853069c38.png) You can find the corresponding file based on the time when the memory issue occurred, decompress it, and open it in your browser. In the file, you will see the flame graph of the stack memory allocation. The wider a frame, the more memory resources the stack had been allocated to. For example, in the flame graph below, the width of frame `BDBEnvironment.getDatabaseNamesWithPrefix` exceeds 50% of the graph, indicating that more than half of the memory resources were allocated to this function. ![FE Memory FAQ - 2](/assets/images/fe_mem_faq_2-e49cdbb24bed89137b9c2b628ca00b3a.png) If no memory profile was printed during the surge, you can manually disable the CPU profile, and set the profile printing interval to 5 minutes by setting the following items in the FE configuration file **fe.conf**, and restarting FE. ```properties proc_profile_cpu_enable = false proc_profile_collect_interval_s = 300 ``` ##### Workaround for versions earlier than v3.3.6[​](#workaround-for-versions-earlier-than-v336 "Direct link to Workaround for versions earlier than v3.3.6") For versions earlier than v3.3.6, you can print memory profiles regularly via a script. Run the following Shell script under the **fe** directory: ```bash #!/bin/bash mkdir -p mem_alloc_log while true do current_time=$(date +'%Y-%m-%d-%H-%M-%S') file_name="mem_alloc_log/alloc-profile-${current_time}.html" ./bin/profiler.sh -e alloc --alloc 2m -d 300 -f "$file_name" `cat bin/fe.pid` done ``` #### Slow increase of heap memory usage[​](#slow-increase-of-heap-memory-usage "Direct link to Slow increase of heap memory usage") From v3.3.7 onwards, StarRocks supports printing Memory Usage Tracker logs for tracking memory leak issues. Memory Usage Tracker will regularly record the memory usage of each module. Example: ```plain 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Agent - AgentTaskTracker estimated 0B of memory. Contains AgentTask with 0 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Backup - BackupHandler estimated 0B of memory. Contains BackupOrRestoreJob with 0 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Compaction - CompactionMgr estimated 0B of memory. Contains PartitionStats with 0 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Coordinator - QeProcessorImpl estimated 0B of memory. Contains QueryCoordinator with 0 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Delete - DeleteMgrEPack estimated 0B of memory. Contains DeleteInfo with 0 object(s). DeleteJob with 0 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Dict - CacheDictManager estimated 0B of memory. Contains ColumnDict with 0 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Export - ExportMgr estimated 0B of memory. Contains ExportJob with 0 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Load - InsertOverwriteJobMgr estimated 0B of memory. Contains insertOverwriteJobs with 0 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Load - LoadMgrEPack estimated 245.7KB of memory. Contains LoadJob with 1165 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Load - RoutineLoadMgrEPack estimated 0B of memory. Contains RoutineLoad with 0 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Load - StreamLoadMgrEPack estimated 0B of memory. Contains StreamLoad with 0 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module LocalMetastore - LocalMetastore estimated 7KB of memory. Contains Partition with 45 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module MV - MVTimelinessMgr estimated 0B of memory. Contains mvTimelinessMap with 0 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Profile - ProfileManager estimated 96B of memory. Contains QueryProfile with 4 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Query - QueryTracker estimated 0B of memory. Contains QueryDetail with 0 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Report - ReportHandler estimated 0B of memory. Contains PendingTask with 0 object(s). ReportQueue with 0 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Statistics - CachedStatisticStorage estimated 0B of memory. Contains TableStats with 0 object(s). ColumnStats with 0 object(s). PartitionStats with 0 object(s). HistogramStats with 0 object(s). ConnectorTableStats with 0 object(s). ConnectorHistogramStats with 0 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module TabletInvertedIndex - TabletInvertedIndex estimated 49.8KB of memory. Contains TabletMeta with 208 object(s). TabletCount with 208 object(s). ReplicateCount with 216 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Task - TaskManager estimated 0B of memory. Contains Task with 0 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Task - TaskRunManager estimated 0B of memory. Contains PendingTaskRun with 0 object(s). RunningTaskRun with 0 object(s). HistoryTaskRun with 0 object(s). 2025-02-05 19:35:23.287+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():164] (0ms) Module Transaction - GlobalTransactionMgr estimated 311.4KB of memory. Contains Txn with 1329 object(s). TxnCallbackCount with 0 object(s). 2025-02-05 19:35:23.288+08:00 INFO (MemoryUsageTracker|77) [MemoryUsageTracker.trackMemory():111] total tracked memory: 614.2KB, jvm: Process used: 672MB, heap used: 299.9MB, non heap used: 170MB, direct buffer used: 4.2MB ``` --- ### Broker Load #### 1. Does Broker Load support re-running load jobs that have been run successfully and are in the FINISHED state?[​](#1-does-broker-load-support-re-running-load-jobs-that-have-been-run-successfully-and-are-in-the-finished-state "Direct link to 1. Does Broker Load support re-running load jobs that have been run successfully and are in the FINISHED state?") Broker Load does not support re-running load jobs that have been run successfully and are in the FINISHED state. Also, to prevent data loss and duplication, Broker Load does not allow reusing the labels of successfully run load jobs. You can use [SHOW LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SHOW_LOAD.md) to view the history of load jobs and find the load job that you want to re-run. Then, you can copy the information of that load job and use the job information, except the label, to create another load job. #### 2. When I load data from HDFS by using Broker Load, what do I do if the date and time values loaded into the destination StarRocks table are 8 hours later than the date and time values from the source data file?[​](#2-when-i-load-data-from-hdfs-by-using-broker-load-what-do-i-do-if-the-date-and-time-values-loaded-into-the-destination-starrocks-table-are-8-hours-later-than-the-date-and-time-values-from-the-source-data-file "Direct link to 2. When I load data from HDFS by using Broker Load, what do I do if the date and time values loaded into the destination StarRocks table are 8 hours later than the date and time values from the source data file?") Both the destination StarRocks table and the Broker Load job are compiled at creation to use a China Standard Time (CST) time zone (specified by using the `timezone` parameter). However, the server is set to run based on a Coordinated Universal Time (UTC) time zone. As a result, 8 extra hours are added to the date and time values from the source data file during data loading. To prevent this issue, do not specify the `timezone` parameter when you create the destination StarRocks table. #### 3. When I load ORC-formatted data by using Broker Load, what do I do if the `ErrorMsg: type:ETL_RUN_FAIL; msg:Cannot cast '' from VARCHAR to ARRAY` error occurs?[​](#3-when-i-load-orc-formatted-data-by-using-broker-load-what-do-i-do-if-the-errormsg-typeetl_run_fail-msgcannot-cast-slot-6-from-varchar-to-arrayvarchar30-error-occurs "Direct link to 3-when-i-load-orc-formatted-data-by-using-broker-load-what-do-i-do-if-the-errormsg-typeetl_run_fail-msgcannot-cast-slot-6-from-varchar-to-arrayvarchar30-error-occurs") The source data file has different column names than the destination StarRocks table. In this situation, you must use the `SET` clause in the load statement to specify the column mapping between the file and the table. When executing the `SET` clause, StarRocks needs to perform a type inference, but it fails in invoking the [cast](https://docs.starrocks.io/docs/sql-reference/sql-functions/cast.md) function to transform the source data to the destination data types. To resolve this issue, make sure that the source data file has the same column names as the destination StarRocks table. As such, the `SET` clause is not needed and therefore StarRocks does not need to invoke the cast function to perform data type conversions. Then the Broker Load job can be run successfully. #### 4. The Broker Load job does not report errors, but why am I unable to query the loaded data?[​](#4-the-broker-load-job-does-not-report-errors-but-why-am-i-unable-to-query-the-loaded-data "Direct link to 4. The Broker Load job does not report errors, but why am I unable to query the loaded data?") Broker Load is an asynchronous loading method. The load job may still fail even if the load statement does not return errors. After you run a Broker Load job, you can use [SHOW LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SHOW_LOAD.md) to view the result and `errmsg` of the load job. Then, you can modify the job configuration and retry. #### 5. What do I do if the "failed to send batch" or "TabletWriter add batch with unknown id" error occurs?[​](#5-what-do-i-do-if-the-failed-to-send-batch-or-tabletwriter-add-batch-with-unknown-id-error-occurs "Direct link to 5. What do I do if the \"failed to send batch\" or \"TabletWriter add batch with unknown id\" error occurs?") The amount of time taken to write the data exceeds the upper limit, causing a timeout error. To resolve this issue, modify the settings of the [session variable](https://docs.starrocks.io/docs/sql-reference/System_variable.md) `query_timeout` and the [BE configuration item](https://docs.starrocks.io/docs/administration/management/BE_configuration.md#configure-be-static-parameters) `streaming_load_rpc_max_alive_time_sec` based on your business requirements. #### 6. What do I do if the "LOAD-RUN-FAIL; msg:OrcScannerAdapter::init\_include\_columns. col name = xxx not found" error occurs?[​](#6-what-do-i-do-if-the-load-run-fail-msgorcscanneradapterinit_include_columns-col-name--xxx-not-found-error-occurs "Direct link to 6. What do I do if the \"LOAD-RUN-FAIL; msg:OrcScannerAdapter::init_include_columns. col name = xxx not found\" error occurs?") If you are loading Parquet- or ORC-formatted data, check whether the column names held in the first row of the source data file are the same as the column names of the destination StarRocks table. ```sql (tmp_c1,tmp_c2) SET ( id=tmp_c2, name=tmp_c1 ) ``` The preceding example maps the `tmp_c1` and `tmp_c2` columns of the source data file onto the `name` and `id` columns of the destination StarRocks table, respectively. If you do not specify the `SET` clause, the column names specified in the `column_list` parameter are used to declare the column mapping. For more information, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). > **NOTICE** > > If the source data file is an ORC-formatted file generated by Apache Hive™ and the first row of the file holds `(_col0, _col1, _col2, ...)`, the "Invalid Column Name" error may occur. If this error occurs, you need to use the `SET` clause to specify the column mapping. #### 7. How do I handle errors such as the error that causes the Broker Load job to run for an excessively long period of time?[​](#7-how-do-i-handle-errors-such-as-the-error-that-causes-the-broker-load-job-to-run-for-an-excessively-long-period-of-time "Direct link to 7. How do I handle errors such as the error that causes the Broker Load job to run for an excessively long period of time?") View the FE log file **fe.log** and search for the ID of the load job based on the job label. Then, view the BE log file **be.INFO** and retrieve the log records of the load job based on the job ID to locate the root cause of the error. #### 8. How do I configure an Apache HDFS cluster that runs in HA mode?[​](#8-how-do-i-configure-an-apache-hdfs-cluster-that-runs-in-ha-mode "Direct link to 8. How do I configure an Apache HDFS cluster that runs in HA mode?") If an HDFS cluster runs in high availability (HA) mode, configure it as follows: * `dfs.nameservices`: the name of the HDFS cluster, for example, `"dfs.nameservices" = "my_ha"`. * `dfs.ha.namenodes.xxx`: the name of the NameNode in the HDFS cluster. If you specify multiple NameNode names, separate them with commas (`,`). `xxx` is the HDFS cluster name that you have specified in `dfs.nameservices`, for example, `"dfs.ha.namenodes.my_ha" = "my_nn"`. * `dfs.namenode.rpc-address.xxx.nn`: the RPC address of the NameNode in the HDFS cluster. `nn` is the NameNode name that you have specified in `dfs.ha.namenodes.xxx`, for example, `"dfs.namenode.rpc-address.my_ha.my_nn" = "host:port"`. * `dfs.client.failover.proxy.provider`: the provider of the NameNode to which the client will connect. Default value: `org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider`. Example: ```sql ( "dfs.nameservices" = "my-ha", "dfs.ha.namenodes.my-ha" = "my-namenode1, my-namenode2", "dfs.namenode.rpc-address.my-ha.my-namenode1" = "nn1-host:rpc_port", "dfs.namenode.rpc-address.my-ha.my-namenode2" = "nn2-host:rpc_port", "dfs.client.failover.proxy.provider" = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider" ) ``` The HA mode can be used with simple authentication or Kerberos authentication. For example, to use simple authentication to access an HDFS cluster that runs in HA mode, you need to specify the following configurations: ```sql ( "username"="user", "password"="passwd", "dfs.nameservices" = "my-ha", "dfs.ha.namenodes.my-ha" = "my_namenode1, my_namenode2", "dfs.namenode.rpc-address.my-ha.my-namenode1" = "nn1-host:rpc_port", "dfs.namenode.rpc-address.my-ha.my-namenode2" = "nn2-host:rpc_port", "dfs.client.failover.proxy.provider" = "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider" ) ``` You can add the configurations of the HDFS cluster to the **hdfs-site.xml** file. This way, you only need to specify the file path and authentication information when you use brokers to load data from the HDFS cluster. #### 9. How do I configure Hadoop ViewFS Federation?[​](#9-how-do-i-configure-hadoop-viewfs-federation "Direct link to 9. How do I configure Hadoop ViewFS Federation?") Copy the ViewFs-related configuration files `core-site.xml` and `hdfs-site.xml` to the **broker/conf** directory. If you have a custom file system, you also need to copy the file system-related **.jar** files to the **broker/lib** directory. #### 10. When I access an HDFS cluster that requires Kerberos authentication, what do I do if the "Can't get Kerberos realm" error occurs?[​](#10-when-i-access-an-hdfs-cluster-that-requires-kerberos-authentication-what-do-i-do-if-the-cant-get-kerberos-realm-error-occurs "Direct link to 10. When I access an HDFS cluster that requires Kerberos authentication, what do I do if the \"Can't get Kerberos realm\" error occurs?") Check that the **/etc/krb5.conf** file is configured on all hosts on which brokers are deployed. If the error persists, add `-Djava.security.krb5.conf:/etc/krb5.conf` to the end of the `JAVA_OPTS` variable in the broker startup script. #### 11. Will Broker Load fail if the source Hive table is empty?[​](#11-will-broker-load-fail-if-the-source-hive-table-is-empty "Direct link to 11. Will Broker Load fail if the source Hive table is empty?") By default, an empty transaction returns error "all partitions have no load data". You can set the FE configuration `empty_load_as_error` to `false` to allow empty transactions to return success. --- ### DataX #### Why datax writer does not have writemode parameter? Why must the tables be created in the update mode?[​](#why-datax-writer-does-not-have-writemode-parameter-why-must-the-tables-be-created-in-the-update-mode "Direct link to Why datax writer does not have writemode parameter? Why must the tables be created in the update mode?") Currently, only insert is supported. #### How to process keywords when I synchronize data by using DataX Writer?[​](#how-to-process-keywords-when-i-synchronize-data-by-using-datax-writer "Direct link to How to process keywords when I synchronize data by using DataX Writer?") Enclose a keyword in back quotes (\`\`). If you want to download the newest version of DataX, see [DataX Writer.](https://docs.starrocks.io/docs/integrations/loading_tools/DataX-starrocks-writer.md) --- ### Flink Connector #### flink-connector-jdbc\_2.11sink is 8 hours late in StarRocks[​](#flink-connector-jdbc_211sink-is-8-hours-late-in-starrocks "Direct link to flink-connector-jdbc_2.11sink is 8 hours late in StarRocks") **Issue description:** The time generated by localtimestap function is normal in Flink. But it became 8 hours late when sunk to StarRocks. Flink server and StarRocks server are located in the same timezone, namely Asia/Shanghai UTC/GMT+08:00. Flink version is 1.12. Driver: flink-connector-jdbc\_2.11. Can I ask how to resolve this issue? **Solution:** Please try configure the time parameter 'server-time-zone' = 'Asia/Shanghai' in Flink sink table. You may also add \&serverTimezone=Asia/Shanghai in jdbc url. An example is shown below: ```sql CREATE TABLE sk ( sid int, local_dtm TIMESTAMP, curr_dtm TIMESTAMP ) WITH ( 'connector' = 'jdbc', 'url' = 'jdbc:mysql://192.168.110.66:9030/sys_device?characterEncoding=utf-8&serverTimezone=Asia/Shanghai', 'table-name' = 'sink', 'driver' = 'com.mysql.jdbc.Driver', 'username' = 'sr', 'password' = 'sr123', 'server-time-zone' = 'Asia/Shanghai' ); ``` #### In flink import, only the kafka clusters deployed in StarRocks clusters can be imported[​](#in-flink-import-only-the-kafka-clusters-deployed-in-starrocks-clusters-can-be-imported "Direct link to In flink import, only the kafka clusters deployed in StarRocks clusters can be imported") **Issue description:** ```sql failed to query wartermark offset, err: Local: Bad message format ``` **Solution:** Kafka communication needs the hostname. Users need to configure the host name resolution /etc/hosts in StarRocks cluster nodes. #### Can StarRocks export 'create table statements' in batches?[​](#can-starrocks-export-create-table-statements-in-batches "Direct link to Can StarRocks export 'create table statements' in batches?") **Solution:** You can use StarRocks Tools to export the statements. #### Memory requested by BE is not released back to to the operation system[​](#memory-requested-by-be-is-not-released-back-to-to-the-operation-system "Direct link to Memory requested by BE is not released back to to the operation system") This is a normal phenomenon, as large blocks of memory allocated to the database from the operating system are reserved during allocation and deferred during release in order to reuse the memory and make memory allocation more convenient. It is recommended that users validate the test environment by monitoring memory usage over a longer period of time to see if the memory can be released. #### Flink connector does not work after being downloaded[​](#flink-connector-does-not-work-after-being-downloaded "Direct link to Flink connector does not work after being downloaded") **Issue description:** This package needs to be obtained through Aliyun mirror address. **Solution:** Please make sure that the mirror part of `/etc/maven/settings.xml` is all configured to be obtained through Aliyun mirror address. If it is, change it to the following: aliyunmaven centralaliyun public repohttps: //maven.aliyun.com/repository/public #### The meaning of parameter sink.buffer-flush.interval-ms in Flink-connector-StarRocks[​](#the-meaning-of-parameter-sinkbuffer-flushinterval-ms-in-flink-connector-starrocks "Direct link to The meaning of parameter sink.buffer-flush.interval-ms in Flink-connector-StarRocks") **Issue description:** ```plain +----------------------+--------------------------------------------------------------+ | Option | Required | Default | Type | Description | +-------------------------------------------------------------------------------------+ | sink.buffer-flush. | NO | 300000 | String | the flushing time interval, | | interval-ms | | | | range: [1000ms, 3600000ms] | +----------------------+--------------------------------------------------------------+ ``` If this parameter is set as 15s with checkpoint interval being equal to 5 mins, does this value still take effect? **Solution:** Whichever of the three thresholds is reached first, that one will take effect first. This is not affected by the checkpoint interval value which only works for exactly once. Interval-ms is used by at\_least\_once. #### Why does Partial Updates with Flink Connector fail with “NULL value in non-nullable column”?[​](#why-does-partial-updates-with-flink-connector-fail-with-null-value-in-non-nullable-column "Direct link to Why does Partial Updates with Flink Connector fail with “NULL value in non-nullable column”?") Configure the following properties: ```sql sink.properties.partial_update=true sink.properties.columns= ``` #### How to handle the Flink import error with JSON data “The size of this batch exceed the max size \[104857600]”?[​](#how-to-handle-the-flink-import-error-with-json-data-the-size-of-this-batch-exceed-the-max-size-104857600 "Direct link to How to handle the Flink import error with JSON data “The size of this batch exceed the max size [104857600]”?") Reduce batch frequency, or set `sink.properties.ignore_json_size` to `true` (which may cause higher memory usage). #### How to handle if a bigint unsigned field in Flink CDC turned into a string and changed its values?[​](#how-to-handle-if-a-bigint-unsigned-field-in-flink-cdc-turned-into-a-string-and-changed-its-values "Direct link to How to handle if a bigint unsigned field in Flink CDC turned into a string and changed its values?") Add the following configuration: ```sql 'debezium.bigint.unsigned.handling.mode' = 'precise' ``` #### Why do I get “None of hosts in load\_url could be connected” when using Flink connector to import data?[​](#why-do-i-get-none-of-hosts-in-load_url-could-be-connected-when-using-flink-connector-to-import-data "Direct link to Why do I get “None of hosts in load_url could be connected” when using Flink connector to import data?") The `load_url` is unreachable or experiencing timeout. Increase the value of the property `sink.connect.timeout-ms` (Range: \[100, 60000]). --- ### Insert Into #### When performing data insert, each insert in SQL takes up 50 to 100ms. Is there any way to increase efficiency?[​](#when-performing-data-insert-each-insert-in-sql-takes-up-50-to-100ms-is-there-any-way-to-increase-efficiency "Direct link to When performing data insert, each insert in SQL takes up 50 to 100ms. Is there any way to increase efficiency?") It is not recommended to insert data piece by piece to OLAP. It is usually inserted in batches. Both methods take up the same amount of time. #### 'Insert into select' task reports error: index channel has intolerable failure[​](#insert-into-select-task-reports-error-index-channel-has-intolerable-failure "Direct link to 'Insert into select' task reports error: index channel has intolerable failure") You can solve this problem by changing the timeout duration for the Stream Load RPC. Change the following item in **be.conf** and restart the machines to allow the change to take effect: `streaming_load_rpc_max_alive_time_sec`: The RPC timeout for Stream Load. Unit: Seconds. Default: `1200`. Or you can set the INSERT timeout using the following variable: `insert_timeout`: The timeout duration for INSERT statements. Its unit is seconds, and the default value is `14400`. #### The error "execute timeout" occurs when I run the INSERT INTO SELECT command to load a large volume of data[​](#the-error-execute-timeout-occurs-when-i-run-the-insert-into-select-command-to-load-a-large-volume-of-data "Direct link to The error \"execute timeout\" occurs when I run the INSERT INTO SELECT command to load a large volume of data") By default, the INSERT timeout duration is 14400s. You can set the variable `insert_timeout` to extend this duration. The unit is second. #### Why does INSERT INTO SELECT return “Reach limit of connections”?[​](#why-does-insert-into-select-return-reach-limit-of-connections "Direct link to Why does INSERT INTO SELECT return “Reach limit of connections”?") It is because the user connection limit is reached. Increase the value of the user property `max_user_connections`. --- ### Data loading common questions #### 1. What do I do if the "close index channel failed" or "too many tablet versions" error occurs?[​](#1-what-do-i-do-if-the-close-index-channel-failed-or-too-many-tablet-versions-error-occurs "Direct link to 1. What do I do if the \"close index channel failed\" or \"too many tablet versions\" error occurs?") You were running load jobs too frequently, and the data was not compacted in a timely manner. As a result, the number of data versions generated during loading exceeds the maximum number (which defaults to 1000) of data versions that are allowed. Use one of the following methods to resolve this issue: * Increase the amount of data loaded in each individual job, thereby reducing loading frequency. * Modify the some configuration items in the BE configuration file **be.conf** of each BE to accelerate compactions: * For Duplicate Key tables, Aggregate tables, and Unique Key tables, you can appropriately increase the values of `cumulative_compaction_num_threads_per_disk`, `base_compaction_num_threads_per_disk`, and `cumulative_compaction_check_interval_seconds`. Example: ```plain cumulative_compaction_num_threads_per_disk = 4 base_compaction_num_threads_per_disk = 2 cumulative_compaction_check_interval_seconds = 2 ``` * For Primary Key tables, you can appropriately increase the value of `update_compaction_num_threads_per_disk` and decrease the value of `update_compaction_per_tablet_min_interval_seconds`. After you modify the settings of the preceding configuration items, you must observe the memory and I/O to ensure that they are normal. #### 2. What do I do if the "Label Already Exists" error occurs?[​](#2-what-do-i-do-if-the-label-already-exists-error-occurs "Direct link to 2. What do I do if the \"Label Already Exists\" error occurs?") This error occurs because the load job has the same label as another load job, which has been successfully run or is being run, within the same StarRocks database. Stream Load jobs are submitted according to HTTP. In general, request retry logic is embedded in HTTP clients of all programmatic languages. When the StarRocks cluster receives a load job request from an HTTP client, it immediately starts to process the request, but it does not return the job result to the HTTP client in a timely manner. As a result, the HTTP client sends the same load job request again. However, the StarRocks cluster is already processing the first request and therefore returns the `Label Already Exists` error for the second request. Do as follows to check that load jobs submitted by using different loading methods do not have the same label and are not repeatedly submitted: * View the FE log and check whether the label of the failed load job is recorded twice. If the label is recorded twice, the client has submitted the load job request twice. > **NOTE** > > The StarRocks cluster does not distinguish between the labels of load jobs based on loading methods. Therefore, load jobs submitted by using different loading methods may have the same label. * Run SHOW LOAD WHERE LABEL = "xxx" to check for load jobs that have the same label and are in the **FINISHED** state. > **NOTE** > > `xxx` is the label that you want to check. Before you submit a load job, we recommend that you calculate the approximate amount of time required to load the data and then adjust the client-side request timeout period accordingly. This way, you can prevent the client from submitting the load job request multiple times. #### 3. What do I do if the "ETL\_QUALITY\_UNSATISFIED; msg:quality not good enough to cancel" error occurs?[​](#3-what-do-i-do-if-the-etl_quality_unsatisfied-msg-not-good-enough-to-cancel-error-occurs "Direct link to 3-what-do-i-do-if-the-etl_quality_unsatisfied-msg-not-good-enough-to-cancel-error-occurs") Execute [SHOW LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SHOW_LOAD.md), and use the error URL in the returned execution result to view the error details. Common data quality errors are as follows: * "convert csv string to INT failed." Strings from a source column failed to be transformed into the data type of the matching destination column. For example, `abc` failed to be transformed into a numeric value. * "the length of input is too long than schema." Values from a source column are in lengths that are not supported by the matching destination column. For example, the source column values of CHAR data type exceed the destination column's maximum length specified at table creation, or the source column values of INT data type exceed 4 bytes. * "actual column number is less than schema column number." After a source row is parsed based on the specified column separator, the number of columns obtained is smaller than the number of columns in the destination table. A possible reason is that the column separator specified in the load command or statement differs from the column separator that is actually used in that row. * "actual column number is more than schema column number." After a source row is parsed based on the specified column separator, the number of columns obtained is greater than the number of columns in the destination table. A possible reason is that the column separator specified in the load command or statement differs from the column separator that is actually used in that row. * "the frac part length longer than schema scale." The decimal parts of values from a DECIMAL-type source column exceed the specified length. * "the int part length longer than schema precision." The integer parts of values from a DECIMAL-type source column exceed the specified length. * "there is no corresponding partition for this key." The value in the partition column for a source row is not within the partition range. #### 4. What do I do if RPC times out?[​](#4-what-do-i-do-if-rpc-times-out "Direct link to 4. What do I do if RPC times out?") Check the setting of the `write_buffer_size` configuration item in the BE configuration file **be.conf** of each BE. This configuration item is used to control the maximum size per memory block on the BE. The default maximum size is 100 MB. If the maximum size is exceedingly large, Remote Procedure Call (RPC) may time out. To resolve this issue, adjust the settings of the `write_buffer_size` and `tablet_writer_rpc_timeout_sec` configuration items in the BE configuration file. For more information, see [BE configurations](https://docs.starrocks.io/docs/loading/loading_introduction/loading_considerations.md#be-configurations). #### 5. What do I do if the "Value count does not match column count" error occurs?[​](#5-what-do-i-do-if-the-value-count-does-not-match-column-count-error-occurs "Direct link to 5. What do I do if the \"Value count does not match column count\" error occurs?") After my load job failed, I used the error URL returned in the job result to retrieve the error details and found the "Value count does not match column count" error, which indicates a mismatch between the number of columns in the source data file and the number of columns in the destination StarRocks table: ```java Error: Value count does not match column count. Expect 3, but got 1. Row: 2023-01-01T18:29:00Z,cpu0,80.99 Error: Value count does not match column count. Expect 3, but got 1. Row: 2023-01-01T18:29:10Z,cpu1,75.23 Error: Value count does not match column count. Expect 3, but got 1. Row: 2023-01-01T18:29:20Z,cpu2,59.44 ``` The reason for this issue is as follows: The column separator specified in the load command or statement differs from the column separator that is actually used in the source data file. In the preceding example, the CSV-formatted data file consists of three columns, which are separated with commas (`,`). However, `\t` is specified as the column separator in the load command or statement. As a result, the three columns from the source data file are incorrectly parsed into one column. Specify commas (`,`) as the column separator in the load command or statement. Then, submit the load job again. #### 6. What do I do if the "current running txns on db XXX is 100, larger than limit 100" error occurs?[​](#6-what-do-i-do-if-the-current-running-txns-on-db-xxx-is-100-larger-than-limit-100-error-occurs "Direct link to 6. What do I do if the \"current running txns on db XXX is 100, larger than limit 100\" error occurs?") Increase the value of the FE configuration `max_running_txn_num_per_db`. #### 7. Why do I get a curl ERRORURL saying `be/storage/error_log` does not exist during data import?[​](#7-why-do-i-get-a-curl-errorurl-saying-bestorageerror_log-does-not-exist-during-data-import "Direct link to 7-why-do-i-get-a-curl-errorurl-saying-bestorageerror_log-does-not-exist-during-data-import") BE error logs are kept for 48 hours by default and are cleaned up afterward. You can adjust the retention time using `load_error_log_reserve_hours`. #### 8. How do I troubleshoot the error “Tablet is in error state … prepare\_segment\_writer meet invalid rssid” during import?[​](#8-how-do-i-troubleshoot-the-error-tablet-is-in-error-state--prepare_segment_writer-meet-invalid-rssid-during-import "Direct link to 8. How do I troubleshoot the error “Tablet is in error state … prepare_segment_writer meet invalid rssid” during import?") This issue is usually caused by version lag. Compare tablet versions at the partition level to check whether publish is stuck. Use the following SQL to compare versions: ```sql SELECT * FROM information_schema.be_tablets; SELECT * FROM information_schema.partitions_meta; ``` If only a few tablets are inconsistent, mark the lagging replicas as bad so they can be cloned from healthy ones. If it's caused by an ongoing large table update or schema change, locate the affected partition based on the error and consider deleting and reloading it. If the issue persists, try restarting FE and the problematic BE; if still ineffective, restart all BEs. #### 9. Why does DELETE fail with “failed to execute delete, transaction id xxx, timeout(ms) 30000”?[​](#9-why-does-delete-fail-with-failed-to-execute-delete-transaction-id-xxx-timeoutms-30000 "Direct link to 9. Why does DELETE fail with “failed to execute delete, transaction id xxx, timeout(ms) 30000”?") Increase the value of the FE configuration `load_straggler_wait_second` to 600 (Default: 300). #### 10. How to handle the error “StarRocks planner use long time 3000 ms …”?[​](#10-how-to-handle-the-error-starrocks-planner-use-long-time-3000-ms- "Direct link to 10. How to handle the error “StarRocks planner use long time 3000 ms …”?") The SQL may be too complex. Increase the value of the session variable `new_planner_optimize_timeout`. #### 11. How to fix the error “Primary-key index exceeds the limit.”?[​](#11-how-to-fix-the-error-primary-key-index-exceeds-the-limit "Direct link to 11. How to fix the error “Primary-key index exceeds the limit.”?") It is because that the Primary Key index exceeded memory limits. You can enable persistent index by setting the table property `enable_persistent_index` to `true`. #### 12. How to resolve “current running txns on db XXX is 100, larger than limit 100”?[​](#12-how-to-resolve-current-running-txns-on-db-xxx-is-100-larger-than-limit-100 "Direct link to 12. How to resolve “current running txns on db XXX is 100, larger than limit 100”?") Increase the value of the FE configuration `max_running_txn_num_per_db`. --- ### Routine Load #### How can I improve loading performance?[​](#how-can-i-improve-loading-performance "Direct link to How can I improve loading performance?") **Method 1: Increase the actual load task parallelism** by splitting a load job into as many parallel load tasks as possible. > **NOTICE** > > This method may consume more CPU resources and cause too many tablet versions. The actual load task parallelism is determined by the following formula composed of several parameters, with an upper limit of the number of BE nodes alive or the number of partitions to be consumed. ```plaintext min(alive_be_number, partition_number, desired_concurrent_number, max_routine_load_task_concurrent_num) ``` Parameter description: * `alive_be_number`: the number of BE nodes alive. * `partition_number`: the number of partitions to be consumed. * `desired_concurrent_number`: the desired load task parallelism for a Routine Load job. The default value is `3`. You can set a higher value for this parameter to increase the actual load task parallelism. * If you have not created a Routine Load job, you need to set this parameter when using [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md) to create a Routine Load job. * If you have already created a Routine Load job, you need to use [ALTER ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/ALTER_ROUTINE_LOAD.md) to modify this parameter. * `max_routine_load_task_concurrent_num`: the default maximum task parallelism for a Routine Load job. The default value is `5`. This parameter is a an FE dynamic parameter. For more information and the configuration method, see [Parameter configuration](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#loading-and-unloading). Therefore, when the number of partitions to be consumed and the number of BE nodes alive are greater than the other two parameters, you can increase the values of `desired_concurrent_number` and `max_routine_load_task_concurrent_num` parameters to increase the actual load task parallelism. For example, the number of partitions to be consumed is `7`, the number of live BE nodes is `5`, and `max_routine_load_task_concurrent_num` is the default value `5`. At this time, if you need to increase the load task parallelism to the upper limit, you need to set `desired_concurrent_number` to `5` (the default value is `3`). Then, the actual task parallelism `min(5,7,5,5)` is computed to be `5`. For more parameter descriptions, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). **Method 2: Increase the amount of data consumed by a Routine Load task from one or more partitions.** > **NOTICE** > > This method may cause delay in data loading. The upper limit of the number of messages that a Routine Load task can consume is determined by either the parameter `max_routine_load_batch_size` which means the maximum number of messages that a load task can consume or the parameter `routine_load_task_consume_second` which means the maximum duration of message consumption. Once an load task consumes enough data that meets either requirement, the consumption is complete. These two parameters are FE dynamic parameters. For more information and the configuration method, see [Parameter configuration](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#loading-and-unloading). You can analyze which parameter determines the upper limit of the amount of data consumed by a load task by viewing the log in **be/log/be.INFO**. By increasing that parameter, you can increase the amount of data consumed by a load task. ```plaintext I0325 20:27:50.410579 15259 data_consumer_group.cpp:131] consumer group done: 41448fb1a0ca59ad-30e34dabfa7e47a0. consume time(ms)=3261, received rows=179190, received bytes=9855450, eos: 1, left_time: -261, left_bytes: 514432550, blocking get time(us): 3065086, blocking put time(us): 24855 ``` Normally, the field `left_bytes` in the log is greater than or equal to `0`, indicating that the amount of data consumed by a load task has not exceeded `max_routine_load_batch_size` within `routine_load_task_consume_second`. This means that a batch of scheduled load tasks can consume all data from Kafka without delay in consumption. In this scenario, you can set a larger value for `routine_load_task_consume_second` to increase the amount of data consumed by a load task from one or more partitions. If the field `left_bytes` is less than `0`, it means that the amount of data consumed by a load task has reached `max_routine_load_batch_size` within `routine_load_task_consume_second`. Every time data from Kafka fills the batch of scheduled load tasks. Therefore, it is highly likely that there is remaining data that has not been consumed in Kafka, causing delay in consumption. In this case, you can set a larger value for `max_routine_load_batch_size`. #### What do I do if the result of SHOW ROUTINE LOAD shows that the load job is in the `PAUSED` state?[​](#what-do-i-do-if-the-result-of-show-routine-load-shows-that-the-load-job-is-in-the-paused-state "Direct link to what-do-i-do-if-the-result-of-show-routine-load-shows-that-the-load-job-is-in-the-paused-state") * Check the field `ReasonOfStateChanged` and it reports the error message `Broker: Offset out of range`. **Cause analysis:** The consumer offset of the load job does not exist in the Kafka partition. **Solution:** You can execute [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) and check the latest consumer offset of the load job in the parameter `Progress`. Then, you can verify if the corresponding message exists in the Kafka partition. If it does not exist, it may be because * The consumer offset specified when the load job is created is an offset in the future. * The message at the specified consumer offset in the Kafka partition has been removed before being consumed by the load job. It is recommended to set a reasonable Kafka log cleaning policy and parameters, such as `log.retention.hours and log.retention.bytes`, based on the loading speed. * Check the field `ReasonOfStateChanged` and it doesn't report the error message `Broker: Offset out of range`. **Cause analysis:** The number of error rows in the load task exceeds the threshold `max_error_number`. **Solution:** You can troubleshoot and fix the issue by using error messages in the fields `ReasonOfStateChanged` and `ErrorLogUrls`. * If it is caused by incorrect data format in the data source, you need to check the data format and fix the issue. After successfully fixing the issue, you can use [RESUME ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/RESUME_ROUTINE_LOAD.md) to resume the paused load job. * If it is because that StarRocks cannot parse the data format in the data source, you need to adjust the threshold `max_error_number`. You can first execute [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) to view the value of `max_error_number`, and then use [ALTER ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/ALTER_ROUTINE_LOAD.md) to increase the threshold. After modifying the threshold, you can use [RESUME ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/RESUME_ROUTINE_LOAD.md) to resume the paused load job. #### What do I do if the result of SHOW ROUTINE LOAD shows that the load job is in the `CANCELLED` state?[​](#what-do-i-do-if-the-result-of-show-routine-load-shows-that-the-load-job-is-in-the-cancelled-state "Direct link to what-do-i-do-if-the-result-of-show-routine-load-shows-that-the-load-job-is-in-the-cancelled-state") **Cause analysis:** The load job encountered an exception during loading, such as the table deleted. **Solution:** When troubleshooting and fixing the issue, you can refer to the error messages in the fields `ReasonOfStateChanged` and `ErrorLogUrls`. However, after fixing the issue, you cannot resume the cancelled load job. #### Can Routine Load guarantee consistency semantics when consuming from Kafka and writing to StarRocks?[​](#can-routine-load-guarantee-consistency-semantics-when-consuming-from-kafka-and-writing-to-starrocks "Direct link to Can Routine Load guarantee consistency semantics when consuming from Kafka and writing to StarRocks?") Routine Load guarantees exactly-once semantics. Each load task is a individual transaction. If an error occurs during the execution of the transaction, the transaction is aborted, and the FE does not update the consumption progress of the relevant partitions of the load tasks. When the FE schedules the load tasks from the task queue next time, the load tasks send the consumption request from the last saved consumption position of the partitions, thus ensuring exactly-once semantics. #### What do I do if Routine Load returns an SSL Authentication error?[​](#what-do-i-do-if-routine-load-returns-an-ssl-authentication-error "Direct link to What do I do if Routine Load returns an SSL Authentication error?") **Error Message:** `routines:tls_process_server_certificate:certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 273ms in state SSL_HANDSHAKE)` **Cause analysis:** The domain in the certificate is different with that of the Kafka Broker. See [more details](https://github.com/confluentinc/librdkafka/issues/4349). **Solution:** Add the property `"property.ssl.endpoint.identification.algorithm"= "none"` to the Routine Load job. #### Why does Routine Load report “JSON data is an array.strip\_outer\_array must be set true”?[​](#why-does-routine-load-report-json-data-is-an-arraystrip_outer_array-must-be-set-true "Direct link to Why does Routine Load report “JSON data is an array.strip_outer_array must be set true”?") Your input data is a JSON array `([{},{}])`. Set the property `strip_outer_array` to `true` to expand it. #### Why do I get “There are more than 100 routine load jobs running” when creating a Routine Load job?[​](#why-do-i-get-there-are-more-than-100-routine-load-jobs-running-when-creating-a-routine-load-job "Direct link to Why do I get “There are more than 100 routine load jobs running” when creating a Routine Load job?") Increase the value of the FE configuration `max_routine_load_job_num`. #### Why does Routine Load job creation fails with “failed to get partition meta” even after configuring SASL?[​](#why-does-routine-load-job-creation-fails-with-failed-to-get-partition-meta-even-after-configuring-sasl "Direct link to Why does Routine Load job creation fails with “failed to get partition meta” even after configuring SASL?") The actual cause can be incorrect SASL configuration. #### How should I handle Routine Load error “Create replicas failed …”?[​](#how-should-i-handle-routine-load-error-create-replicas-failed- "Direct link to How should I handle Routine Load error “Create replicas failed …”?") Adjust the following FE configurations: ```sql admin set frontend config ("tablet_create_timeout_second"="5"); admin set frontend config ("max_create_table_timeout_second"="600"); ``` You set also set them in the configuration file to persist the modification. #### Why does Routine Load report “Bad message format” when consuming Kafka?[​](#why-does-routine-load-report-bad-message-format-when-consuming-kafka "Direct link to Why does Routine Load report “Bad message format” when consuming Kafka?") Kafka uses hostname for communication. Add hostname resolution for Kafka nodes in `/etc/hosts` on all server that host the StarRocks nodes. #### What causes Routine Load to fail with error "failed to send task: failed to submit task. error code: TOO MANY TASKS"?[​](#what-causes-routine-load-to-fail-with-error-failed-to-send-task-failed-to-submit-task-error-code-too-many-tasks "Direct link to What causes Routine Load to fail with error \"failed to send task: failed to submit task. error code: TOO MANY TASKS\"?") It is because the total Routine Load concurrency exceeds cluster capability (which equals to `routine_load_thread_pool_size × number of active BEs`). Solutions: * Reduce loading QPS (recommended cluster QPS < 10). Calculate the cluster QPS based on `cluster routine_load_task_num / routine_load_task_consume_second`. * Increase per-task batch size (> 1 GB) by adjusting `max_routine_load_batch_size` and `routine_load_task_timeout_second`. * Ensure `routine_load_thread_pool_size` is less than half of BE CPU cores. A job’s concurrency is determined by the minimum of the following values: * `kafka_partition_num` * `desired_concurrent_number` * `alive_be_num` * `max_routine_load_task_concurrent_num` You may start adjusting the concurrency by reducing `max_routine_load_task_concurrent_num`. --- ### Stream Load #### 1. Does Stream Load support identifying column names held in the first few rows of a CSV-formatted file, or skipping the first few rows during data reading?[​](#1-does-stream-load-support-identifying-column-names-held-in-the-first-few-rows-of-a-csv-formatted-file-or-skipping-the-first-few-rows-during-data-reading "Direct link to 1. Does Stream Load support identifying column names held in the first few rows of a CSV-formatted file, or skipping the first few rows during data reading?") Stream Load does not support identifying column names held in the first few rows of a CSV-formatted file. Stream Load considers the first few rows to be normal data like the other rows. In v2.5 and earlier, Stream Load does not support skipping the first few rows of a CSV file during data reading. If the first few rows of the CSV file you want to load hold column names, take one of the following actions: * Modify the settings of the tool that you use to export the data. Then, re-export the data as a CSV file that does not hold column names in the first few rows. * Use commands such as `sed -i '1d' filename` to delete the first few rows of the CSV file. * In the load command or statement, use `-H "where: != ''"` to filter out the first few rows of the CSV file. `` is any of the column names held in the first few rows. Note that StarRocks first transforms and then filters the source data. Therefore, if the column names in the first few rows fail to be transformed into their matching destination data types, `NULL` values are returned for them. This means the destination StarRocks table cannot contain columns that are set to `NOT NULL`. * In the load command or statement, add `-H "max_filter_ratio:0.01"` to set a maximum error tolerance that is 1% or lower but can tolerate a few error rows, thereby allowing StarRocks to ignore the data transformation failures in the first few rows. In this case, the Stream Load job can still succeed even if `ErrorURL` is returned to indicate error rows. Do not set `max_filter_ratio` to a large value. If you set `max_filter_ratio` to a large value, some important data quality issues may be missed. From v3.0 onwards, Stream Load supports the `skip_header` parameter, which specifies whether to skip the first few rows of a CSV file. For more information,see [CSV parameters](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md#csv-parameters). #### 2. The data to be loaded into the partition column is not of standard DATE or INT type. For example, the data is in a format like 202106.00. How do I transform the data if I load it by using Stream Load?[​](#2-the-data-to-be-loaded-into-the-partition-column-is-not-of-standard-date-or-int-type-for-example-the-data-is-in-a-format-like-20210600-how-do-i-transform-the-data-if-i-load-it-by-using-stream-load "Direct link to 2. The data to be loaded into the partition column is not of standard DATE or INT type. For example, the data is in a format like 202106.00. How do I transform the data if I load it by using Stream Load?") StarRocks supports transforming data at loading. For more information, see [Transform data at loading](https://docs.starrocks.io/docs/loading/Etl_in_loading.md). Suppose that you want to load a CSV-formatted file named `TEST` and the file consists of four columns, `NO`, `DATE`, `VERSION`, and `PRICE`, among which the data from the `DATE` column is in a non-standard format such as 202106.00. If you want to use `DATE` as the partition column in StarRocks, you need to first create a StarRocks table, for example, one that consists of the following four columns: `NO`, `VERSION`, `PRICE`, and `DATE`. Then, you need to specify the data type of the `DATE` column of the StarRocks table as DATE, DATETIME, or INT. Finally, when you create a Stream Load job, you need to specify the following setting in the load command or statement to transform data from the source `DATE` column's data type to the destination column's data type: ```plain -H "columns: NO,DATE_1, VERSION, PRICE, DATE=LEFT(DATE_1,6)" ``` In the preceding example, `DATE_1` can be considered to be a temporarily named column mapping the destination `DATE` column, and the final results loaded into the destination `DATE` column are computed by the `left()` function. Note that you must first list the temporary names of the source columns and then use functions to transform data. The functions supported are scalar functions, including non-aggregate functions and window functions. #### 3. What do I do if my Stream Load job reports the "body exceed max size: 10737418240, limit: 10737418240" error?[​](#3-what-do-i-do-if-my-stream-load-job-reports-the-body-exceed-max-size-10737418240-limit-10737418240-error "Direct link to 3. What do I do if my Stream Load job reports the \"body exceed max size: 10737418240, limit: 10737418240\" error?") The size of the source data file exceeds 10 GB, which is the maximum file size supported by Stream Load. Take one of the following actions: * Use `seq -w 0 n` to split the source data file into smaller files. * Use `curl -XPOST http://be_host:http_port/api/update_config?streaming_load_max_mb=` to adjust the value of the [BE configuration item](https://docs.starrocks.io/docs/administration/management/BE_configuration.md#configure-be-dynamic-parameters) `streaming_load_max_mb` to increase the maximum file size. #### 4. How can I load real NULL values instead of writing “null” into a string column via Stream Load?[​](#4-how-can-i-load-real-null-values-instead-of-writing-null-into-a-string-column-via-stream-load "Direct link to 4. How can I load real NULL values instead of writing “null” into a string column via Stream Load?") Use the replace function: ```bash -H "columns: pk, temp, pd_type=replace(temp,'NULL',NULL)" ``` #### 5. The field name “role” causes a Stream Load error. How should columns be named?[​](#5-the-field-name-role-causes-a-stream-load-error-how-should-columns-be-named "Direct link to 5. The field name “role” causes a Stream Load error. How should columns be named?") `role` is a reserved keyword. Use backticks to wrap the reserved keyword when you use it in names. Example: ```bash -H $'columns:k1,`role`' ``` --- ### Synchronize data from MySQL in real time #### What do I do if a Flink job reports an error?[​](#what-do-i-do-if-a-flink-job-reports-an-error "Direct link to What do I do if a Flink job reports an error?") A Flink job reports the error `Could not execute SQL statement. Reason:org.apache.flink.table.api.ValidationException: One or more required options are missing.` A possible reason is that the required configuration information is missing in multiple sets of rules, such as `[table-rule.1]` and `[table-rule.2]`, in the SMT configuration file **config\_prod.conf**. You can check whether each set of rules, such as `[table-rule.1]` and `[table-rule.2]` is configured with the required database, table, and Flink connector information. #### How can I make Flink automatically restart failed tasks?[​](#how-can-i-make-flink-automatically-restart-failed-tasks "Direct link to How can I make Flink automatically restart failed tasks?") Flink automatically restarts failed tasks through the [checkpointing mechanism](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/checkpointing/) and [restart strategy](https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/ops/state/task_failure_recovery/). For example, if you need to enable the checkpointing mechanism and use the default restart strategy, which is the fixed delay restart strategy, you can configure the following information in the configuration file **flink-conf.yaml**: ```bash execution.checkpointing.interval: 300000 state.backend: filesystem state.checkpoints.dir: file:///tmp/flink-checkpoints-directory ``` Parameter description: > **NOTE** > > For more detailed parameter descriptions in Flink documentation, see [Checkpointing](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/checkpointing/). * `execution.checkpointing.interval`: the base time interval of checkpointing. Unit: millisecond. To enable the checkpointing mechanism, you need to set this parameter to a value greater than `0`. * `state.backend`: specifies the state backend to determine how the state is represented internally, and how and where it is persisted upon checkpointing. Common values are `filesystem` or `rocksdb`. After the checkpointing mechanism is enabled, the state is persisted upon checkpoints to prevent data loss and ensure data consistency after recovery. For more information on state, see [State Backends](https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/state_backends/). * `state.checkpoints.dir`: the directory to which checkpoints are written to. #### How can I manually stop a Flink job and later restore it to the state before stopping?[​](#how-can-i-manually-stop-a-flink-job-and-later-restore-it-to-the-state-before-stopping "Direct link to How can I manually stop a Flink job and later restore it to the state before stopping?") You can manually trigger a [savepoint](https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/savepoints/) when stopping a Flink job (a savepoint is a consistent image of the execution state of a streaming Flink job, and is created based on the checkpointing mechanism). Later, you can restore the Flink job from the specified savepoint. 1. Stop the Flink job with a savepoint. The following command automatically triggers a savepoint for the Flink job `jobId` and stops the Flink job. Additionally, you can specify a target file system directory to store the savepoint. ```bash bin/flink stop --type [native/canonical] --savepointPath [:targetDirectory] :jobId ``` Parameter description: * `jobId`: You can view the Flink job ID from the Flink WebUI or by running `flink list -running` on the command line. * `targetDirectory`: You can specify `state.savepoints.dir` as the default directory for storing savepoints in the Flink configuration file **flink-conf.yml**. When a savepoint is triggered, the savepoint is stored in this default directory and you do not need to specify a directory . ```bash state.savepoints.dir: [file:// or hdfs://]/home/user/savepoints_dir ``` 2. Resubmit the Flink job with the preceding savepoint specified. ```bash ./flink run -c com.starrocks.connector.flink.tools.ExecuteSQL -s savepoints_dir/savepoints-xxxxxxxx flink-connector-starrocks-xxxx.jar -f flink-create.all.sql ``` --- ### Operation and Maintenance This topic provides answers to some questions related to operation and maintenance. #### Can the `trash` directory be cleaned up?[​](#can-the-trash-directory-be-cleaned-up "Direct link to can-the-trash-directory-be-cleaned-up") You can configure the FE parameter `catalog_trash_expire_second` to specify how long files stay in the FE `trash` directory (Default: 24 hours). The BE parameter `trash_file_expire_time_sec` controls BE trash cleanup intervals (Default: 24 hours). After a DROP TABLE or DROP DATABASE, the data first enters the FE trash and is kept for one day, during which it can be recovered with RECOVER. After that, it moves to the BE trash, which also keeps it for 24 hours. #### Do tablets have a primary–secondary relationship? If some replicas are missing, how does it impact queries?[​](#do-tablets-have-a-primarysecondary-relationship-if-some-replicas-are-missing-how-does-it-impact-queries "Direct link to Do tablets have a primary–secondary relationship? If some replicas are missing, how does it impact queries?") If the table property `replicated_storage` is set to `true`, writes use a primary–secondary mechanism: data is written to the primary replica first and then synchronized to others. Multiple replicas generally do not have a major impact on query performance. #### Can I measure CPU and memory usage for a task through the monitoring interface?[​](#can-i-measure-cpu-and-memory-usage-for-a-task-through-the-monitoring-interface "Direct link to Can I measure CPU and memory usage for a task through the monitoring interface?") You can check the `cpucostns` and `memcostbytes` fields in `fe.audit.log`. #### Creating a materialized view on a Unique Key table returns an error "The aggregation type of column\[now\_time] must be same as the aggregate type of base column in aggregate table". Does Unique Key table support materialized views?[​](#creating-a-materialized-view-on-a-unique-key-table-returns-an-error-the-aggregation-type-of-columnnow_time-must-be-same-as-the-aggregate-type-of-base-column-in-aggregate-table-does-unique-key-table-support-materialized-views "Direct link to Creating a materialized view on a Unique Key table returns an error \"The aggregation type of column[now_time] must be same as the aggregate type of base column in aggregate table\". Does Unique Key table support materialized views?") The error indicates that the aggregation type of the materialized view must match that of the base table. For Unique Key tables, you can only materialized views to adjust their sort key order. For example, if base table `tableA` has columns `k1`, `k2`, `k3`, with `k1` and `k2` as sort keys. While your queries contain the clause `WHERE k3=x` and need to be accelerated prefix index, you may create a materialized view that uses `k3` as the first column: ```sql CREATE MATERIALIZED VIEW k3_as_key AS SELECT k3, k2, k1 FROM tableA; ``` #### How can I get import volume metrics with precise timestamps? Is `query_latency` the metric for average response time?[​](#how-can-i-get-import-volume-metrics-with-precise-timestamps-is-query_latency-the-metric-for-average-response-time "Direct link to how-can-i-get-import-volume-metrics-with-precise-timestamps-is-query_latency-the-metric-for-average-response-time") Table-level import metrics can be obtained from: `http://user:password@fe_host:http_port/metrics?type=json&with_table_metrics=all`. Cluster-level data can be retrieved from: `http://fe_host:http_port/api/show_data` (increment must be calculated manually). `query_latency` provides percentile query response time. #### What is the difference between SHOW PROC '/backends' and SHOW BACKENDS?[​](#what-is-the-difference-between-show-proc-backends-and-show-backends "Direct link to What is the difference between SHOW PROC '/backends' and SHOW BACKENDS?") `SHOW PROC '/backends'` retrieve metadata from the current FE and may lag. While `SHOW BACKENDS` retrieve metadata from the Leader FE and is authoritative. #### Does StarRocks have a timeout mechanism? Why do some client connections persist for a long time?[​](#does-starrocks-have-a-timeout-mechanism-why-do-some-client-connections-persist-for-a-long-time "Direct link to Does StarRocks have a timeout mechanism? Why do some client connections persist for a long time?") Yes. You can configure the system variable `wait_timeout` (Default: 8 hours) to adjust the connection timeout. Example: ```sql SET GLOBAL wait_timeout = 3600; ``` #### Can GRANT authorize multiple tables in one statement?[​](#can-grant-authorize-multiple-tables-in-one-statement "Direct link to Can GRANT authorize multiple tables in one statement?") No. Statements like `GRANT on db1.tb1, db1.tb2` are not supported. #### Can I revoke privileges on a specific table when the ALL TABLES privilege is granted?[​](#can-i-revoke-privileges-on-a-specific-table-when-the-all-tables-privilege-is-granted "Direct link to Can I revoke privileges on a specific table when the ALL TABLES privilege is granted?") Subset revocation is not supported. It is recommended to grant privileges at the database or table level. #### Does StarRocks support table-level and row-level privileges?[​](#does-starrocks-support-table-level-and-row-level-privileges "Direct link to Does StarRocks support table-level and row-level privileges?") Table-level access control is supported. Row- and column-level access controls are not supported in the Open-source Edition. #### If a Primary Key table is not partitioned, does hot–cold data separation still work?[​](#if-a-primary-key-table-is-not-partitioned-does-hotcold-data-separation-still-work "Direct link to If a Primary Key table is not partitioned, does hot–cold data separation still work?") No. Hot–cold separation is based on partitions. #### Can I change the data storage path if the data was mistakenly placed in the root directory?[​](#can-i-change-the-data-storage-path-if-the-data-was-mistakenly-placed-in-the-root-directory "Direct link to Can I change the data storage path if the data was mistakenly placed in the root directory?") Yes. Update `storage_root_path` in `be.conf` to add a new disk, and use semicolons to separate paths. #### How do I check the StarRocks version of FE?[​](#how-do-i-check-the-starrocks-version-of-fe "Direct link to How do I check the StarRocks version of FE?") Run `SHOW FRONTENDS;` and check the `Version` field. #### Does StarRocks support DELETE with nested subqueries?[​](#does-starrocks-support-delete-with-nested-subqueries "Direct link to Does StarRocks support DELETE with nested subqueries?") From v2.3 onwards, Primary Key tables supports full DELETE WHERE syntax. See [Reference - DELETE](https://docs.starrocks.io/zh/docs/sql-reference/sql-statements/table_bucket_part_index/DELETE/) for details. #### For dynamic partitions, if I don't want old partitions automatically cleaned, can I simply omit dynamic\_partition.start?[​](#for-dynamic-partitions-if-i-dont-want-old-partitions-automatically-cleaned-can-i-simply-omit-dynamic_partitionstart "Direct link to For dynamic partitions, if I don't want old partitions automatically cleaned, can I simply omit dynamic_partition.start?") No. Set it to a very large value. #### If a BE machine has faulty memory and needs maintenance, what should be done?[​](#if-a-be-machine-has-faulty-memory-and-needs-maintenance-what-should-be-done "Direct link to If a BE machine has faulty memory and needs maintenance, what should be done?") [Decommission](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md#be) the BE. After repair, add it back to the cluster. #### Can I enforce all future partitions to use SSD by default?[​](#can-i-enforce-all-future-partitions-to-use-ssd-by-default "Direct link to Can I enforce all future partitions to use SSD by default?") No. Default is HDD. Manual configuration is required. #### After adding new BEs, tablets automatically rebalanced. Can I decommission old BEs immediately?[​](#after-adding-new-bes-tablets-automatically-rebalanced-can-i-decommission-old-bes-immediately "Direct link to After adding new BEs, tablets automatically rebalanced. Can I decommission old BEs immediately?") Yes. You do not need to wait until the rebalancing completed. Up to two nodes can be decommissioned at once. #### Will adding new BEs and removing old ones affect performance?[​](#will-adding-new-bes-and-removing-old-ones-affect-performance "Direct link to Will adding new BEs and removing old ones affect performance?") Rebalancing happens automatically and should not affect normal operations. It is recommended to remove nodes one at a time. #### How to replace six BE nodes with six new ones?[​](#how-to-replace-six-be-nodes-with-six-new-ones "Direct link to How to replace six BE nodes with six new ones?") Add six new BE nodes, and then decommission the old ones one by one. #### Why doesn't unhealthy tablet count decrease after decommissioning a node?[​](#why-doesnt-unhealthy-tablet-count-decrease-after-decommissioning-a-node "Direct link to Why doesn't unhealthy tablet count decrease after decommissioning a node?") Check for single-replica tables. Other repairs may be blocked if they continuously retry. #### What does this BE log mean? "tcmalloc: large alloc xxxxxxxx bytes"[​](#what-does-this-be-log-mean-tcmalloc-large-alloc-xxxxxxxx-bytes "Direct link to What does this BE log mean? \"tcmalloc: large alloc xxxxxxxx bytes\"") A large memory allocation request occurred, often caused by large queries. Check the corresponding `query_id` in `be.INFO` to locate the SQL. #### Will tablet migration after adding nodes cause disk I/O fluctuations?[​](#will-tablet-migration-after-adding-nodes-cause-disk-io-fluctuations "Direct link to Will tablet migration after adding nodes cause disk I/O fluctuations?") Yes, temporary I/O fluctuation is expected during balancing. #### What are the recommended data migration methods for deployments on the cloud?[​](#what-are-the-recommended-data-migration-methods-for-deployments-on-the-cloud "Direct link to What are the recommended data migration methods for deployments on the cloud?") * To migrate a single table, you can: * Create a StarRocks external table, and then load the data from it using INSERT INTO SELECT. * Read data from the source BE using a Spark-connector-based programme, and load the data into the target BE using encapsulated STREAM LOAD. * To migrate multiple tables, you can use Backup and Restore. First, back up the data from the source cluster to the remote storage. Then, restore the data from the remote storage to the target cluster. * If your cluster uses HDFS as the remote storage, you can first use `distcp` to migrate the data files, and then use Broker Load to load the data into the target cluster. #### How can I resolve the error "failed to create task: disk ... exceed limit usage"?[​](#how-can-i-resolve-the-error-failed-to-create-task-disk--exceed-limit-usage "Direct link to How can I resolve the error \"failed to create task: disk ... exceed limit usage\"?") The disk is full. Scale up the storage or clean trash. #### FE logs show "tablet migrate failed". How can I solve this?[​](#fe-logs-show-tablet-migrate-failed-how-can-i-solve-this "Direct link to FE logs show \"tablet migrate failed\". How can I solve this?") It is likely because the `storage_root_path` is HDD and the table property `storage_medium` is set to `SSD`. You can set the table property to `HDD`: ```sql ALTER TABLE db.table MODIFY PARTITION (*) SET("storage_medium"="HDD"); ``` #### Can I migrate FE to another machine by copying metadata only?[​](#can-i-migrate-fe-to-another-machine-by-copying-metadata-only "Direct link to Can I migrate FE to another machine by copying metadata only?") No. Use the recommended method: add a new node, and then remove the old one. #### To disks on one BE have uneven usage (500 GB 99%, 2 TB 20%). Why is balancing not happening?[​](#to-disks-on-one-be-have-uneven-usage-500-gb-99-2-tb-20-why-is-balancing-not-happening "Direct link to To disks on one BE have uneven usage (500 GB 99%, 2 TB 20%). Why is balancing not happening?") Balancing assumes equal-sized disks. Use disks of the same size to ensure even distribution. #### If `max_backend_down_time_second` is set to `3600`, does it mean that I must recover the failed BE within one hour?[​](#if-max_backend_down_time_second-is-set-to-3600-does-it-mean-that-i-must-recover-the-failed-be-within-one-hour "Direct link to if-max_backend_down_time_second-is-set-to-3600-does-it-mean-that-i-must-recover-the-failed-be-within-one-hour") If the downtime of the BE exceeds this configuration, FE will replenish replicas on other BE. When the failed BE is added back to the cluster, large rebalancing costs may occur. #### Is there an IDE tool to export table structures and views from the production environment?[​](#is-there-an-ide-tool-to-export-table-structures-and-views-from-the-production-environment "Direct link to Is there an IDE tool to export table structures and views from the production environment?") Yes. See [olapdb-tool](https://github.com/Astralidea/olapdb-tool). #### Can multiple system variables be set in a single SQL (for example, timeout and parallelism)?[​](#can-multiple-system-variables-be-set-in-a-single-sql-for-example-timeout-and-parallelism "Direct link to Can multiple system variables be set in a single SQL (for example, timeout and parallelism)?") Yes. Example: ```sql SELECT /*+ SET_VAR(query_timeout=1, is_report_success=true, parallel_fragment_exec_instance_num=2) */ COUNT(1) FROM table; ``` #### Do VARCHAR columns in sort keys follow the 36-byte prefix limit?[​](#do-varchar-columns-in-sort-keys-follow-the-36-byte-prefix-limit "Direct link to Do VARCHAR columns in sort keys follow the 36-byte prefix limit?") VARCHAR columns are truncated based on actual length. Only the first column gets a short-key index. Place VARCHAR sort keys in the third position if possible. #### BE fails to start with "while lock file" error. How can I deal with it?[​](#be-fails-to-start-with-while-lock-file-error-how-can-i-deal-with-it "Direct link to BE fails to start with \"while lock file\" error. How can I deal with it?") BE process is still running. Kill the daemon process and restart. #### Do clients need to explicitly connect to observer FEs for read-only queries?[​](#do-clients-need-to-explicitly-connect-to-observer-fes-for-read-only-queries "Direct link to Do clients need to explicitly connect to observer FEs for read-only queries?") No. Write requests automatically route to the leader FE; observers serve read-only queries. #### FE exits with `LOG_FILE_NOT_FOUND` caused by too many open files. How can I solve this?[​](#fe-exits-with-log_file_not_found-caused-by-too-many-open-files-how-can-i-solve-this "Direct link to fe-exits-with-log_file_not_found-caused-by-too-many-open-files-how-can-i-solve-this") Check OS file descriptor limits by running `cat /proc/$pid/limits`, and run `lsof -n -p $fe_pid>/tmp/fe_fd.txt` to inspect the file descriptor in use. #### Are there limits on partition and bucket numbers?[​](#are-there-limits-on-partition-and-bucket-numbers "Direct link to Are there limits on partition and bucket numbers?") * For partitions, the default limit `4096` (configurable via the FE configuration `max_partitions_in_one_batch`). * For buckets, there is no limit. The recommended size for each bucket is 1 GB. #### How to manually switch FE leader?[​](#how-to-manually-switch-fe-leader "Direct link to How to manually switch FE leader?") Stop the current Leader, and a new leader will be elected automatically. #### How to replace three FE nodes (1 Leader and 2 Follower) with new machines?[​](#how-to-replace-three-fe-nodes-1-leader-and-2-follower-with-new-machines "Direct link to How to replace three FE nodes (1 Leader and 2 Follower) with new machines?") 1. Add 2 new Followers to the cluster. 2. Remove 1 old Follower. 3. Add one last new Follower. 4. Remove the remaining old Follower. 5. Remove the old Leader. #### Dynamic partition creation not working as expected. Why?[​](#dynamic-partition-creation-not-working-as-expected-why "Direct link to Dynamic partition creation not working as expected. Why?") Dynamic partition check runs every 10 minutes. This behavior is controlled by the FE configuration `dynamic_partition_check_interval_seconds`. #### FE logs frequently show "connect processor exception because，java.io.IOException: Connection reset by peer". Why?[​](#fe-logs-frequently-show-connect-processor-exception-becausejavaioioexception-connection-reset-by-peer-why "Direct link to FE logs frequently show \"connect processor exception because，java.io.IOException: Connection reset by peer\". Why?") It is likely because client-side disconnects, connection pool drops, or network issues. Check OS backlog metrics and network stability. Check if the following metrics have changed: ```bash netstat -s | grep -i LISTEN netstat -s | grep TCPBacklogDrop cat /proc/sys/net/core/somaxconn ``` #### After I execute TRUNCATE statements, when will the storage space to be released?[​](#after-i-execute-truncate-statements-when-will-the-storage-space-to-be-released "Direct link to After I execute TRUNCATE statements, when will the storage space to be released?") Immediately. #### How to check whether bucket distribution is balanced?[​](#how-to-check-whether-bucket-distribution-is-balanced "Direct link to How to check whether bucket distribution is balanced?") Run the following command: ```sql SHOW TABLET FROM db.table PARTITION (); ``` #### How to solve the error "Fail to get master client from cache"?[​](#how-to-solve-the-error-fail-to-get-master-client-from-cache "Direct link to How to solve the error \"Fail to get master client from cache\"?") It is an FE–BE communication failure. Check IP and port connectivity. #### How to migrate StarRocks when IP changes?[​](#how-to-migrate-starrocks-when-ip-changes "Direct link to How to migrate StarRocks when IP changes?") FQDN-mode deployment is recommended. #### Can a non-partitioned table be converted to a partitioned table?[​](#can-a-non-partitioned-table-be-converted-to-a-partitioned-table "Direct link to Can a non-partitioned table be converted to a partitioned table?") No. You can create a new partitioned table and use INSERT INTO SELECT to migrate the data. #### Can I query historical SQL execution? Is there an audit log?[​](#can-i-query-historical-sql-execution-is-there-an-audit-log "Direct link to Can I query historical SQL execution? Is there an audit log?") Yes. See `fe.audit.log`. #### How to convert a non-partitioned table to SSD storage?[​](#how-to-convert-a-non-partitioned-table-to-ssd-storage "Direct link to How to convert a non-partitioned table to SSD storage?") Run the following SQL: ```sql ALTER TABLE db.tbl MODIFY PARTITION (*) SET ("storage_medium"="SSD"); ``` #### After executing ALTER TABLE ADD COLUMN, queries against `information_schema.COLUMNS` show delay. Is this normal?[​](#after-executing-alter-table-add-column-queries-against-information_schemacolumns-show-delay-is-this-normal "Direct link to after-executing-alter-table-add-column-queries-against-information_schemacolumns-show-delay-is-this-normal") Yes. ALTER operations are asynchronous. Check progress with `SHOW ALTER COLUMN`. #### If retention for a dynamic partition table is changed from 366 to 732 days, can historical partitions be auto-created?[​](#if-retention-for-a-dynamic-partition-table-is-changed-from-366-to-732-days-can-historical-partitions-be-auto-created "Direct link to If retention for a dynamic partition table is changed from 366 to 732 days, can historical partitions be auto-created?") Follow these steps: 1. Disable dynamic partitions. ```sql ALTER TABLE db.tbl SET ("dynamic_partition.enable" = "false"); ``` 2. Manually add partitions. ```sql ALTER TABLE db.tbl ADD PARTITIONS START ("2019-01-01") END ("2019-12-31") EVERY (interval 1 day); ``` 3. Re-enable dynamic partitions. ```sql ALTER TABLE db.tbl SET ("dynamic_partition.enable" = "true"); ``` #### Can Routine Load tasks be monitored and alerted when switching from Running to Paused?[​](#can-routine-load-tasks-be-monitored-and-alerted-when-switching-from-running-to-paused "Direct link to Can Routine Load tasks be monitored and alerted when switching from Running to Paused?") Yes. StarRocks supports monitoring metrics for Routine Load tasks and can be connected to alert systems. #### How to diagnose unhealthy replicas?[​](#how-to-diagnose-unhealthy-replicas "Direct link to How to diagnose unhealthy replicas?") Run the following statements to identify UnhealthyTablets: ```sql SHOW PROC '/statistic' SHOW PROC '/statistic/' ``` Then, analyze UnhealthyTablets with `SHOW TABLET tablet_id`. If the result shows that two replicas have consistent data but one replica has inconsistent data—meaning two out of three replicas completed the write successfully—this is considered a successful write. You can then check whether the tablets in UnhealthyTablets are fixed. If they are fixed, it indicates an issue. If the status is changing, you can adjust the loading frequency for the corresponding table. #### Error: "SyntaxErrorException: Reach limit of connections". How to troubleshoot?[​](#error-syntaxerrorexception-reach-limit-of-connections-how-to-troubleshoot "Direct link to Error: \"SyntaxErrorException: Reach limit of connections\". How to troubleshoot?") Increase per-user limits by running the following command: ```sql ALTER USER 'jack' SET PROPERTIES ('max_user_connections'='1000'); ``` Also check load balancers and idle connection accumulation (`wait_timeout`). #### How does XFS and ext4 affect QPS?[​](#how-does-xfs-and-ext4-affect-qps "Direct link to How does XFS and ext4 affect QPS?") StarRocks typically performs better with XFS. #### How long before a BE is considered down and tablet migration begins?[​](#how-long-before-a-be-is-considered-down-and-tablet-migration-begins "Direct link to How long before a BE is considered down and tablet migration begins?") 1. When Heartbeat (default: every 5 seconds) failed 3 times, the BE is marked as not alive. 2. After that, there is an intended delay before executing Clone, which is 60 seconds. 3. Then, the replica clone starts. If BE recovers later, its replicas are deleted. #### Tablet scheduling on new nodes is slow (only 100 at a time). How to adjust?[​](#tablet-scheduling-on-new-nodes-is-slow-only-100-at-a-time-how-to-adjust "Direct link to Tablet scheduling on new nodes is slow (only 100 at a time). How to adjust?") Tune the following FE configurations: ```sql ADMIN SET FRONTEND CONFIG ("schedule_slot_num_per_path"="8"); ADMIN SET FRONTEND CONFIG ("max_scheduling_tablets"="1000"); ADMIN SET FRONTEND CONFIG ("max_balancing_tablets"="1000"); ``` #### Is BACKUP operation serial? It seems only one HDFS directory is changing[​](#is-backup-operation-serial-it-seems-only-one-hdfs-directory-is-changing "Direct link to Is BACKUP operation serial? It seems only one HDFS directory is changing") BACKUP operations are parallel, but upload to HDFS uses a single worker, which is controlled by the BE configuration `upload_worker_count`. Adjust it with caution because it might affect the disk I/O and network I/O. #### How to solve the FE OOM error "OutOfMemoryError: GC overhead limit exceeded"?[​](#how-to-solve-the-fe-oom-error-outofmemoryerror-gc-overhead-limit-exceeded "Direct link to How to solve the FE OOM error \"OutOfMemoryError: GC overhead limit exceeded\"?") Increase FE JVM memory. #### How to solve the error "Cannot truncate a file by broker"?[​](#how-to-solve-the-error-cannot-truncate-a-file-by-broker "Direct link to How to solve the error \"Cannot truncate a file by broker\"?") 1. Check broker logs for error messages. 2. Check BE warning logs for error "remote file checksum is invalid. remote:\*\*\*\* local: \*\*\*\*\*". 3. Search the remote number the broker `apache_hdfs_broker.log` for error "receive a check path request, request detail", and identify the duplicate files. 4. Remove or rename problematic remote files and retry. #### How to verify that disk removal has completed?[​](#how-to-verify-that-disk-removal-has-completed "Direct link to How to verify that disk removal has completed?") Run `SHOW PROC '/statistic'` and ensure `UnhealthyTablet` count is zero. #### How to modify replica count for historical partitions?[​](#how-to-modify-replica-count-for-historical-partitions "Direct link to How to modify replica count for historical partitions?") Run the following command: ```sql ALTER TABLE db.tbl MODIFY PARTITION (*) SET("replication_num"="3"); ``` #### For a three-replica table, if the disk for a BE node is damaged, will replicas automatically recover to maintain three copies?[​](#for-a-three-replica-table-if-the-disk-for-a-be-node-is-damaged-will-replicas-automatically-recover-to-maintain-three-copies "Direct link to For a three-replica table, if the disk for a BE node is damaged, will replicas automatically recover to maintain three copies?") Yes, if enough BE nodes are available. #### How to troubleshoot if the error "reach limit connections" continues even after increasing user limits?[​](#how-to-troubleshoot-if-the-error-reach-limit-connections-continues-even-after-increasing-user-limits "Direct link to How to troubleshoot if the error \"reach limit connections\" continues even after increasing user limits?") Check load balancers (ProxySQL, F5), idle connection accumulation, and reduce `wait_timeout` to 2–4 hours. #### If FE metadata is lost, is all cluster metadata lost?[​](#if-fe-metadata-is-lost-is-all-cluster-metadata-lost "Direct link to If FE metadata is lost, is all cluster metadata lost?") Metadata resides in FE. With only one FE, it's unrecoverable. With multiple FEs, you can re-add the failed node and metadata will be replicated. #### How to solve the loading error "INTERNAL\_ERROR, FE leader shows NullPointerException"?[​](#how-to-solve-the-loading-error-internal_error-fe-leader-shows-nullpointerexception "Direct link to How to solve the loading error \"INTERNAL_ERROR, FE leader shows NullPointerException\"?") Add JVM option `-XX:-OmitStackTraceInFastThrow` and restart FE to get full stack trace. #### How to solve the error "The partition column could not be aggregated column" when setting the partition column for a Primary Key table?[​](#how-to-solve-the-error-the-partition-column-could-not-be-aggregated-column-when-setting-the-partition-column-for-a-primary-key-table "Direct link to How to solve the error \"The partition column could not be aggregated column\" when setting the partition column for a Primary Key table?") The partition columns must be key columns. --- ### Other FAQ This topic provides answers to some general questions. #### Do VARCHAR (32) and STRING occupy the same storage space?[​](#do-varchar-32-and-string-occupy-the-same-storage-space "Direct link to Do VARCHAR (32) and STRING occupy the same storage space?") Both are variable-length data types. When you store data of the same length, VARCHAR (32) and STRING occupy the same storage space. #### Do VARCHAR (32) and STRING perform the same for the data query?[​](#do-varchar-32-and-string-perform-the-same-for-the-data-query "Direct link to Do VARCHAR (32) and STRING perform the same for the data query?") Yes. #### Why do TXT files imported from Oracle still appear garbled after I set the character set to UTF-8?[​](#why-do-txt-files-imported-from-oracle-still-appear-garbled-after-i-set-the-character-set-to-utf-8 "Direct link to Why do TXT files imported from Oracle still appear garbled after I set the character set to UTF-8?") To solve this problem, perform the following steps: 1. For example, there is a file named **original**, whose text is garbled. The character set of this file is ISO-8859-1. Run the following code to obtain the character set of the file. ```plaintext file --mime-encoding origin.txt origin.txt: iso-8859-1 ``` 2. Run the `iconv` command to convert the character set of this file into UTF-8. ```plaintext iconv -f iso-8859-1 -t utf-8 origin.txt > origin_utf-8.txt ``` 3. After the conversion, the text of this file still appears garbled. You can then regrade the character set of this file as GBK and convert the character set into UTF-8 again. ```plaintext iconv -f gbk -t utf-8 origin.txt > origin_utf-8.txt ``` #### Is the length of STRING defined by MySQL the same as that defined by StarRocks?[​](#is-the-length-of-string-defined-by-mysql-the-same-as-that-defined-by-starrocks "Direct link to Is the length of STRING defined by MySQL the same as that defined by StarRocks?") For VARCHAR(n), StarRocks defines "n" by bytes and MySQL defines "n" by characters. According to UTF-8, one Chinese character is equal to three bytes. When StarRocks and MySQL define "n" as the same number, MySQL saves three times as many characters as StarRocks. #### Can the data type of partitioned fields of a table be FLOAT, DOUBLE, or DECIMAL?[​](#can-the-data-type-of-partitioned-fields-of-a-table-be-float-double-or-decimal "Direct link to Can the data type of partitioned fields of a table be FLOAT, DOUBLE, or DECIMAL?") No, only DATE, DATETIME, and INT are supported. #### How to check the storage space that is occupied by the data in a table?[​](#how-to-check-the-storage-space-that-is-occupied-by-the-data-in-a-table "Direct link to How to check the storage space that is occupied by the data in a table?") Execute the SHOW DATA statement to see the corresponding storage space. You can also see the data volume, the number of copies, and the number of rows. **Note**: There is a time delay in data statistics. #### How to request a quota increase for the StarRocks database?[​](#how-to-request-a-quota-increase-for-the-starrocks-database "Direct link to How to request a quota increase for the StarRocks database?") To request a quota increase, run the following code: ```plaintext ALTER DATABASE example_db SET DATA QUOTA 10T; ``` #### Does StarRocks support updating particular fields in a table by executing the UPSERT statement?[​](#does-starrocks-support-updating-particular-fields-in-a-table-by-executing-the-upsert-statement "Direct link to Does StarRocks support updating particular fields in a table by executing the UPSERT statement?") StarRocks 2.2 and later support updating specific fields in a table by using the Primary Key table. StarRocks 1.9 and later support updating all fields in a table by using the Primary Key table. For more information, see [Primary Key table](https://docs.starrocks.io/docs/table_design/table_types/primary_key_table.md) in StarRocks 2.2. #### How to swap the data between two tables or two partitions?[​](#how-to-swap-the-data-between-two-tables-or-two-partitions "Direct link to How to swap the data between two tables or two partitions?") Execute the SWAP WITH statement to swap the data between two tables or two partitions. The SWAP WITH statement is more secure than the INSERT OVERWRITE statement. Before you swap the data, check the data first and then see whether the data after the swapping is consistent with the data before the swapping. * Swap two tables: For example, there is a table named table 1. If you want to replace table 1 with another one, perform the following steps: 1. Create a new table named table 2. ```sql create table2 like table1; ``` 2. Use Stream Load, Broker Load, or Insert Into to load data from table 1 into table 2. 3. Replace table 1 with table 2. ```sql ALTER TABLE table1 SWAP WITH table2; ``` By doing so, the data is loaded accurately into table 1. * Swap two partitions: For example, there is a table named table 1. If you want to replace the partition data in table 1, perform the following steps: 1. Create a temporary partition. ```sql ALTER TABLE table1 ADD TEMPORARY PARTITION tp1 VALUES LESS THAN("2020-02-01"); ``` 2. Load the partition data from table 1 into the temporary partition. 3. Replace the partition of table 1 with the temporary partition. ```sql ALTER TABLE table1 REPLACE PARTITION (p1) WITH TEMPORARY PARTITION (tp1); ``` #### This error "error to open replicated environment, will exit" occurs when I restart a frontend (FE)[​](#this-error-error-to-open-replicated-environment-will-exit-occurs-when-i-restart-a-frontend-fe "Direct link to This error \"error to open replicated environment, will exit\" occurs when I restart a frontend (FE)") This error occurs due to BDBJE's bug. To solve this problem, update the BDBJE version to 1.17 or later. #### This error "Broker list path exception" occurs when I query data from a new Apache Hive table[​](#this-error-broker-list-path-exception-occurs-when-i-query-data-from-a-new-apache-hive-table "Direct link to This error \"Broker list path exception\" occurs when I query data from a new Apache Hive table") ##### Problem description[​](#problem-description "Direct link to Problem description") ```plaintext msg:Broker list path exception path=hdfs://172.31.3.136:9000/user/hive/warehouse/zltest.db/student_info/*, broker=TNetworkAddress(hostname:172.31.4.233, port:8000) ``` ##### Solution[​](#solution "Direct link to Solution") Contact the StarRocks technical support and check whether the address and port of the namenode are correct and whether you have permission to access the address and port of the namenode. #### This error "get hive partition metadata failed" occurs when I query data from a new Apache Hive table[​](#this-error-get-hive-partition-metadata-failed-occurs-when-i-query-data-from-a-new-apache-hive-table "Direct link to This error \"get hive partition metadata failed\" occurs when I query data from a new Apache Hive table") ##### Problem description[​](#problem-description-1 "Direct link to Problem description") ```plaintext msg:get hive partition meta data failed: java.net.UnknownHostException: emr-header-1.cluster-242 ``` ##### Solution[​](#solution-1 "Direct link to Solution") Ensure that the network is connected and upload the **host** file to each backend (BE) in your StarRocks cluster. #### This error "do\_open failed. reason = Invalid ORC postscript length" occurs when I access ORC external table in Apache Hive[​](#this-error-do_open-failed-reason--invalid-orc-postscript-length-occurs-when-i-access-orc-external-table-in-apache-hive "Direct link to This error \"do_open failed. reason = Invalid ORC postscript length\" occurs when I access ORC external table in Apache Hive") ##### Problem description[​](#problem-description-2 "Direct link to Problem description") The metadata of the Apache Hive is cached in the FEs. But there is a two-hours time lag for StarRocks to update the metadata. Before StarRocks finishes the update, If you insert new data or update data in the Apache Hive table, the data in HDFS scanned by the BEs and the data obtained by the FEs are different. Therefore, this error occurs. ```plaintext MySQL [bdp_dim]> select * from dim_page_func_s limit 1; ERROR 1064 (HY000): HdfsOrcScanner::do_open failed. reason = Invalid ORC postscript length ``` ##### Solution[​](#solution-2 "Direct link to Solution") To solve this problem, perform one of the following operations: * Upgrade your current version to StarRocks 2.2 or later. * Manually refresh your Apache Hive table. For more information, see [Metadata caching strategy](https://docs.starrocks.io/docs/data_source/External_table.md). #### This error "caching\_sha2\_password cannot be loaded" occurs when I connect external tables of MySQL[​](#this-error-caching_sha2_password-cannot-be-loaded-occurs-when-i-connect-external-tables-of-mysql "Direct link to This error \"caching_sha2_password cannot be loaded\" occurs when I connect external tables of MySQL") ##### Problem description[​](#problem-description-3 "Direct link to Problem description") The default authentication plugin of MySQL 8.0 is caching\_sha2\_password. The default authentication plugin of MySQL 5.7 is mysql\_native\_password. This error occurs because you use the wrong authentication plugin. ##### Solution[​](#solution-3 "Direct link to Solution") To solve this problem, perform one of the following operations: * Connect to the StarRocks. ```sql ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'yourpassword'; ``` * Modify the `my.cnf` file. ```plaintext vim my.cnf [mysqld] default_authentication_plugin=mysql_native_password ``` #### How to release disk space immediately after deleting a table?[​](#how-to-release-disk-space-immediately-after-deleting-a-table "Direct link to How to release disk space immediately after deleting a table?") If you execute the DROP TABLE statement to delete a table, StarRocks takes a while to release the allocated disk space. To release the allocated disk space immediately, execute the DROP TABLE FORCE statement to delete a table. When you execute the DROP TABLE FORCE statement, the StarRocks deletes the table directly without checking whether there are unfinished events in it. We recommend that you execute the DROP TABLE FORCE statement with caution. Because once the table is deleted, you cannot restore it. #### How to view the current version of StarRocks?[​](#how-to-view-the-current-version-of-starrocks "Direct link to How to view the current version of StarRocks?") Run the `select current_version();` command or the CLI command `./bin/show_fe_version.sh` to view the current version. #### How to set the memory size of an FE?[​](#how-to-set-the-memory-size-of-an-fe "Direct link to How to set the memory size of an FE?") The metadata is stored in the memory used by the FE. You can set the memory size of the FE according to the number of tablets as shown in the table below. For example, if the number of tablets is below 1 million, you should allocate a minimum of 16 GB memory to the FE. You can configure the values of the parameters `-Xms` and `-Xmx` in the **JAVA\_OPTS** configuration item in the **fe.conf** file, and the values of the parameters `-Xms` and `-Xmx` should be consistent. Note that the configuration should be same across all FEs because any of the FEs can be elected as a Leader. | Number of tablets | Memory size of each FE | | ----------------- | ---------------------- | | below 1 million | 16 GB | | 1 ～ 2 million | 32 GB | | 2 ～ 5 million | 64 GB | | 5 ～ 10 million | 128 GB | #### How does StarRocks calculate its query time?[​](#how-does-starrocks-calculate-its-query-time "Direct link to How does StarRocks calculate its query time?") StarRocks supports querying data by using multiple threads. Query time refers to the time used by multiple threads to query data. #### Does StarRocks support setting the path when I export data locally?[​](#does-starrocks-support-setting-the-path-when-i-export-data-locally "Direct link to Does StarRocks support setting the path when I export data locally?") No. #### What are the concurrency upper limits of StarRocks?[​](#what-are-the-concurrency-upper-limits-of-starrocks "Direct link to What are the concurrency upper limits of StarRocks?") You can test the concurrency limitations based on the actual business scenarios or simulated business scenarios. According to the feedback of some users, maximum of 20,000 QPS or 30,000 QPS can be achieved. #### Why is the first-time SSB test performance of StarRocks slower than that done the second time?[​](#why-is-the-first-time-ssb-test-performance-of-starrocks-slower-than-that-done-the-second-time "Direct link to Why is the first-time SSB test performance of StarRocks slower than that done the second time?") The speed to read disks for the first query relates to the performance of disks. After the first query, the page cache is generated for the subsequent queries, so the query is faster than before. #### How many BEs need to be configured at least for a cluster?[​](#how-many-bes-need-to-be-configured-at-least-for-a-cluster "Direct link to How many BEs need to be configured at least for a cluster?") StarRocks supports single node deployment, so you need to configure at least one BE. BEs need to be run with AVX2, so we recommend that you deploy BEs on machines with 8-core and 16GB or higher configurations. #### How to set data permissions when I use Apache Superset to visualize the data in StarRocks?[​](#how-to-set-data-permissions-when-i-use-apache-superset-to-visualize-the-data-in-starrocks "Direct link to How to set data permissions when I use Apache Superset to visualize the data in StarRocks?") You can create a new user account and then set the data permission by granting permissions on the table query to the user. #### Why does the profile fail to display after I set `enable_profile` to `true`?[​](#why-does-the-profile-fail-to-display-after-i-set-enable_profile-to-true "Direct link to why-does-the-profile-fail-to-display-after-i-set-enable_profile-to-true") The report is only submitted to the leader FE for access. #### How to check field annotations in the tables of StarRocks?[​](#how-to-check-field-annotations-in-the-tables-of-starrocks "Direct link to How to check field annotations in the tables of StarRocks?") Run the `show create table xxx` command. #### When I create a table, how to specify the default value for the NOW() function?[​](#when-i-create-a-table-how-to-specify-the-default-value-for-the-now-function "Direct link to When I create a table, how to specify the default value for the NOW() function?") Only StarRocks 2.1 or later version supports specifying the default value for a function. For versions earlier than StarRocks 2.1, you can only specify a constant for a function. #### How can I release the storage space of BE nodes?[​](#how-can-i-release-the-storage-space-of-be-nodes "Direct link to How can I release the storage space of BE nodes?") You can remove the directory `trash` using `rm -rf` command. If you have already restored your data from snapshot, you can remove the directory `snapshot`. #### Can add extra disks to BE nodes?[​](#can-add-extra-disks-to-be-nodes "Direct link to Can add extra disks to BE nodes?") Yes. You can add the disks to the directory specified by the BE configuration item `storage_root_path`. #### How can I prevent expression partition conflicts caused by concurrent execution of loading tasks and partition creation tasks?[​](#how-can-i-prevent-expression-partition-conflicts-caused-by-concurrent-execution-of-loading-tasks-and-partition-creation-tasks "Direct link to How can I prevent expression partition conflicts caused by concurrent execution of loading tasks and partition creation tasks?") Currently, for tables with the expression partitioning strategy, partitions created during loading tasks conflict with those created during ALTER TABLE tasks. Since loading tasks take priority, any conflicting ALTER tasks will fail. To prevent this issue, consider the following workarounds: * If you use coarse time-based partitions (or example, partitioning by day or month), you can prevent ALTER operations from crossing time boundaries, reducing the risk of partition creation failures. * If you use fine-grained time-based partitions (or example, partitioning by hour), you can manually create partitions for a future time range to ensure that ALTER operations are not disrupted by new partition created by loading tasks. You can use the [EXPLAIN ANALYZE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plan_profile/EXPLAIN_ANALYZE.md) feature to trigger partition creation by executing an INSERT statement without committing the transaction. This allows you to create the necessary partitions without affecting actual data. The following example demonstrates how to create partitions for the next 8 hours: ```sql CREATE TABLE t( event_time DATETIME ) PARTITION BY date_trunc('hour', event_time); EXPLAIN ANALYZE INSERT INTO t (event_time) SELECT DATE_ADD(NOW(), INTERVAL d hour) FROM table(generate_series(0, 8)) AS g(d); SHOW PARTITIONS FROM t; ``` --- ### Troubleshooting Resource Isolation This topic provides answers to some frequently asked questions about resource isolation. #### Resource Group[​](#resource-group "Direct link to Resource Group") ##### What resources must be configured in a resource group?[​](#what-resources-must-be-configured-in-a-resource-group "Direct link to What resources must be configured in a resource group?") The CPU resource limit must be configured. You must set either `cpu_weight` or `exclusive_cpu_core`, and the values must be greater than 0. ##### Does StarRocks support hard resource limits?[​](#does-starrocks-support-hard-resource-limits "Direct link to Does StarRocks support hard resource limits?") Yes. StarRocks supports hard limits for memory. From v3.3.5 onwards, StarRocks supports hard limits for CPU via `exclusive_cpu_cores`. ##### How is CPU allocated among resource groups?[​](#how-is-cpu-allocated-among-resource-groups "Direct link to How is CPU allocated among resource groups?") When multiple resource groups run queries simultaneously, CPU usage is proportional to each group’s `cpu_core_limit`. If the normal group exceeds the `BE vCPU cores - short_query.cpu_core_limit` within a scheduling cycle, it will not be scheduled further in that cycle. ##### How are resources calculated for the `short_query` resource group?[​](#how-are-resources-calculated-for-the-short_query-resource-group "Direct link to how-are-resources-calculated-for-the-short_query-resource-group") When the `short_query` resource group has running queries, the CPU limit of all normal groups becomes `BE vCPU cores − short_query.cpu_core_limit`. If the `short_query` resource group is idle, its resources can be used by normal groups. ##### How are queries without a matched resource group handled?[​](#how-are-queries-without-a-matched-resource-group-handled "Direct link to How are queries without a matched resource group handled?") They use the default resource group `default_wg`, which has the following resource limits and properties: * `cpu_core_limit` = vCPU cores * `mem_limit` = 100% * `type` = `normal` ##### If the resource group `rg3` has no queries and all resources are allocated to resource groups `rg1` and `rg2`, will those resources be reallocated when `rg3` receives a large query?[​](#if-the-resource-group-rg3-has-no-queries-and-all-resources-are-allocated-to-resource-groups-rg1-and-rg2-will-those-resources-be-reallocated-when-rg3-receives-a-large-query "Direct link to if-the-resource-group-rg3-has-no-queries-and-all-resources-are-allocated-to-resource-groups-rg1-and-rg2-will-those-resources-be-reallocated-when-rg3-receives-a-large-query") Yes. Reclamation happens gradually and stabilizes within tens of milliseconds to seconds. ##### What is the role of classifiers? What if no classifier matches, or two classifiers overlap?[​](#what-is-the-role-of-classifiers-what-if-no-classifier-matches-or-two-classifiers-overlap "Direct link to What is the role of classifiers? What if no classifier matches, or two classifiers overlap?") If no classifier matches, the query falls back to the default resource group `default_wg`. Classifiers have weights; if a query matches multiple classifiers, the one with the highest weight is selected. ##### Why do resource groups require `mem_limit` if BE already uses 90% memory by default?[​](#why-do-resource-groups-require-mem_limit-if-be-already-uses-90-memory-by-default "Direct link to why-do-resource-groups-require-mem_limit-if-be-already-uses-90-memory-by-default") `mem_limit` restricts memory at the resource-group level. It applies only to queries match that resource group. ##### If the resource group's `query_type` is set to `insert`, does INSERT INTO SELECT only limit INSERT or also SELECT?[​](#if-the-resource-groups-query_type-is-set-to-insert-does-insert-into-select-only-limit-insert-or-also-select "Direct link to if-the-resource-groups-query_type-is-set-to-insert-does-insert-into-select-only-limit-insert-or-also-select") Only the SELECT part is limited by the resource group. The INSERT operation is not limited. ##### If the resource groups `mem_limit` is set to `20%`, is the usable memory calculated as `BE_memory * 90% * 20%`? What if total `mem_limit` exceeds 100% when there are multiple resource groups?[​](#if-the-resource-groups-mem_limit-is-set-to-20-is-the-usable-memory-calculated-as-be_memory--90--20-what-if-total-mem_limit-exceeds-100-when-there-are-multiple-resource-groups "Direct link to if-the-resource-groups-mem_limit-is-set-to-20-is-the-usable-memory-calculated-as-be_memory--90--20-what-if-total-mem_limit-exceeds-100-when-there-are-multiple-resource-groups") Total `mem_limit` across groups can exceed 100%. But if a query exceeds its resource group limit, it will fail. ##### How can I verify whether a resource group is applied to a query?[​](#how-can-i-verify-whether-a-resource-group-is-applied-to-a-query "Direct link to How can I verify whether a resource group is applied to a query?") Check `fe.audit.log` or run `EXPLAIN VERBOSE ` to view the matched resource group. ##### Are resource groups defined per cluster or per BE node?[​](#are-resource-groups-defined-per-cluster-or-per-be-node "Direct link to Are resource groups defined per cluster or per BE node?") Resources are divided per BE node, and the resource group configuration applies to all BE nodes in the cluster. ##### How to inspect resource group usage or monitoring metrics?[​](#how-to-inspect-resource-group-usage-or-monitoring-metrics "Direct link to How to inspect resource group usage or monitoring metrics?") Use FE/BE metrics endpoints to view specific resource-group-related metrics. * For FE, collect the following metrics from `fe_host:8030/metrics?type=json`: * `starrocks_fe_query_resource_group`: The number of queries historically run in this resource group (including those currently running). * `starrocks_fe_query_resource_group_latency`: The query latency percentile for this resource group. The label type indicates specific percentiles, including `mean`, `75_quantile`, `95_quantile`, `98_quantile`, `99_quantile`, `999_quantile`. * `starrocks_fe_query_resource_group_err`: The number of queries in this resource group that encountered an error. * For BE, collect the following metrics from `be_host:8040/metrics?type=json`: * `starrocks_be_resource_group_cpu_limit_ratio`: The ratio of this resource group's `cpu_core_limit` to the total `cpu_core_limit` across all resource groups. * `starrocks_be_resource_group_mem_limit_bytes`: The memory limit for this resource group. ##### What is the difference between `short_query` and normal resource groups? Can I create multiple `short_query` resource groups?[​](#what-is-the-difference-between-short_query-and-normal-resource-groups-can-i-create-multiple-short_query-resource-groups "Direct link to what-is-the-difference-between-short_query-and-normal-resource-groups-can-i-create-multiple-short_query-resource-groups") Only one `short_query` resource group is allowed. When there are queries running in `short_query` resource group, it uses actual BE cores, while normal groups share remaining resources proportionally. ##### Does StarRocks provide query priority or large-query-based prioritization?[​](#does-starrocks-provide-query-priority-or-large-query-based-prioritization "Direct link to Does StarRocks provide query priority or large-query-based prioritization?") No priority system exists. A query becomes a “large query” when exceeding any of the configured resource thresholds. ##### Does a resource group belong to a specific BE node or the BE running the query?[​](#does-a-resource-group-belong-to-a-specific-be-node-or-the-be-running-the-query "Direct link to Does a resource group belong to a specific BE node or the BE running the query?") Resource groups apply uniformly across all BEs in the cluster. ##### How should I configure `concurrency_limit`?[​](#how-should-i-configure-concurrency_limit "Direct link to how-should-i-configure-concurrency_limit") It depends on query complexity, cluster size, and workload patterns. ##### How does classifier-based matching work? Is it tied to user and/or database?[​](#how-does-classifier-based-matching-work-is-it-tied-to-user-andor-database "Direct link to How does classifier-based matching work? Is it tied to user and/or database?") Matching depends on classifier attributes such as IP, user, db, role, or `query_type`. ##### How is resource group specified through session variables?[​](#how-is-resource-group-specified-through-session-variables "Direct link to How is resource group specified through session variables?") You can set it as a variable: ```sql SET resource_group = ''; ``` Or specify it in queries via hint: ```sql SELECT /*+ SET_VAR(SET resource_group = '') */ * FROM tbl; ``` ##### Does concurrency control take effect globally, per user, or per BE?[​](#does-concurrency-control-take-effect-globally-per-user-or-per-be "Direct link to Does concurrency control take effect globally, per user, or per BE?") `concurrency_limit` restricts concurrency per resource group, while `pipeline_dop` controls parallelism of a single pipeline instance. ##### Does memory limit take effect globally, per user, or per BE?[​](#does-memory-limit-take-effect-globally-per-user-or-per-be "Direct link to Does memory limit take effect globally, per user, or per BE?") `mem_limit` applies to the resource group per BE. Per-instance memory is controlled by `exec_mem_limit`. ##### Are concurrency and memory limits only effective when resource groups are enabled?[​](#are-concurrency-and-memory-limits-only-effective-when-resource-groups-are-enabled "Direct link to Are concurrency and memory limits only effective when resource groups are enabled?") Concurrency is controlled only by resource groups. Query parallelism is controlled by session variables such as `pipeline_dop`. ##### Will CTAS tasks match resource groups if the `query_type` is set to `INSERT`?[​](#will-ctas-tasks-match-resource-groups-if-the-query_type-is-set-to-insert "Direct link to will-ctas-tasks-match-resource-groups-if-the-query_type-is-set-to-insert") Yes. Resource group will restrict the resources for the SELECT part of CTAS tasks. The big query limits will also take effect if the SELECT part exceeds the threshold. ##### Why can't the queries in the `short_query` resource group consume all CPU?[​](#why-cant-the-queries-in-the-short_query-resource-group-consume-all-cpu "Direct link to why-cant-the-queries-in-the-short_query-resource-group-consume-all-cpu") The `short_query` resource group must leave at least 1 CPU core for normal groups. ##### Without query queue and resource groups, are concurrent queries limited?[​](#without-query-queue-and-resource-groups-are-concurrent-queries-limited "Direct link to Without query queue and resource groups, are concurrent queries limited?") No. Overload results in query timeouts or memory limit errors. ##### With resource groups enabled but query queue disabled, is concurrency limited by resource groups?[​](#with-resource-groups-enabled-but-query-queue-disabled-is-concurrency-limited-by-resource-groups "Direct link to With resource groups enabled but query queue disabled, is concurrency limited by resource groups?") Yes. New queries exceeding the resource group `concurrency_limit` will fail. ##### When is a query recognized as a "big query"?[​](#when-is-a-query-recognized-as-a-big-query "Direct link to When is a query recognized as a \"big query\"?") A query is considered as a big query when it exceeds any of: * `big_query_cpu_second_limit` * `big_query_scan_rows_limit` * `big_query_mem_limit` ##### Can `default_wg` resource limits be changed?[​](#can-default_wg-resource-limits-be-changed "Direct link to can-default_wg-resource-limits-be-changed") No. As a workaround, create a general resource group that can match all queries. Example: ```sql CREATE RESOURCE GROUP general_group TO ( query_type IN ('select', 'insert') ) WITH ( 'cpu_core_limit' = '6', 'mem_limit' = '0.0000000000001' ); ``` ##### Why does the “query\_resource\_group” metric not show newly created groups?[​](#why-does-the-query_resource_group-metric-not-show-newly-created-groups "Direct link to Why does the “query_resource_group” metric not show newly created groups?") The metric is lazy-initialized; it appears only after a query hits that group. ##### If many queries run on a BE, and one hits CPU limit, do all fail?[​](#if-many-queries-run-on-a-be-and-one-hits-cpu-limit-do-all-fail "Direct link to If many queries run on a BE, and one hits CPU limit, do all fail?") Only the query reaching the limit fails. ##### If BE nodes have different memory/CPU sizes, does `mem_limit` or `cpu_core_limit` affect results?[​](#if-be-nodes-have-different-memorycpu-sizes-does-mem_limit-or-cpu_core_limit-affect-results "Direct link to if-be-nodes-have-different-memorycpu-sizes-does-mem_limit-or-cpu_core_limit-affect-results") Memory limit is hard and may cause failures on BEs with less memory size first. CPU is soft and does not cause errors. ##### Are `big_query_` parameters applied per node or globally?[​](#are-big_query_-parameters-applied-per-node-or-globally "Direct link to are-big_query_-parameters-applied-per-node-or-globally") They apply per BE node. ##### How to configure resource groups for Broker Load?[​](#how-to-configure-resource-groups-for-broker-load "Direct link to How to configure resource groups for Broker Load?") Example classifier: `query_type="insert", user="alice"`. ##### `exec_mem_limit`-related questions[​](#exec_mem_limit-related-questions "Direct link to exec_mem_limit-related-questions") Q: How many instances will be generated for a query? A: It is unpredictable because different queries can generate different number of fragments. Q: How to check the number of instance? A: You cannot check the number of instance. Q: If a query consumes 128 GB memory in total and generates 60 instances, while `query_mem_limit=0` and `exec_mem_limit=2G`. Will the query fail? A: The query fails as long as any instance consumes more than 2 GB (`exec_mem_limit`) memory. ##### How to disable global query queue for one resource group only?[​](#how-to-disable-global-query-queue-for-one-resource-group-only "Direct link to How to disable global query queue for one resource group only?") 1. Enable resource group-level query queue: ```sql SET GLOBAL enable_show_all_variables = true; SET enable_group_level_query_queue = true; ``` 2. Disable Query Queue for the current session or at user level: ```sql -- Disable Query Queue for the current session SET enable_query_queue = false; -- Disable Query Queue at user level ALTER USER 'xxx' SET PROPERTIES ("session.enable_query_queue" = "false"); ``` ##### Can queries that doesn't match any resource group be forced to fail?[​](#can-queries-that-doesnt-match-any-resource-group-be-forced-to-fail "Direct link to Can queries that doesn't match any resource group be forced to fail?") No. They will always fall back to the default resource group `default_wg`. #### Query Queue[​](#query-queue "Direct link to Query Queue") ##### How is the query queue memory trigger calculated?[​](#how-is-the-query-queue-memory-trigger-calculated "Direct link to How is the query queue memory trigger calculated?") Query Queue trigger = BE available memory size \* `query_queue_mem_used_pct_limit`. ##### Which takes priority: resource-group concurrency or query-queue concurrency?[​](#which-takes-priority-resource-group-concurrency-or-query-queue-concurrency "Direct link to Which takes priority: resource-group concurrency or query-queue concurrency?") If `enable_group_level_query_queue` is set to: * `false`: global or group limit may trigger first. * `true`: both apply; the smaller limit triggers queueing. ##### If queue size or timeout is reached, will queries fail immediately?[​](#if-queue-size-or-timeout-is-reached-will-queries-fail-immediately "Direct link to If queue size or timeout is reached, will queries fail immediately?") * When `query_queue_max_queued_queries` is reached, the query immediately fails. * When `query_queue_concurrency_limit` is reached, the query waits in queue. ##### What is the difference between resource group limits and query queue limits?[​](#what-is-the-difference-between-resource-group-limits-and-query-queue-limits "Direct link to What is the difference between resource group limits and query queue limits?") Resource groups restrict resource usage per group per BE node. Query queue uses BE-level limits for all queries. `concurrency_limit` and `max_cpu_cores` both apply when resource group-level query queue is enabled. ##### What is the difference between `pipeline_dop`, `exec_mem_limit`, and resource group concurrency limits?[​](#what-is-the-difference-between-pipeline_dop-exec_mem_limit-and-resource-group-concurrency-limits "Direct link to what-is-the-difference-between-pipeline_dop-exec_mem_limit-and-resource-group-concurrency-limits") `pipeline_dop` controls in-query parallelism. Resource groups/query queue control cluster-wide concurrent queries. `query_mem_limit` controls per-query-per-BE memory. --- ### Shared-data This topic provides answers to some frequently asked questions about shared-data clusters. #### Why does table creation fail?[​](#why-does-table-creation-fail "Direct link to Why does table creation fail?") Check the BE log (`be.INFO`) to identify the exact cause. Common causes include: * Misconfigured object storage settings (for example, `aws_s3_path`, `endpoint`, `authentication`). * Object storage service instability or exceptions. * For `CREATE STORAGE VOLUME` or `ALTER STORAGE VOLUME`, StarRocks validates storage accessibility in shared-data mode when the FE configuration `enable_storage_volume_access_check` is enabled (enabled by default). If this check is disabled, validation is skipped. If validation fails, fix credentials, endpoint, or network access first, and then retry. Other errors: Error message: "Error 1064 (HY000): Unexpected exception: Failed to create shards. INVALID\_ARGUMENT: shard info cannot be empty" Cause: This often caused when automatic bucket inference is used while no CN or BE nodes are alive. This issues is fixed in v3.2. #### Why does table creation take too long?[​](#why-does-table-creation-take-too-long "Direct link to Why does table creation take too long?") Excessive bucket numbers (especially in partitioned tables) cause StarRocks to create many tablets. The system needs to write a tablet metadata file for each tablet in the object storage, whose high latency can drastically increase total creation time. You may consider: * Reducing the number of buckets. * Increasing the tablet creation thread pool size via the BE configuration `create_tablet_worker_count`. * Checking and troubleshooting high write latency in object storage. #### Why is data in object storage not cleaned up after dropping a table?[​](#why-is-data-in-object-storage-not-cleaned-up-after-dropping-a-table "Direct link to Why is data in object storage not cleaned up after dropping a table?") StarRocks supports two DROP TABLE modes: * `DROP TABLE xxx`: moves table metadata to FE recycle bin (data is not deleted). * `DROP TABLE xxx FORCE`: immediately deletes table metadata and data. If cleanup fails, check: * Whether `DROP TABLE xxx FORCE` was used. * Whether recycle bin retention parameters are set too high. Parameters include: * FE configuration `catalog_trash_expire_second` * BE configuration `trash_file_expire_time_sec` * FE logs for deletion errors (for example, RPC timeout). Increase RPC timeout if needed. #### How can I find the storage path of table data in object storage?[​](#how-can-i-find-the-storage-path-of-table-data-in-object-storage "Direct link to How can I find the storage path of table data in object storage?") Run the following command to get the storage path. ```sql SHOW PROC '/dbs/'; ``` Example: ```sql mysql> SHOW PROC '/dbs/load_benchmark'; +---------+-------------+----------+---------------------+--------------+--------+--------------+--------------------------+--------------+---------------+--------------------------------------------------------------------------------------------------------------+ | TableId | TableName | IndexNum | PartitionColumnName | PartitionNum | State | Type | LastConsistencyCheckTime | ReplicaCount | PartitionType | StoragePath | +---------+-------------+----------+---------------------+--------------+--------+--------------+--------------------------+--------------+---------------+--------------------------------------------------------------------------------------------------------------+ | 17152 | store_sales | 1 | NULL | 1 | NORMAL | CLOUD_NATIVE | NULL | 64 | UNPARTITIONED | s3://starrocks-common/xxxxxxxxx-xxxx_load_benchmark-1699408425544/5ce4ee2c-98ba-470c-afb3-8d0bf4795e48/17152 | +---------+-------------+----------+---------------------+--------------+--------+--------------+--------------------------+--------------+---------------+--------------------------------------------------------------------------------------------------------------+ 1 row in set (0.18 sec) ``` In versions earlier than v3.1.4, table data is scattered under a single directory. From v3.1.4 onwards, data is organized by partition. The same command displays the table root path, which now contains subdirectories named after Partition IDs, and each partition directory holds sub-directories `data/` (segment data files) and `meta/` (tablet metadata files). #### Why are queries slow in shared-data clusters?[​](#why-are-queries-slow-in-shared-data-clusters "Direct link to Why are queries slow in shared-data clusters?") Common causes include: * Cache miss. * Insufficient compaction, causing too many small segment files and thus excessive I/O. * Bad parallelism (for example, too few tablets). * Improper `datacache.partition_duration` settings, causing caching failures. You need to analyze the Query Profile first to identify the root cause. ##### Cache miss[​](#cache-miss "Direct link to Cache miss") In shared-data clusters, data is stored remotely, so Data Cache is crucial. If queries become unexpectedly slow, check Query Profile metrics such as `CompressedBytesReadRemote` and `IOTimeRemote`. Cache misses may be caused by: * Data Cache disabled during table creation. * Insufficient local cache space. * Tablet migration due to elastic scaling. * Improper `datacache.partition_duration` settings preventing caching. ##### Insufficient compaction[​](#insufficient-compaction "Direct link to Insufficient compaction") Without adequate compaction, many historical data versions remain, increasing the number of segment files accessed during queries. This increases I/O and slows down queries. You can diagnose insufficient compaction by: * Checking Compaction Score for relevant partitions. Compaction Score should remain below ~10. Excessively high Compaction Scores often indicate compaction failures. * Reviewing Query Profile metrics such as `SegmentsReadCount`. If Segment counts are high, compaction may be lagging or stuck. ##### Improper tablet settings[​](#improper-tablet-settings "Direct link to Improper tablet settings") Tablets distribute data across Compute Nodes. Poor bucketing or skewed bucket keys can cause queries to run on only a subset of nodes. Recommendations: * Choose bucket columns that ensure balanced distribution. * Set reasonable bucket numbers (formula: `total data size / (1–5 GB)`). ##### Improper `datacache.partition_duration` settings[​](#improper-datacachepartition_duration-settings "Direct link to improper-datacachepartition_duration-settings") If this value is set to too small, data from “cold” partitions may not be cached, causing repeated remote reads. In Query Profile, if `CompressedBytesReadRemote` or `IOCountRemote` is non-zero, this may be the reason. Tune `datacache.partition_duration` accordingly. #### Why do all queries under a warehouse time out with errors like “Timeout was reached” or “Deadline Exceeded”?[​](#why-do-all-queries-under-a-warehouse-time-out-with-errors-like-timeout-was-reached-or-deadline-exceeded "Direct link to Why do all queries under a warehouse time out with errors like “Timeout was reached” or “Deadline Exceeded”?") Check whether the Compute Nodes under the warehouse can access object storage endpoint. #### How to retrieve tablet metadata in shared-data clusters?[​](#how-to-retrieve-tablet-metadata-in-shared-data-clusters "Direct link to How to retrieve tablet metadata in shared-data clusters?") 1. Obtain visible version by running: ```sql SHOW PARTITIONS FROM ` ``` 2. Execute the following statement to retrieve tablet metadata: ```sql admin execute on 'System.print(StorageEngine.get_lake_tablet_metadata_json(, ))' ``` #### Why is loading slow during high-frequency ingestion?[​](#why-is-loading-slow-during-high-frequency-ingestion "Direct link to Why is loading slow during high-frequency ingestion?") StarRocks serializes transaction commit, so high ingestion rates may hit limits. Monitor the following aspects: * **Loading queue**: If loading queue is full, you can increase I/O worker threads. * **Publish Version latency**: High publish times cause ingestion delays. #### How does compaction work in shared-data clusters, and why does it get stuck?[​](#how-does-compaction-work-in-shared-data-clusters-and-why-does-it-get-stuck "Direct link to How does compaction work in shared-data clusters, and why does it get stuck?") Key behaviors include: * Compaction is scheduled by FE and executed by CN. * Each compaction produces a new version and goes through the process of Write, Commit, and Publish. * FE does not track runtime compaction tasks; stale tasks on BE may block new ones. To clean up stuck compaction tasks, follow these steps: 1. Check version information of the partition, and compare `CompactVersion` and `VisibleVersion`. ```sql SHOW PARTITIONS FROM ``` 2. Check compaction task status. ```sql SHOW PROC '/compactions' ``` ```sql SELECT * FROM information_schema.be_cloud_native_compactions WHERE TXN_ID = ``` 3. Cancel expired tasks. a. Disable compaction and migration. ```sql ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "0"); ADMIN SET FRONTEND CONFIG ('tablet_sched_disable_balance' = 'true'); ADMIN SHOW FRONTEND CONFIG LIKE 'lake_compaction_max_tasks'; ADMIN SHOW FRONTEND CONFIG LIKE 'tablet_sched_disable_balance'; ``` b. Restart all BE nodes c. Verify all compactions have failed. ```sql SHOW PROC '/compactions' ``` d. Re-enable compaction & migration. ```sql ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "-1"); ADMIN SET FRONTEND CONFIG ('tablet_sched_disable_balance' = 'false'); ADMIN SHOW FRONTEND CONFIG LIKE 'lake_compaction_max_tasks'; ADMIN SHOW FRONTEND CONFIG LIKE 'tablet_sched_disable_balance'; ``` #### Why is compaction slow in Kubernetes shared-data clusters?[​](#why-is-compaction-slow-in-kubernetes-shared-data-clusters "Direct link to Why is compaction slow in Kubernetes shared-data clusters?") If ingestion happens in one warehouse but compaction runs in another (for example, `default_warehouse`), compaction must pull data across warehouses with no cache, slowing it down. Solution: * Set BE configuration `lake_enable_vertical_compaction_fill_data_cache` to `true`. * Perform writes in the same warehouse as compaction. #### Why does storage usage appear inflated in shared-data clusters?[​](#why-does-storage-usage-appear-inflated-in-shared-data-clusters "Direct link to Why does storage usage appear inflated in shared-data clusters?") Storage usage on object storage includes all historical versions, while `SHOW DATA` outputs only reflect the latest version. However, if the difference is excessively large, it could be caused by the following issues: * Compaction may be lagging. * Vacuum tasks may be backlogged (check queue size). You may consider tuning compaction or vacuum thread pools if necessary. #### Why do big queries fail with “meta does not exist?[​](#why-do-big-queries-fail-with-meta-does-not-exist "Direct link to Why do big queries fail with “meta does not exist?") The query may be using a data version already compacted and vacuumed. To resolve this, you can increase file retention by modifying the FE configuration `lake_autovacuum_grace_period_minutes`, and then retry the query. #### What causes excessive small files in object storage?[​](#what-causes-excessive-small-files-in-object-storage "Direct link to What causes excessive small files in object storage?") Excessive number of small files may cause performance degradation. Common causes include: * Too many buckets caused by improper bucket settings. * High ingestion frequency causing very small individual loads. Compaction will eventually merge small files, but tuning bucket count and batching ingestion helps prevent performance degradation. --- ### SQL query This topic provides answers to some frequently asked questions about SQL. #### Does StarRocks support caching query results?[​](#does-starrocks-support-caching-query-results "Direct link to Does StarRocks support caching query results?") StarRocks does not directly cache final query results. From v2.5 onwards, StarRocks uses the Query Cache feature to save the intermediate results of first-stage aggregation in the cache. New queries that are semantically equivalent to previous queries can reuse the cached computation results to accelerate computations. Query cache uses BE memory. For more information, see [Query cache](https://docs.starrocks.io/docs/using_starrocks/caching/query_cache.md). #### When a `Null` is included in the calculation, the calculation results of functions are false except for the ISNULL() function[​](#when-a-null-is-included-in-the-calculation-the-calculation-results-of-functions-are-false-except-for-the-isnull-function "Direct link to when-a-null-is-included-in-the-calculation-the-calculation-results-of-functions-are-false-except-for-the-isnull-function") In standard SQL, every calculation that includes an operand with a `NULL` value returns a `NULL`. #### Does StarRocks support the DECODE function?[​](#does-starrocks-support-the-decode-function "Direct link to Does StarRocks support the DECODE function?") StarRocks does not support the DECODE function of the Oracle database. StarRocks is compatible with MySQL, so you can use the CASE WHEN statement. #### Can the latest data be queried immediately after data is loaded into the Primary Key table of StarRocks?[​](#can-the-latest-data-be-queried-immediately-after-data-is-loaded-into-the-primary-key-table-of-starrocks "Direct link to Can the latest data be queried immediately after data is loaded into the Primary Key table of StarRocks?") Yes. StarRocks merges data in a way that references Google Mesa. In StarRocks, a BE triggers the data merge and it has two kinds of compaction to merge data. If the data merge is not completed, it is finished during your query. Therefore, you can read the latest data after data loading. #### Do the utf8mb4 characters stored in StarRocks get truncated or appear garbled?[​](#do-the-utf8mb4-characters-stored-in-starrocks-get-truncated-or-appear-garbled "Direct link to Do the utf8mb4 characters stored in StarRocks get truncated or appear garbled?") No. #### This error "table's state is not normal" occurs when I run the `alter table` command[​](#this-error-tables-state-is-not-normal-occurs-when-i-run-the-alter-table-command "Direct link to this-error-tables-state-is-not-normal-occurs-when-i-run-the-alter-table-command") This error occurs because the previous alteration has not been completed. You can run the following code to check the status of the previous alteration: ```sql show tablet from lineitem where State="ALTER"; ``` The time spent on the alteration operation relates to the data volume. In general, the alteration can be completed in minutes. We recommend that you stop loading data into StarRocks while you are altering tables because data loading lowers the speed at which alteration completes. #### This error "get hive partition meta data failed: java.net.UnknownHostException:hadooptest" occurs when I query the external tables of Apache Hive[​](#this-error-get-hive-partition-meta-data-failed-javanetunknownhostexception-occurs-when-i-query-the-external-tables-of-apache-hive "Direct link to this-error-get-hive-partition-meta-data-failed-javanetunknownhostexception-occurs-when-i-query-the-external-tables-of-apache-hive") This error occurs when the metadata of Apache Hive partitions cannot be obtained. To solve this problem, copy **core-sit.xml** and **hdfs-site.xml** to the **fe.conf** file and the **be.conf** file. #### This error "planner use long time 3000 remaining task num 1" occurs when I query data[​](#this-error-planner-use-long-time-3000-remaining-task-num-1-occurs-when-i-query-data "Direct link to This error \"planner use long time 3000 remaining task num 1\" occurs when I query data") This error occurs usually due to a full garbage collection (full GC), which can be checked by using backend monitoring and the **fe.gc** log. To solve this problem, perform one of the following operations: * Allows SQL's client to access multiple frontends (FEs) simultaneously to spread the load. * Change the heap size of Java Virtual Machine (JVM) from 8 GB to 16 GB in the **fe.conf** file to increase memory and reduce the impact of full GC. #### When cardinality of column A is small, the query results of `select B from tbl order by A limit 10` vary each time[​](#when-cardinality-of-column-a-is-small-the-query-results-of-select-b-from-tbl-order-by-a-limit-10-vary-each-time "Direct link to when-cardinality-of-column-a-is-small-the-query-results-of-select-b-from-tbl-order-by-a-limit-10-vary-each-time") SQL can only guarantee that column A is ordered, and it cannot guarantee that the order of column B is the same for each query. MySQL can guarantee the order of column A and column B because it is a standalone database. StarRocks is a distributed database, of which data stored in the underlying table is in a sharding pattern. The data of column A is distributed across multiple machines, so the order of column B returned by multiple machines may be different for each query, resulting in inconsistent order of B each time. To solve this problem, change `select B from tbl order by A limit 10` to `select B from tbl order by A,B limit 10`. #### Why is there a large gap in column efficiency between SELECT \* and SELECT?[​](#why-is-there-a-large-gap-in-column-efficiency-between-select--and-select "Direct link to Why is there a large gap in column efficiency between SELECT * and SELECT?") To solve this problem, check the profile and see MERGE details: * Check whether the aggregation on the storage layer takes up too much time. * Check whether there are too many indicator columns. If so, aggregate hundreds of columns of millions of rows. ```plaintext MERGE: - aggr: 26s270ms - sort: 15s551ms ``` #### Does DELETE support nested functions?[​](#does-delete-support-nested-functions "Direct link to Does DELETE support nested functions?") Nested functions are not supported, such as `to_days(now())` in `DELETE from test_new WHERE to_days(now())-to_days(publish_time) >7;`. #### How to improve the usage efficiency of a database when there are hundreds of tables in it?[​](#how-to-improve-the-usage-efficiency-of-a-database-when-there-are-hundreds-of-tables-in-it "Direct link to How to improve the usage efficiency of a database when there are hundreds of tables in it?") To improve efficiency, add the `-A` parameter when you connect to MySQL's client server: `mysql -uroot -h127.0.0.1 -P8867 -A`. MySQL's client server does not pre-read database information. #### How to reduce the disk space occupied by the BE log and the FE log?[​](#how-to-reduce-the-disk-space-occupied-by-the-be-log-and-the-fe-log "Direct link to How to reduce the disk space occupied by the BE log and the FE log?") Adjust the log level and corresponding parameters. For more information, see [Parameter Configuration](https://docs.starrocks.io/docs/administration/management/BE_configuration.md). #### This error "table \*\*\* is colocate table, cannot change replicationNum" occurs when I modify the replication number[​](#this-error-table--is-colocate-table-cannot-change-replicationnum-occurs-when-i-modify-the-replication-number "Direct link to This error \"table *** is colocate table, cannot change replicationNum\" occurs when I modify the replication number") When you create colocated tables, you need to set the `group` property. Therefore, you cannot modify the replication number for a single table. You can perform the following steps to modify the replication number for all tables in a group: 1. Set `group_with` to `empty` for all tables in a group. 2. Set a proper `replication_num` for all tables in a group. 3. Set `group_with` back to its original value. #### Does setting VARCHAR to the maximum value affect storage?[​](#does-setting-varchar-to-the-maximum-value-affect-storage "Direct link to Does setting VARCHAR to the maximum value affect storage?") VARCHAR is a variable-length data type, which has a specified length that can be changed based on the actual data length. Specifying a different varchar length when you create a table has little impact on the query performance on the same data. #### This error "create partititon timeout" occurs when I truncate a table[​](#this-error-create-partititon-timeout-occurs-when-i-truncate-a-table "Direct link to This error \"create partititon timeout\" occurs when I truncate a table") To truncate a table, you need to create the corresponding partitions and then swap them. If there are a larger number of partitions that need to be created, this error occurs. In addition, if there are many data load tasks, the lock will be held for a long time during the compaction process. Therefore, the lock cannot be acquired when you create tables. If there are too many data load tasks, set `tablet_map_shard_size` to `512` in the **be.conf** file to reduce the lock contention. #### This error "Failed to specify server's Kerberos principal name" occurs when I access external tables of Apache Hive[​](#this-error-failed-to-specify-servers-kerberos-principal-name-occurs-when-i-access-external-tables-of-apache-hive "Direct link to This error \"Failed to specify server's Kerberos principal name\" occurs when I access external tables of Apache Hive") Add the following information to **hdfs-site.xml** in the **fe.conf** file and the **be.conf** file: ```html dfs.namenode.kerberos.principal.pattern * ``` #### Is "2021-10" a date format in StarRocks?[​](#is-2021-10-a-date-format-in-starrocks "Direct link to Is \"2021-10\" a date format in StarRocks?") No. #### Can "2021-10" be used as a partition field?[​](#can-2021-10-be-used-as-a-partition-field "Direct link to Can \"2021-10\" be used as a partition field?") No, use functions to change "2021-10" to "2021-10-01" and then use "2021-10-01" as a partition field. #### Where can I query the size of a StarRocks database or table?[​](#where-can-i-query-the-size-of-a-starrocks-database-or-table "Direct link to Where can I query the size of a StarRocks database or table?") You can use the [SHOW DATA](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATA.md) command. `SHOW DATA;` displays the data size and replicas of all tables in the current database. `SHOW DATA FROM .;` displays the data size, number of replicas, and number of rows in a specified table of a specified database. #### In StarRocks on ES, when creating an Elasticsearch external table, if the relevant string length is too long, exceeding 256, and Elasticsearch uses dynamic mapping, using a select statement will result in the inability to query that column[​](#in-starrocks-on-es-when-creating-an-elasticsearch-external-table-if-the-relevant-string-length-is-too-long-exceeding-256-and-elasticsearch-uses-dynamic-mapping-using-a-select-statement-will-result-in-the-inability-to-query-that-column "Direct link to In StarRocks on ES, when creating an Elasticsearch external table, if the relevant string length is too long, exceeding 256, and Elasticsearch uses dynamic mapping, using a select statement will result in the inability to query that column") In dynamic mapping, Elasticsearch's data type is ```json "k4": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } } ``` StarRocks uses the keyword data type to convert the query statement. Since the keyword length of the column exceeds 256, the column cannot be queried. Solution: Remove the field mapping ```json "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } ``` to use the text type instead. #### How to quickly count the size of StarRocks databases and tables and the disk resources they occupy?[​](#how-to-quickly-count-the-size-of-starrocks-databases-and-tables-and-the-disk-resources-they-occupy "Direct link to How to quickly count the size of StarRocks databases and tables and the disk resources they occupy?") You can use the [SHOW DATA](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATA.md) command to view the storage size of databases and tables. `SHOW DATA;` displays the data volume and replica count of all tables in the current database. `SHOW DATA FROM .;` displays the data volume, replica count, and row count of a specific table in a specified database. #### Why does using a function on a partition key slow down queries?[​](#why-does-using-a-function-on-a-partition-key-slow-down-queries "Direct link to Why does using a function on a partition key slow down queries?") Using a function on a partition key can lead to inaccurate partition pruning, thereby reducing query performance. #### Why doesn't the DELETE statement support nested functions?[​](#why-doesnt-the-delete-statement-support-nested-functions "Direct link to Why doesn't the DELETE statement support nested functions?") ```sql mysql > DELETE FROM starrocks.ods_sale_branch WHERE create_time >= concat(substr(202201,1,4),'01') and create_time <= concat(substr(202301,1,4),'12'); SQL Error [1064][42000]: Right expr of binary predicate should be value ``` BINARY predicates must be of the `column op literal` type and cannot be expressions. There are currently no plans to support expressions as comparison values. #### How to name columns with reserved keywords?[​](#how-to-name-columns-with-reserved-keywords "Direct link to How to name columns with reserved keywords?") Reserved keywords (e.g., `rank`) need to be escaped, such as using `` `rank` ``. #### How to stop an executing SQL?[​](#how-to-stop-an-executing-sql "Direct link to How to stop an executing SQL?") You can use `show processlist;` to view executing SQL and use `kill ;` to terminate the corresponding SQL. You can also view and manage through `SHOW PROC '/current_queries';`. #### How to clean up idle connections?[​](#how-to-clean-up-idle-connections "Direct link to How to clean up idle connections?") You can control the timeout for idle connections through the session variable `wait_timeout` (unit: seconds). MySQL automatically cleans up idle connections after about 8 hours by default. #### Are multiple SQL segments in UNION ALL executed in parallel?[​](#are-multiple-sql-segments-in-union-all-executed-in-parallel "Direct link to Are multiple SQL segments in UNION ALL executed in parallel?") Yes, they are executed in parallel. #### What should be done if a SQL causes BE to crash?[​](#what-should-be-done-if-a-sql-causes-be-to-crash "Direct link to What should be done if a SQL causes BE to crash?") 1. Based on the `be.out` error stack, find the `query_id` that caused the crash. 2. Find the corresponding SQL in `fe.audit.log` using the `query_id`. Please collect and send the following information to the support team: * `be.out` log * Run `pstack $be_pid > pstack.log` to execute SQL. * Core Dump file Steps to collect Core files: 1. Get the corresponding BE process: ```bash ps aux| grep be ``` 2. Set the Core file size limit to unlimited. ```bash prlimit -p $bePID --core=unlimited:unlimited ``` Verify if the size limit is unlimited. ```bash cat /proc/$bePID/limits ``` If it is not `0`, the system will generate a Core file in the root directory of the BE deployment when the process crashes. #### How to use Hints to control table join optimizer behavior?[​](#how-to-use-hints-to-control-table-join-optimizer-behavior "Direct link to How to use Hints to control table join optimizer behavior?") Supports `broadcast` and `shuffle` Hints. For example: * `select * from a join [broadcast] b on a.id = b.id;` * `select * from a join [shuffle] b on a.id = b.id;` #### How to increase SQL query concurrency?[​](#how-to-increase-sql-query-concurrency "Direct link to How to increase SQL query concurrency?") By adjusting the session variable `pipeline_dop`. #### How to check the execution progress of DDL?[​](#how-to-check-the-execution-progress-of-ddl "Direct link to How to check the execution progress of DDL?") * View all column modification tasks in the default database: ```sql SHOW ALTER TABLE COLUMN; ``` * View the most recent column modification task for a specific table: ```sql SHOW ALTER TABLE COLUMN WHERE TableName="table1" ORDER BY CreateTime DESC LIMIT 1; ``` #### Why does comparing floating-point numbers sometimes result in inconsistent query results?[​](#why-does-comparing-floating-point-numbers-sometimes-result-in-inconsistent-query-results "Direct link to Why does comparing floating-point numbers sometimes result in inconsistent query results?") Directly using floating-point numbers `=` for comparison can lead to instability due to errors. It is recommended to use range checks. #### Why does floating-point calculation result in errors?[​](#why-does-floating-point-calculation-result-in-errors "Direct link to Why does floating-point calculation result in errors?") FLOAT/DOUBLE types have precision errors in `avg`, `sum`, and other calculations, leading to potentially inconsistent query results. For high precision, use the DECIMAL type, but note that performance will decrease by 2-3 times. #### Why does ORDER BY in a subquery not take effect?[​](#why-does-order-by-in-a-subquery-not-take-effect "Direct link to Why does ORDER BY in a subquery not take effect?") In distributed execution, if ORDER BY is not specified in the outer layer of the subquery, global ordering cannot be guaranteed. This is expected behavior. #### Why is the result of row\_number() inconsistent across multiple executions?[​](#why-is-the-result-of-row_number-inconsistent-across-multiple-executions "Direct link to Why is the result of row_number() inconsistent across multiple executions?") If the ORDER BY field has duplicates (e.g., multiple rows with the same `createTime`), SQL standards do not guarantee stable sorting. It is recommended to include a unique field (e.g., `employee_id`) in the ORDER BY to ensure stability. #### What information is needed for SQL optimization or troubleshooting?[​](#what-information-is-needed-for-sql-optimization-or-troubleshooting "Direct link to What information is needed for SQL optimization or troubleshooting?") * `EXPLAIN COSTS ` (includes statistics) * `EXPLAIN VERBOSE ` (includes data types, nullable, optimization strategies) * Query Profile (viewable through the FE Web interface at `http://:` and navigating to the Queries Tab) * Query Dump (obtained via HTTP API) ```bash wget --user=${username} --password=${password} --post-file ${query_file} http://${fe_host}:${fe_http_port}/api/query_dump?db=${database} -O ${dump_file} ``` Query Dump includes the following information: * Query statement * Table schema referenced in the query * Session variables * Number of BEs * Statistics (Min, Max values) * Exception information (exception stack) #### How to check data skew?[​](#how-to-check-data-skew "Direct link to How to check data skew?") Use `ADMIN SHOW REPLICA DISTRIBUTION FROM
` to view the distribution of tablets. #### How to troubleshoot memory-related errors?[​](#how-to-troubleshoot-memory-related-errors "Direct link to How to troubleshoot memory-related errors?") There are three common scenarios: * **Single query memory limit exceeded:** * Error: `Mem usage has exceed the limit of single query, You can change the limit by set session variable exec_mem_limit.` * Solution: Adjust `exec_mem_limit` * **Query pool memory limit exceeded:** * Error: `Mem usage has exceed the limit of query pool` * Solution: Optimize the SQL. * **BE total memory limit exceeded:** * Error: `Mem usage has exceed the limit of BE` * Solution: Analyze memory usage. Memory analysis methods: ```bash curl -XGET -s http://BE_IP:BE_HTTP_PORT/metrics | grep "^starrocks_be_.*_mem_bytes\|^starrocks_be_tcmalloc_bytes_in_use" curl -XGET -s http://BE_IP:BE_HTTP_PORT/mem_tracker ``` *** #### What to do when encountering the error `StarRocks planner use long time xxx ms in logical phase`?[​](#what-to-do-when-encountering-the-error-starrocks-planner-use-long-time-xxx-ms-in-logical-phase "Direct link to what-to-do-when-encountering-the-error-starrocks-planner-use-long-time-xxx-ms-in-logical-phase") 1. Analyze `fe.gc.log` to check for Full GC occurrences. 2. If the SQL execution plan is complex, increase `new_planner_optimize_timeout` (unit: ms): ```sql set global new_planner_optimize_timeout = 6000; ``` #### How to troubleshoot Unknown Error?[​](#how-to-troubleshoot-unknown-error "Direct link to How to troubleshoot Unknown Error?") Try adjusting the following parameters one by one and then re-execute the SQL: ```sql set disable_join_reorder = true; set enable_global_runtime_filter = false; set enable_query_cache = false; set cbo_enable_low_cardinality_optimize = false; ``` Then collect EXPLAIN COSTS, EXPLAIN VERBOSE, PROFILE, and Query Dump, and provide them to the support team. #### What time zone does `select now()` return?[​](#what-time-zone-does-select-now-return "Direct link to what-time-zone-does-select-now-return") It returns the time zone specified by the `time_zone` system variable. FE/BE logs use the machine's local time zone. #### Why does SQL slow down under high concurrency even when resources are normal?[​](#why-does-sql-slow-down-under-high-concurrency-even-when-resources-are-normal "Direct link to Why does SQL slow down under high concurrency even when resources are normal?") The reason is high network or RPC latency. You can adjust the BE parameter `brpc_connection_type` to `pooled` and then restart BE. #### How to disable statistics collection?[​](#how-to-disable-statistics-collection "Direct link to How to disable statistics collection?") * Disable automatic collection: ```sql enable_statistic_collect = false; ``` * Disable import-triggered collection: ```sql enable_statistic_collect_on_first_load = false; ``` * For versions upgraded to v3.3 and above, manually set: ```sql set global analyze_mv = ""; ``` --- ### FAQ: Hadoop 3.4.3 Wildfly Native SSL Library Issue in StarRocks #### Background[​](#background "Direct link to Background") StarRocks upgraded its bundled Hadoop dependency from 3.4.2 to 3.4.3 ([PR #69503](https://github.com/StarRocks/starrocks/pull/69503)). Hadoop 3.4.3 includes [HADOOP-19719](https://issues.apache.org/jira/browse/HADOOP-19719), which upgrades the Wildfly OpenSSL bindings from `2.1.4.Final` to `2.2.5.Final` to add OpenSSL 3.0 compatibility. However, the new native libraries (`libwfssl.so`) were built against **GLIBC 2.34+**, which introduces compatibility issues on older Linux distributions. **Affected StarRocks versions:** 4.1.0+, 4.0.7+, 3.5.14+ ##### Typical Symptoms[​](#typical-symptoms "Direct link to Typical Symptoms") When this issue occurs, CN/BE nodes crash with a **SIGSEGV** (segmentation fault) during SSL context initialization. The FE reports errors like: ```text SQL Error [1064] [42000]: Access storage error. Error message: failed to get file schema: A error occurred: errorCode=2001 errorMessage:Channel inactive error! ``` From [GitHub Issue #70478](https://github.com/StarRocks/starrocks/issues/70478) — querying Parquet files on Azure Data Lake via `FILES()` function: ```text *** SIGSEGV (@0x0) received by PID (TID 0x...) *** @ 0x7b9b67093453 SSL_CTX_new_ex @ 0x7b9b6a09d95a Java_org_wildfly_openssl_SSLImpl_makeSSLContext0 @ 0x7b9a68544be1 (unknown) ``` The Wildfly JNI call `Java_org_wildfly_openssl_SSLImpl_makeSSLContext0` loads `libwfssl.so`, which is compiled against the OpenSSL 3.x ABI. Inside the `starrocks_be` process, however, OpenSSL symbol lookups from `libwfssl.so` are satisfied by the **OpenSSL 1.x symbols that are statically linked into the `starrocks_be` binary** (from StarRocks' thirdparty) before they can ever reach the system's OpenSSL 3.x shared libraries on disk. A 3.x call site (`SSL_CTX_new_ex`) dispatching into a 1.x implementation is what actually crashes the JVM. This is why merely installing OpenSSL 3.x or its development headers on the host does NOT fix the crash -- the BE's in-process OpenSSL 1.x always wins symbol resolution. *** #### Q1: What is the Wildfly OpenSSL native library?[​](#q1-what-is-the-wildfly-openssl-native-library "Direct link to Q1: What is the Wildfly OpenSSL native library?") Hadoop uses the [Wildfly OpenSSL](https://github.com/wildfly-security/wildfly-openssl) library to bind native OpenSSL to the Java JSSE (Java Secure Socket Extension) APIs. This allows Hadoop's S3A, Azure Blob (ABFS), and Azure Data Lake (ADL) connectors to use native OpenSSL for TLS/SSL connections instead of the JVM's built-in SSL implementation. When it works, this provides a performance benefit for high-throughput encrypted I/O. *** #### Q2: What went wrong in Hadoop 3.4.3?[​](#q2-what-went-wrong-in-hadoop-343 "Direct link to Q2: What went wrong in Hadoop 3.4.3?") Hadoop 3.4.3 upgraded Wildfly OpenSSL to `2.2.5.Final` to support OpenSSL 3.0 ([HADOOP-19719](https://issues.apache.org/jira/browse/HADOOP-19719)). The new native library (`libwfssl.so`) was built against **GLIBC 2.34+** and targets the **OpenSSL 3.x ABI**. Because StarRocks' `starrocks_be` binary still statically links OpenSSL 1.x from its thirdparty, this produces two distinct failure modes: | Platform | GLIBC Version | System OpenSSL | Failure Mode | | ------------------------- | ------------- | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **RHEL 8 / CentOS 8** | 2.28 | 1.1.1 | **Load-time failure**: `UnsatisfiedLinkError: GLIBC_2.34 not found` -- `libwfssl.so` cannot load at all. | | **Ubuntu 22.04** | 2.35 | 3.0.2 | **Runtime ABI collision**: `libwfssl.so` loads, but its OpenSSL symbol lookups resolve to the static OpenSSL 1.x embedded in `starrocks_be`, not the system's OpenSSL 3.x. Crashes in `SSL_CTX_new_ex`. Installing `libssl-dev` does NOT fix this. | | **RHEL 9 / Ubuntu 24.04** | 2.34+ | 3.0+ | Same runtime ABI collision as Ubuntu 22.04: the BE's in-process OpenSSL 1.x shadows the system OpenSSL 3.x. Installing `openssl-devel` does NOT fix this either. | On affected systems, the Wildfly native library crash can cause JVM instability (including `hs_err` crash logs on CN/BE nodes), failed SSL/TLS connections to cloud storage (AWS S3, Azure Blob, Azure Data Lake), and silent fallback to unencrypted or broken connections. *** #### Q3: How do I fix this?[​](#q3-how-do-i-fix-this "Direct link to Q3: How do I fix this?") There are multiple solutions depending on your environment and requirements. ##### Solution 1: Disable Wildfly Native SSL[​](#solution-1-disable-wildfly-native-ssl "Direct link to Solution 1: Disable Wildfly Native SSL") Force Hadoop to use the JVM's built-in SSL implementation instead of the native Wildfly/OpenSSL path. This is the **safest and most portable fix**. Add the following to your `core-site.xml` on **all CN/BE nodes**: ```xml fs.s3a.ssl.channel.mode default_jsse adl.ssl.channel.mode Default_JSSE fs.azure.ssl.channel.mode Default_JSSE ``` **Where to place `core-site.xml`:** * For StarRocks BE/CN: `$STARROCKS_HOME/conf/core-site.xml` * For Broker processes: `$BROKER_HOME/conf/core-site.xml` **Restart all affected services** after making the change. ##### Solution 2: Remove the `wildfly-openssl` JAR from the BE Package[​](#solution-2-remove-the-wildfly-openssl-jar-from-the-be-package "Direct link to solution-2-remove-the-wildfly-openssl-jar-from-the-be-package") The root cause is that `libwfssl.so` (shipped inside `wildfly-openssl-*.Final.jar`) is compiled for the OpenSSL 3.x ABI, while the `starrocks_be` binary still statically links OpenSSL 1.x from the StarRocks thirdparty. Inside the BE process, OpenSSL symbol lookups from `libwfssl.so` are resolved against the 1.x symbols baked into `starrocks_be` before they can reach any system OpenSSL 3.x shared library, and the 3.x-call-into-1.x-implementation dispatch crashes in `SSL_CTX_new_ex`. Installing `libssl-dev` on Ubuntu or `openssl-devel` on RHEL cannot change this: even if those packages upgrade the host's OpenSSL to 3.x, the BE's in-process OpenSSL 1.x still wins symbol resolution every time. Treat any older guidance that suggests `apt install libssl-dev` or `yum install openssl-devel` as ineffective for this crash. A reliable workaround is to delete the offending JAR so Hadoop cannot load the native library at all and falls back to JSSE automatically: ```bash rm -f $STARROCKS_HOME/lib/hadoop/common/wildfly-openssl-2.2.5.Final.jar ``` Run this on every CN/BE node (and on every Broker node if a Broker is deployed) and restart the service. Starting from StarRocks builds that include the fix for [issue #71898](https://github.com/StarRocks/starrocks/issues/71898), the dependency is excluded at the Maven level in `java-extensions/pom.xml` (for `hadoop-common`, `hadoop-aws`, `hadoop-azure`, and `hadoop-azure-datalake`), so the JAR is never produced by the `java-extensions/hadoop-lib` assembly. `build.sh` additionally `rm -rf`s `${STARROCKS_OUTPUT}/be/lib/hadoop/common/wildfly-openssl-2.2.5.Final.jar` from the BE output as a belt-and-braces safety net. No manual cleanup is required after upgrading. *** #### Q4: Which cloud storage connectors are affected?[​](#q4-which-cloud-storage-connectors-are-affected "Direct link to Q4: Which cloud storage connectors are affected?") | Connector | Configuration Property | Recommended Value | | ----------------------------- | --------------------------- | ----------------- | | **AWS S3** (S3A) | `fs.s3a.ssl.channel.mode` | `default_jsse` | | **Azure Data Lake** (ADL) | `adl.ssl.channel.mode` | `Default_JSSE` | | **Azure Blob Storage** (ABFS) | `fs.azure.ssl.channel.mode` | `Default_JSSE` | note The Azure Data Lake Store SDK also supports `AdlStoreOptions.setSSLChannelMode()` programmatically, but for StarRocks the `core-site.xml` approach is the standard method. *** #### Q5: Is there a performance impact from disabling native SSL?[​](#q5-is-there-a-performance-impact-from-disabling-native-ssl "Direct link to Q5: Is there a performance impact from disabling native SSL?") Yes, but it is minor for most workloads. The native OpenSSL path (via Wildfly) can provide better throughput for large-scale encrypted I/O operations. However, the JVM's built-in JSSE implementation is well-optimized in modern JDKs (Java 11+) and is sufficient for most StarRocks data lake query patterns. If you are on a platform that supports the new Wildfly native library (RHEL 9 / Ubuntu 24.04 with GLIBC 2.34+ and OpenSSL 3.0), you can keep the `default` or `openssl` mode for maximum performance. *** #### Q6: How do I verify the fix is working?[​](#q6-how-do-i-verify-the-fix-is-working "Direct link to Q6: How do I verify the fix is working?") After applying the configuration change and restarting services: 1. **Re-run the failing workload** (e.g., query against external S3/Azure tables). 2. **Check BE/CN logs** for the absence of: * `hs_err_pid*.log` files (JVM crash logs) * `UnsatisfiedLinkError` mentioning `GLIBC_2.34` or `libwfssl.so` * `org.wildfly.openssl` error stack traces 3. **Confirm successful SSL connections** in the log: * With `default_jsse`: You should see standard JSSE handshake logs (no Wildfly references). * With `default`: You may see `Failed to load OpenSSL. Falling back to the JSSE` (this is expected and safe). *** #### Q7: Does this affect RHEL 8 / CentOS 8 specifically?[​](#q7-does-this-affect-rhel-8--centos-8-specifically "Direct link to Q7: Does this affect RHEL 8 / CentOS 8 specifically?") Yes. RHEL 8 ships with **GLIBC 2.28**, which is significantly older than the 2.34 required by the new Wildfly native library. On RHEL 8: * The native library **will not load at all** (`UnsatisfiedLinkError`). * The Hadoop 3.4.3 release notes explicitly state: *"as the native libraries were built with GLIBC 2.34+, these do not work on RHEL8. For deployment on those systems, stick with the JVM ssl support."* **Recommendation for RHEL 8:** Always use `default_jsse` mode. Plan migration to RHEL 9 for long-term support. *** #### Q8: What about FIPS-compliant environments?[​](#q8-what-about-fips-compliant-environments "Direct link to Q8: What about FIPS-compliant environments?") Linux distributions with a FIPS-compliant SSL library may not be compatible with the Wildfly native OpenSSL bindings. If you are running in a FIPS-compliant environment, **always use `default_jsse`** (JVM SSL) unless you have verified that the native library is compatible with your specific FIPS OpenSSL implementation. *** #### Q9: What has StarRocks done to mitigate this?[​](#q9-what-has-starrocks-done-to-mitigate-this "Direct link to Q9: What has StarRocks done to mitigate this?") 1. **Exclude `wildfly-openssl` at the Java-extensions build layer** ([issue #71898](https://github.com/StarRocks/starrocks/issues/71898)): recent StarRocks builds add `org.wildfly.openssl:wildfly-openssl` to the `hadoop-common`, `hadoop-aws`, `hadoop-azure`, and `hadoop-azure-datalake` dependencies in `java-extensions/pom.xml`, so `java-extensions/hadoop-lib` no longer ships the JAR to the BE at all. For extra safety, `build.sh` also `rm -rf`s any straggling `wildfly-openssl-2.2.5.Final.jar` from `${STARROCKS_OUTPUT}/be/lib/hadoop/common/`. The exclusion is intentionally temporary: it will be restored once StarRocks' thirdparty OpenSSL is upgraded to 3.x, at which point the in-process ABI will match what `libwfssl.so` expects. 2. **Documentation**: this FAQ and related guidance for configuring `core-site.xml`. > Note: an earlier attempt to install `libssl-dev` inside the Ubuntu runtime Docker image ([PR #70688](https://github.com/StarRocks/starrocks/pull/70688)) has been superseded. The crash is caused by `libwfssl.so`'s OpenSSL 3.x call sites dispatching into the **static OpenSSL 1.x embedded in the `starrocks_be` binary**, not by a missing host-side OpenSSL package -- so installing `libssl-dev` / `openssl-devel` cannot change the outcome. *** #### Quick Reference: Minimal Fix[​](#quick-reference-minimal-fix "Direct link to Quick Reference: Minimal Fix") Pick **either** of the following on **every CN/BE node** (and every Broker node, if deployed), then restart the service. Do NOT rely on installing `libssl-dev` or `openssl-devel` -- those packages do not fix this crash. **Option A -- delete the `wildfly-openssl` JAR:** ```bash rm -f $STARROCKS_HOME/lib/hadoop/common/wildfly-openssl-2.2.5.Final.jar ``` **Option B -- force JSSE via `$STARROCKS_HOME/conf/core-site.xml`:** ```xml fs.s3a.ssl.channel.mode default_jsse adl.ssl.channel.mode Default_JSSE fs.azure.ssl.channel.mode Default_JSSE ``` *** #### References[​](#references "Direct link to References") * [HADOOP-19719: Upgrade to wildfly version with support for openssl 3](https://issues.apache.org/jira/browse/HADOOP-19719) * [HADOOP-19262: Upgrade wildfly-openssl for JDK 17+](https://issues.apache.org/jira/browse/HADOOP-19262) * [FLINK-38284: Flink downstream tracking issue](https://issues.apache.org/jira/browse/FLINK-38284) * [Hadoop S3A Troubleshooting Guide](https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/troubleshooting_s3a.html) * [Hadoop S3A Performance Guide (SSL Channel Mode)](https://apache.github.io/hadoop/hadoop-aws/tools/hadoop-aws/performance.html) * [Wildfly OpenSSL GitHub Repository](https://github.com/wildfly-security/wildfly-openssl) * [StarRocks Issue #70478: SIGSEGV on Azure ADLS2 query (4.0.7)](https://github.com/StarRocks/starrocks/issues/70478) * [StarRocks PR #69503: Upgrade Hadoop 3.4.2 to 3.4.3](https://github.com/StarRocks/starrocks/pull/69503) * [StarRocks Issue #71898: Drop `wildfly-openssl` from the BE package](https://github.com/StarRocks/starrocks/issues/71898) * [StarRocks PR #70688: Install libssl-dev for Ubuntu runtime (superseded, does not fix the crash)](https://github.com/StarRocks/starrocks/pull/70688) --- ### table_design_faq #### Troubleshooting Table Design[​](#troubleshooting-table-design "Direct link to Troubleshooting Table Design") This topic provides answers to some frequently asked questions about table design. #### What is the maximum length of the VARCHAR type and how does the length affects query performance?[​](#what-is-the-maximum-length-of-the-varchar-type-and-how-does-the-length-affects-query-performance "Direct link to What is the maximum length of the VARCHAR type and how does the length affects query performance?") The maximum length of the VARCHAR type is 65533, which requires 1 MB storage size. It is recommended to set the VARCHAR length to a minimum value that is necessary. It is because, despite that the VARCHAR type data size is based on the actual length, in query scenarios where memory pre-allocation is required, memory resources are allocated based on the pre-defined length of the VARCHAR types instead of the actual length. For example, for an `address` field, 100 bytes is enough, which means that VARCHAR(100) is recommended over STRING, because the STRING type is equivalent to VARCHAR(65533). --- ## Flink_connector ### Releases of StarRocks Connector for Flink #### Notifications[​](#notifications "Direct link to Notifications") **User guide:** * [Load data into StarRocks using Flink connector](https://docs.starrocks.io/docs/loading/Flink-connector-starrocks/) * [Read data from StarRocks using Flink connector](https://docs.starrocks.io/docs/unloading/Flink_connector/) **Source codes:** [starrocks-connector-for-apache-flink](https://github.com/StarRocks/starrocks-connector-for-apache-flink) **Naming format of the JAR file:** * Flink 1.15 and later: `flink-connector-starrocks-${connector_version}_flink-${flink_version}.jar` * Prior to Flink 1.15: `flink-connector-starrocks-${connector_version}_flink-${flink_version}_${scala_version}.jar` **Methods to obtain the JAR file:** * Directly download the the Flink connector JAR file from the [Maven Central Repository](https://repo1.maven.org/maven2/com/starrocks). * Add the Flink connector as a dependency in your Maven project's `pom.xml` file and download it. For specific instructions, see [user guide](https://docs.starrocks.io/docs/loading/Flink-connector-starrocks/#obtain-flink-connector). * Compile the source codes into Flink connector JAR file. For specific instructions, see [user guide](https://docs.starrocks.io/docs/loading/Flink-connector-starrocks/#obtain-flink-connector). **Version requirements:** | Connector | Flink | StarRocks | Java | Scala | | --------- | ----------------------------- | ------------- | ---- | --------- | | 1.2.15 | 1.16,1.17,1.18,1.19,1.20 | 2.1 and later | 8 | 2.11,2.12 | | 1.2.14 | 1.16,1.17,1.18,1.19,1.20 | 2.1 and later | 8 | 2.11,2.12 | | 1.2.12 | 1.16,1.17,1.18,1.19,1.20 | 2.1 and later | 8 | 2.11,2.12 | | 1.2.11 | 1.15,1.16,1.17,1.18,1.19,1.20 | 2.1 and later | 8 | 2.11,2.12 | > **NOTICE** > > In general, the latest version of the Flink connector only maintains compatibility with the three most recent versions of Flink. #### Release notes[​](#release-notes "Direct link to Release notes") ##### 1.2[​](#12 "Direct link to 1.2") ###### 1.2.15[​](#1215 "Direct link to 1.2.15") Release date: June 18, 2026 ###### Features[​](#features "Direct link to Features") * Added multi-table transaction Stream Load support. [#487](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/487) ###### Improvements[​](#improvements "Direct link to Improvements") * Merge Commit supports logging data quality error messages. [#484](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/484) ###### BugFix[​](#bugfix "Direct link to BugFix") * Fix multi-table transaction concurrency: serialize per-table loads and align cross-table commits. [#491](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/491) * Fallback to FE cancel API when rollback fails for PREPARE-state lingering transactions. [#488](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/488) * Do not quote CURRENT\_TIMESTAMP in the DEFAULT clause when building column statements. [#486](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/486) ###### 1.2.14[​](#1214 "Direct link to 1.2.14") Release date: February 11, 2026 ###### Features[​](#features-1 "Direct link to Features") * Supports Merge Commit. [#474](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/474) ###### Improvements[​](#improvements-1 "Direct link to Improvements") * Supports setting `sink.buffer-flush.interval-ms` to lower than 1 second. [#475](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/475) * Supports configurable transaction Publish timeout. [#480](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/480) ###### Bug Fixes[​](#bug-fixes "Direct link to Bug Fixes") The following issues have been fixed: * Fix CVE-2023-2976 by upgrading guava version to `32.0.1-jre`. [#467](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/467) ###### 1.2.12[​](#1212 "Direct link to 1.2.12") Release date: September 19, 2025 ###### Improvements[​](#improvements-2 "Direct link to Improvements") * Supports specifying a warehouse for the source. [#423](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/423) * Added security policy. [#434](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/434) * Sanitized sensitive data in the error log. [#446](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/446) * Supports configuring `prepared_timeout` for the Stream Load transaction interface. [#453](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/453) ###### Bug Fixes[​](#bug-fixes-1 "Direct link to Bug Fixes") The following issues have been fixed: * The source reader was not closed if the open failed. [#441](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/441) * False success caused by any exception in StreamLoadManagerV2.flush. [#451](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/451) ###### 1.2.11[​](#1211 "Direct link to 1.2.11") Release data: June 3, 2025 **Features** * Supports LZ4 compression for the CSV format. [#408](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/408) * Adds support for Flink 1.20. [#409](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/409) **Improvements** * Adds an option to disable wrapping JSON into JSON arrays. [#344](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/344) * Updated FastJSON to resolve CVE-2022-25845. [#394](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/394) * Removed data row metrics from warn logs to avoid exposing payload in logs. [#420](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/420) **Bug Fixes** * Wrong pushdown results caused by the shadow clone of StarRocksDynamicTableSource (After the fix, a deep copy of StarRocksDynamicTableSource will be used). [#421](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/421) ###### 1.2.10[​](#1210 "Direct link to 1.2.10") **Features** * Supports reading JSON columns. [#334](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/334) * Supports reading ARRAY, STRUCT, and MAP columns. [#347](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/347) * Supports LZ4 compression when sinking data with the JSON format. [#354](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/354) * Supports Flink 1.19. [#379](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/379) **Improvements** * Supports configuring socket timeout. [#319](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/319) * The Stream Load transaction interface supports asynchronous `prepare` and `commit` operations. [#328](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/328) * Supports mapping a subset of columns in a StarRocks table to a Flink source table. [#352](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/352) * Supports setting a specific warehouse when using the Stream Load transaction interface. [#361](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/361) **Bug Fixes** Fixed the following issues: * `StarRocksSourceBeReader` in `StarRocksDynamicLookupFunction` is not closed after data reading completes. [#351](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/351) * An exception was thrown when loading an empty JSON string into a JSON column. [#380](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/380) ###### 1.2.9[​](#129 "Direct link to 1.2.9") This release includes some features and bug fixes. The notable change is that the Flink connector is integrated with Flink CDC 3.0 to easily build a streaming ELT pipeline from CDC sources (such as MySQL and Kafka) to StarRocks. You can see [Synchronize data with Flink CDC 3.0 (with schema change supported)](https://docs.starrocks.io/docs/loading/Flink-connector-starrocks/#synchronize-data-with-flink-cdc-30-with-schema-change-supported) for details. **Features** * Implement catalog to support Flink CDC 3.0. [#295](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/295) * Implement new sink API in [FLP-191](https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction) to support Flink CDC 3.0. [#301](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/301) * Support Flink 1.18. [#305](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/305) **Bug Fixes** * Fix misleading thread name and log. [#290](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/290) * Fix wrong stream-load-sdk configurations used for writing to multiple tables. [#298](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/298) ###### 1.2.8[​](#128 "Direct link to 1.2.8") This release includes some improvements and bug fixes. The notable changes are as follows: * Support Flink 1.16 and 1.17. * Recommend to set `sink.label-prefix` when the sink is configured to guarantee the exactly-once semantics. For the specific instructions, see [Exactly Once](https://docs.starrocks.io/docs/loading/Flink-connector-starrocks/#exactly-once). **Improvements** * Support to configure whether to use Stream Load transaction interface to guarantee at-least-once. [#228](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/228) * Add retry metrics for sink V1. [#229](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/229) * No need to getLabelState when EXISTING\_JOB\_STATUS is FINISHED. [#231](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/231) * Remove useless stack trace log for sink V1. [#232](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/232) * \[Refactor] Move StarRocksSinkManagerV2 to stream-load-sdk. [#233](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/233) * Automatically detect partial updates according to a Flink table's schema instead of the `sink.properties.columns` parameter explicitly specified by users. [#235](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/235) * \[Refactor] Move probeTransactionStreamLoad to stream-load-sdk. [#240](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/240) * Add git-commit-id-plugin for stream-load-sdk. [#242](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/242) * Use info log for DefaultStreamLoader#close. [#243](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/243) * Support to generate stream-load-sdk JAR file without dependencies. [#245](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/245) * Replace fastjson with jackson in stream-load-sdk. [#247](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/247) * Support to process update\_before record. [#250](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/250) * Add the Apache license into files. [#251](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/251) * Support to get the exception in stream-load-sdk. [#252](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/252) * Enable `strip_outer_array` and `ignore_json_size` by default. [#259](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/259) * Try to cleanup lingering transactions when a Flink job restores and the sink semantics is exactly-once. [#271](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/271) * Return the first exception after the retrying fails. [#279](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/279) **Bug Fixes** * Fix typos in StarRocksStreamLoadVisitor. [#230](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/230) * Fix the fastjson classloader leak. [#260](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/260) **Tests** * Add the test framework for loading from Kafka to StarRocks. [#249](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/249) **Doc** * Refactor the docs. [#262](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/262) * Improve the doc for the sink. [#268](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/268) [#275](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/275) * Add examples of DataStream API for the sink. [#253](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/253) --- ## Integrations ### Apache Airflow Enables orchestration and scheduling of data workflows with StarRocks using DAGs (Directed Acyclic Graphs) and SQL operators. Use Airflow for data loading and transformation using the `SQLExecuteQueryOperator` and `MySQLHook` without any implementation or complex configuration. [Apache Airflow GitHub repo](https://github.com/apache/airflow). #### Supported features[​](#supported-features "Direct link to Supported features") * SQL Execution through MySQL protocol * Connection management * Transaction support * Parameterized queries * Task dependencies * Retry logic #### Installation[​](#installation "Direct link to Installation") ##### Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Apache Airflow 2.0+ or 3.0+ * Python 3.8+ * Access to a StarRocks cluster (see the [quickstart guide](https://docs.starrocks.io/docs/quick_start/)) ##### Install[​](#install "Direct link to Install") The MySQL provider package is required to use StarRocks as StarRocks uses MySQL protocol. ```sh pip install apache-airflow-providers-mysql ``` Verify the installation by checking the installed providers: ```sh airflow providers list ``` This should list `apache-airflow-providers-mysql` in the output. #### Configuration[​](#configuration "Direct link to Configuration") ##### Create a StarRocks Connection[​](#create-a-starrocks-connection "Direct link to Create a StarRocks Connection") Create a StarRocks connection in the Airflow UI or via environment variable. The name of the connection will be used by the DAGs later. ###### Via Airflow UI[​](#via-airflow-ui "Direct link to Via Airflow UI") 1. Navigate to Admin > Connections 2. Click the + button to add a new connection 3. Configure the connection: * Connection Id: `starrocks_default` * Connection Type: MySQL * Host: `your-starrocks-host.com` * Schema: `your_database` * Login: `your_username` * Password: `your_password` * Port: `9030` ###### Via Airflow CLI[​](#via-airflow-cli "Direct link to Via Airflow CLI") ```sh airflow connections add 'starrocks_default' \ --conn-type 'mysql' \ --conn-host 'your-starrocks-host.com' \ --conn-schema 'your_database' \ --conn-login 'your_username' \ --conn-password 'your_password' \ --conn-port 9030 ``` #### Usage Examples[​](#usage-examples "Direct link to Usage Examples") These examples demonstrate common patterns for integrating StarRocks with Airflow. Each example builds on core concepts while showcasing different approaches to data loading, transformation, and workflow orchestration. **What You'll Learn:** * **Data Loading**: Efficiently load data from CSV files and cloud storage into StarRocks * **Data Transformation**: Execute SQL queries and process results with Python * **Advanced Patterns**: Implement incremental loading, async operations, and query optimization * **Production Best Practices**: Handle errors gracefully and build resilient pipelines All examples use the crash data tables described in the [quickstart guide](https://docs.starrocks.io/docs/quick_start/shared-nothing.md). ##### Data Loading[​](#data-loading "Direct link to Data Loading") ###### Stream Data Loading[​](#stream-data-loading "Direct link to Stream Data Loading") Load large CSV files efficiently using StarRocks Stream Load API. Stream Load is the recommended approach for: * High-throughput data loading (supports parallel loads) * Loading data with column transformations and filtering Stream Load provides better performance than INSERT INTO VALUES statements for large datasets and includes built-in features like error tolerance. Note that this does require the CSV file is accessible on the Airflow worker's filesystem. ```py from airflow.sdk import dag, task from airflow.hooks.base import BaseHook from datetime import datetime import requests from requests.auth import HTTPBasicAuth from urllib.parse import urlparse class PreserveAuthSession(requests.Session): """ Custom session that preserves Authorization header across redirects. StarRocks FE may redirect Stream Load requests to BE nodes. """ def rebuild_auth(self, prepared_request, response): old = urlparse(response.request.url) new = urlparse(prepared_request.url) # Only preserve auth when redirecting to same hostname if old.hostname == new.hostname: prepared_request.headers["Authorization"] = response.request.headers.get("Authorization") @dag( dag_id="starrocks_stream_load_example", schedule=None, start_date=datetime(2024, 1, 1), catchup=False, tags=["starrocks", "stream_load", "example"], ) def starrocks_stream_load_example(): @task def load_csv_to_starrocks(): # Configuration DATABASE = "quickstart" TABLE = "crashdata" CSV_PATH = "/path/to/crashdata.csv" conn = BaseHook.get_connection("starrocks_default") url = f"http://{conn.host}:{conn.port}/api/{DATABASE}/{TABLE}/_stream_load" # Generate unique label from airflow.sdk import get_current_context context = get_current_context() execution_date = context['logical_date'].strftime('%Y%m%d_%H%M%S') label = f"{TABLE}_load_{execution_date}" headers = { "label": label, "column_separator": ",", "skip_header": "1", "max_filter_ratio": "0.1", # Allow up to 10% error rate "Expect": "100-continue", "columns": """ tmp_CRASH_DATE, tmp_CRASH_TIME, CRASH_DATE=str_to_date(concat_ws(' ', tmp_CRASH_DATE, tmp_CRASH_TIME), '%m/%d/%Y %H:%i'), BOROUGH, ZIP_CODE, LATITUDE, LONGITUDE, LOCATION, ON_STREET_NAME, CROSS_STREET_NAME, OFF_STREET_NAME, NUMBER_OF_PERSONS_INJURED, NUMBER_OF_PERSONS_KILLED, NUMBER_OF_PEDESTRIANS_INJURED, NUMBER_OF_PEDESTRIANS_KILLED, NUMBER_OF_CYCLIST_INJURED, NUMBER_OF_CYCLIST_KILLED, NUMBER_OF_MOTORIST_INJURED, NUMBER_OF_MOTORIST_KILLED, CONTRIBUTING_FACTOR_VEHICLE_1, CONTRIBUTING_FACTOR_VEHICLE_2, CONTRIBUTING_FACTOR_VEHICLE_3, CONTRIBUTING_FACTOR_VEHICLE_4, CONTRIBUTING_FACTOR_VEHICLE_5, COLLISION_ID, VEHICLE_TYPE_CODE_1, VEHICLE_TYPE_CODE_2, VEHICLE_TYPE_CODE_3, VEHICLE_TYPE_CODE_4, VEHICLE_TYPE_CODE_5 """.replace("\n", "").replace(" ", ""), } session = PreserveAuthSession() with open(CSV_PATH, "rb") as f: response = session.put( url, headers=headers, data=f, auth=HTTPBasicAuth(conn.login, conn.password or ""), timeout=3600, ) result = response.json() print(f"\nStream Load Response:") print(f" Status: {result.get('Status')}") print(f" Loaded Rows: {result.get('NumberLoadedRows', 0):,}") if result.get("Status") == "Success": return result else: error_msg = result.get("Message", "Unknown error") raise Exception(f"Stream Load failed: {error_msg}") load_csv_to_starrocks() starrocks_stream_load_example() ``` ###### Insert From Files[​](#insert-from-files "Direct link to Insert From Files") Use StarRocks' [FILES()](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files/) table function to load data directly from files. This approach is ideal for: * Loading data from S3, HDFS, Google Cloud Storage * One-step data ingestion with transformations applied during load * Ad-hoc data loads from various file sources `FILES()` supports multiple file formats and storage systems, making it a flexible alternative to Stream Load for certain use cases. The data is read and inserted in a single SQL statement. ```py from airflow.sdk import dag, task from airflow.providers.mysql.hooks.mysql import MySqlHook from datetime import datetime FILE_PATH = "path_to_file_here" @dag( dag_id='crashdata_dynamic_files_load', schedule='@daily', start_date=datetime(2024, 1, 1), catchup=False, tags=['starrocks', 'files', 'dynamic'], ) def crashdata_files(): @task def load_file(file_path: str): hook = MySqlHook(mysql_conn_id='starrocks_default') sql = f""" INSERT INTO crashdata ( CRASH_DATE, BOROUGH, ZIP_CODE, LATITUDE, LONGITUDE, LOCATION, ON_STREET_NAME, CROSS_STREET_NAME, OFF_STREET_NAME, CONTRIBUTING_FACTOR_VEHICLE_1, CONTRIBUTING_FACTOR_VEHICLE_2, COLLISION_ID, VEHICLE_TYPE_CODE_1, VEHICLE_TYPE_CODE_2 ) SELECT STR_TO_DATE(CONCAT_WS(' ', `CRASH DATE`, `CRASH TIME`), '%m/%d/%Y %H:%i'), BOROUGH, `ZIP CODE`, CAST(LATITUDE as INT), CAST(LONGITUDE as INT), LOCATION, `ON STREET NAME`, `CROSS STREET NAME`, `OFF STREET NAME`, `CONTRIBUTING FACTOR VEHICLE 1`, `CONTRIBUTING FACTOR VEHICLE 2`, CAST(`COLLISION_ID` as INT), `VEHICLE TYPE CODE 1`, `VEHICLE TYPE CODE 2` FROM FILES( "path" = "s3://{file_path}", "format" = "parquet", "aws.s3.access_key" = "XXXXXXXXXX", "aws.s3.secret_key" = "YYYYYYYYYY", "aws.s3.region" = "us-west-2" ) """ result = hook.run(sql) return file_path load_file(FILE_PATH) crashdata_files() ``` ##### Data Transformation[​](#data-transformation "Direct link to Data Transformation") Execute SQL queries against StarRocks for table creation and data insertion. This is useful for: * Setting up database schema * Loading small datasets * Running ad-hoc queries ```py from airflow.sdk import dag, chain from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator from datetime import datetime @dag( dag_id='crashdata_basic_setup', schedule='@daily', start_date=datetime(2024, 1, 1), catchup=False, tags=['starrocks', 'crashdata'], ) def crashdata_basic_pipeline(): """Create crashdata table and insert sample NYC crash data.""" create_table = SQLExecuteQueryOperator( task_id='create_crashdata_table', conn_id='starrocks_default', sql=""" CREATE TABLE IF NOT EXISTS crashdata ( CRASH_DATE DATETIME, BOROUGH STRING, ZIP_CODE STRING, LATITUDE INT, LONGITUDE INT, LOCATION STRING, ON_STREET_NAME STRING, CROSS_STREET_NAME STRING, OFF_STREET_NAME STRING, CONTRIBUTING_FACTOR_VEHICLE_1 STRING, CONTRIBUTING_FACTOR_VEHICLE_2 STRING, COLLISION_ID INT, VEHICLE_TYPE_CODE_1 STRING, VEHICLE_TYPE_CODE_2 STRING ) DUPLICATE KEY(CRASH_DATE) DISTRIBUTED BY HASH(COLLISION_ID) BUCKETS 10 PROPERTIES ( "replication_num" = "1" ) """, ) insert_data = SQLExecuteQueryOperator( task_id='insert_sample_data', conn_id='starrocks_default', sql=""" INSERT INTO crashdata VALUES ('2024-01-15 08:30:00', 'MANHATTAN', '10001', 40748817, -73985428, '(40.748817, -73.985428)', '5 AVENUE', 'WEST 34 STREET', NULL, 'Driver Inattention/Distraction', 'Unspecified', 4567890, 'Sedan', 'Taxi'), ('2024-01-15 14:20:00', 'BROOKLYN', '11201', 40693139, -73987664, '(40.693139, -73.987664)', 'FLATBUSH AVENUE', 'ATLANTIC AVENUE', NULL, 'Failure to Yield Right-of-Way', 'Unspecified', 4567891, 'SUV', 'Sedan'), ('2024-01-15 18:45:00', 'QUEENS', '11354', 40767689, -73827426, '(40.767689, -73.827426)', 'NORTHERN BOULEVARD', 'MAIN STREET', NULL, 'Following Too Closely', 'Driver Inattention/Distraction', 4567892, 'Sedan', 'Sedan'), ('2024-01-16 09:15:00', 'BRONX', '10451', 40820679, -73925300, '(40.820679, -73.925300)', 'GRAND CONCOURSE', 'EAST 161 STREET', NULL, 'Unsafe Speed', 'Unspecified', 4567893, 'Truck', 'Sedan') """, ) create_table >> insert_data crashdata_basic_pipeline() ``` ###### More complex operations with MySqlHook[​](#more-complex-operations-with-mysqlhook "Direct link to More complex operations with MySqlHook") Use MySqlHook for advanced data analysis and processing within Python tasks. This approach is useful for: * Running analytical queries and processing results in Python * Combining StarRocks queries with Python libraries (pandas, numpy, etc.) * Implementing complex business logic that requires both SQL and Python * Creating data quality checks and validation workflows MySqlHook provides full programmatic access to query results, enabling sophisticated data transformations and analysis within your DAG. ```py from airflow.sdk import dag, task from airflow.providers.mysql.hooks.mysql import MySqlHook from datetime import datetime @dag( dag_id='crashdata_python_analysis', schedule='@daily', start_date=datetime(2024, 1, 1), catchup=False, tags=['starrocks', 'python', 'analytics'], ) def crashdata_python_pipeline(): @task def analyze_crash_hotspots(): """Identify crash hotspots by borough and street.""" hook = MySqlHook(mysql_conn_id='starrocks_default') # Query to find high-frequency crash locations sql = """ SELECT BOROUGH, ON_STREET_NAME, COUNT(*) as crash_count, COUNT(DISTINCT DATE(CRASH_DATE)) as days_with_crashes FROM crashdata WHERE ON_STREET_NAME IS NOT NULL GROUP BY BOROUGH, ON_STREET_NAME HAVING crash_count >= 3 ORDER BY crash_count DESC LIMIT 10 """ results = hook.get_records(sql) print("Top 10 Crash Hotspots:") for row in results: borough, street, count, days = row print(f"{borough:15} | {street:40} | {count:3} crashes over {days} days") return len(results) @task def calculate_contributing_factors(): """Calculate percentage distribution of contributing factors.""" hook = MySqlHook(mysql_conn_id='starrocks_default') sql = """ SELECT CONTRIBUTING_FACTOR_VEHICLE_1 as factor, COUNT(*) as count, ROUND(COUNT(*) * 100.0 / SUM(COUNT(*)) OVER (), 2) as percentage FROM crashdata WHERE CONTRIBUTING_FACTOR_VEHICLE_1 != 'Unspecified' GROUP BY CONTRIBUTING_FACTOR_VEHICLE_1 ORDER BY count DESC """ results = hook.get_records(sql) print("\nContributing Factors Analysis:") for factor, count, percentage in results: print(f"{factor:50} | {count:4} ({percentage}%)") return results # Define task execution order hotspots = analyze_crash_hotspots() factors = calculate_contributing_factors() hotspots >> factors crashdata_python_pipeline() ``` ##### Advanced Patterns[​](#advanced-patterns "Direct link to Advanced Patterns") ###### Incremental Data Loading[​](#incremental-data-loading "Direct link to Incremental Data Loading") Load data incrementally to avoid reprocessing existing records. Incremental loading is essential for: * Efficiently updating tables with new data only * Reducing processing time and resource usage * Managing large datasets that grow over time * Maintaining data freshness without full reloads This pattern uses staging tables and timestamp-based filtering to ensure only new records are loaded, making it ideal for scheduled batch updates. ```py from airflow.sdk import dag, chain from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator from datetime import datetime @dag( dag_id='crashdata_incremental_load', schedule='@hourly', start_date=datetime(2024, 1, 1), catchup=False, tags=['starrocks', 'incremental'], ) def crashdata_incremental_pipeline(): """Incrementally load new crash reports from staging.""" create_staging = SQLExecuteQueryOperator( task_id='create_staging_table', conn_id='starrocks_default', sql=""" CREATE TABLE IF NOT EXISTS crashdata_staging ( CRASH_DATE DATETIME, BOROUGH STRING, ZIP_CODE STRING, LATITUDE INT, LONGITUDE INT, LOCATION STRING, ON_STREET_NAME STRING, CROSS_STREET_NAME STRING, OFF_STREET_NAME STRING, CONTRIBUTING_FACTOR_VEHICLE_1 STRING, CONTRIBUTING_FACTOR_VEHICLE_2 STRING, COLLISION_ID INT, VEHICLE_TYPE_CODE_1 STRING, VEHICLE_TYPE_CODE_2 STRING, loaded_at DATETIME ) DUPLICATE KEY(CRASH_DATE) DISTRIBUTED BY HASH(COLLISION_ID) BUCKETS 10 PROPERTIES ("replication_num" = "1") """, ) incremental_load = SQLExecuteQueryOperator( task_id='load_new_crashes', conn_id='starrocks_default', sql=""" INSERT INTO crashdata SELECT CRASH_DATE, BOROUGH, ZIP_CODE, LATITUDE, LONGITUDE, LOCATION, ON_STREET_NAME, CROSS_STREET_NAME, OFF_STREET_NAME, CONTRIBUTING_FACTOR_VEHICLE_1, CONTRIBUTING_FACTOR_VEHICLE_2, COLLISION_ID, VEHICLE_TYPE_CODE_1, VEHICLE_TYPE_CODE_2 FROM crashdata_staging WHERE loaded_at >= '{{ data_interval_start }}' AND loaded_at < '{{ data_interval_end }}' AND COLLISION_ID NOT IN (SELECT COLLISION_ID FROM crashdata) """, ) create_staging >> incremental_load crashdata_incremental_pipeline() ``` ###### Asynchronous large-scale jobs with `SUBMIT TASK`[​](#asynchronous-large-scale-jobs-with-submit-task "Direct link to asynchronous-large-scale-jobs-with-submit-task") Use `SUBMIT TASK` for long-running queries that shouldn't block the Airflow task. This pattern is beneficial for: * Complex analytical queries that take minutes or hours * Large-scale data transformations (table copies, aggregations) * Resource-intensive operations that might timeout in synchronous mode * Parallel execution of multiple heavy queries * Separating job submission from completion monitoring `SUBMIT TASK` allows Airflow to monitor long-running StarRocks operations without holding database connections open, improving resource efficiency and reliability. ```py from airflow.sdk import dag, chain, task from airflow.providers.mysql.hooks.mysql import MySqlHook from datetime import datetime import time @dag( dag_id='crashdata_submit_task', schedule='@daily', start_date=datetime(2024, 1, 1), catchup=False, tags=['starrocks', 'submit-task'], ) def crashdata_submit_task_pipeline(): """ Example of using SUBMIT TASK for long-running queries. Requires StarRocks 3.4+ for SUBMIT TASK support. """ @task def submit_long_running_query(): """Submit a long-running query as an async task.""" hook = MySqlHook(mysql_conn_id='starrocks_default') submit_sql = """ SUBMIT TASK backup_crashdata AS CREATE TABLE crash_data_backup AS SELECT * FROM crashdata """ conn = hook.get_conn() cursor = conn.cursor() cursor.execute(submit_sql) # Get the task name from result result = cursor.fetchone() task_name = result[0] if result else None cursor.close() conn.close() if task_name: print(f"Task submitted successfully: {task_name}") return task_name else: raise Exception("Failed to submit task") @task def monitor_task_completion(task_name: str): """Monitor the submitted task until completion.""" hook = MySqlHook(mysql_conn_id='starrocks_default') max_wait_time = 600 # 10 minutes poll_interval = 10 # Check every 10 seconds elapsed_time = 0 while elapsed_time < max_wait_time: conn = hook.get_conn() cursor = conn.cursor() # Check task status in information_schema check_sql = f""" SELECT STATE, ERROR_MESSAGE FROM information_schema.task_runs WHERE TASK_NAME = '{task_name}' """ cursor.execute(check_sql) result = cursor.fetchone() if result: state, error_msg = result print(f"[{elapsed_time}s] Task status: {state}") if state == 'SUCCESS': cursor.close() conn.close() return {'status': 'SUCCESS', 'task_name': task_name} elif state == 'FAILED': cursor.close() conn.close() raise Exception(f"Task failed: {error_msg}") cursor.close() conn.close() time.sleep(poll_interval) elapsed_time += poll_interval raise Exception(f"Task did not complete within {max_wait_time} seconds") @task def process_results(): """Process or verify the completed task results.""" print("Task completed successfully - results are now available") return "Processing complete" # Define task flow task_name = submit_long_running_query() monitor_result = monitor_task_completion(task_name) result = process_results() chain(task_name, monitor_result, result) crashdata_submit_task_pipeline() ``` Note that the task name is unique in StarRocks, so future runs may need a qualifier (such as uuid). ###### Materialized Views[​](#materialized-views "Direct link to Materialized Views") Create and manage materialized views for accelerated query performance. Materialized views are ideal for: * Pre-computing complex aggregations for dashboards * Accelerating frequently run analytical queries * Maintaining summary tables that update automatically * Reducing compute costs by avoiding repeated calculations * Serving real-time analytics from pre-aggregated data Materialized views in StarRocks refresh automatically or on-demand, keeping aggregated data fresh while dramatically improving query performance. ```py from airflow.sdk import dag, task from airflow.providers.mysql.hooks.mysql import MySqlHook from datetime import datetime, timedelta @dag( dag_id="starrocks_materialized_view_example", schedule="0 2 * * *", # Run daily at 2 AM start_date=datetime(2024, 1, 1), catchup=False, tags=["starrocks", "materialized_view", "example"], doc_md=__doc__, ) def starrocks_materialized_view_example(): @task def create_materialized_view(): hook = MySqlHook(mysql_conn_id="starrocks_conn") drop_sql = """ DROP MATERIALIZED VIEW IF EXISTS quickstart.mv_daily_crash_stats """ create_sql = """ CREATE MATERIALIZED VIEW IF NOT EXISTS quickstart.mv_daily_crash_stats DISTRIBUTED BY HASH(`crash_date`) REFRESH ASYNC AS SELECT DATE(CRASH_DATE) as crash_date, BOROUGH, COUNT(*) as total_crashes, COUNT(DISTINCT COLLISION_ID) as unique_collisions FROM quickstart.crashdata WHERE CRASH_DATE IS NOT NULL GROUP BY DATE(CRASH_DATE), BOROUGH """ hook.run(drop_sql) hook.run(create_sql) return "mv_daily_crash_stats" @task def refresh_materialized_view(mv_name: str): hook = MySqlHook(mysql_conn_id="starrocks_conn") refresh_sql = f"REFRESH MATERIALIZED VIEW quickstart.{mv_name}" hook.run(refresh_sql) return mv_name @task def check_materialized_view_status(mv_name: str): hook = MySqlHook(mysql_conn_id="starrocks_conn") # Get task name for the MV task_query = f""" SELECT TASK_NAME FROM information_schema.tasks WHERE `DATABASE` = 'quickstart' AND DEFINITION LIKE '%{mv_name}%' ORDER BY CREATE_TIME DESC LIMIT 1 """ task_name = hook.get_first(task_query)[0] # Get latest task run state state_query = f""" SELECT STATE FROM information_schema.task_runs WHERE TASK_NAME = '{task_name}' ORDER BY CREATE_TIME DESC LIMIT 1 """ state = hook.get_first(state_query)[0] print(f"MV: {mv_name} | Task: {task_name} | State: {state}") if state not in ('SUCCESS', 'RUNNING'): raise Exception(f"Materialized view refresh in unexpected state: {state}") return {'task_name': task_name, 'state': state} create = create_materialized_view() refresh = refresh_materialized_view(create) status = check_materialized_view_status(refresh) status starrocks_materialized_view_example() ``` ###### Error Handling[​](#error-handling "Direct link to Error Handling") Implement robust error handling for production reliability. Proper error handling is critical for: * Automatically recovering from transient failures (network issues, timeouts) * Preventing data pipeline disruptions from temporary problems * Providing visibility into failure patterns Airflow's built-in retry mechanisms handle most transient errors. ```py from airflow.sdk import dag from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator from datetime import datetime, timedelta @dag( dag_id='starrocks_with_retries', schedule='@daily', start_date=datetime(2024, 1, 1), catchup=False, default_args={ 'retries': 3, 'retry_delay': timedelta(minutes=5), 'retry_exponential_backoff': True, 'max_retry_delay': timedelta(hours=1), }, ) def starrocks_resilient_pipeline(): critical_query = SQLExecuteQueryOperator( task_id='critical_query', conn_id='starrocks_default', sql='SELECT * FROM important_table', ) starrocks_resilient_pipeline() ``` ##### Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") * Verify that port 9030 is accessible from within the Airflow instance * Test the connection (if enabled) from the Airflow UI * If using localhost, use 127.0.0.1 instead --- ### Authenticate to AWS resources StarRocks supports using three authentication methods to integrate with AWS resources: instance profile-based authentication, assumed role-based authentication, and IAM user-based authentication. This topic describes how to configure AWS credentials by using these authentication methods. #### Authentication methods[​](#authentication-methods "Direct link to Authentication methods") ##### Instance profile-based authentication[​](#instance-profile-based-authentication "Direct link to Instance profile-based authentication") The instance profile-based authentication method allows your StarRocks cluster to inherit the privileges specified in the instance profile of the EC2 instance on which the cluster runs. In theory, any cluster user who can log in to the cluster can perform permitted actions on your AWS resources according to the AWS IAM policies you have configured. The typical scenario for this use case is that you do not need any AWS resource access control between multiple cluster users in the cluster. This authentication method means no isolation is required within the same cluster. However, this authentication method still can be seen as a cluster-level safe access control solution, because whoever can log in to the cluster is controlled by the cluster administrator. ##### Assumed role-based authentication[​](#assumed-role-based-authentication "Direct link to Assumed role-based authentication") Unlike instance profile-based authentication, the assumed role-based authentication method supports assuming an AWS IAM role to gain access to your AWS resources. For more information, see [Assuming a role](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-sharing-logs.html). ##### IAM user-based authentication[​](#iam-user-based-authentication "Direct link to IAM user-based authentication") The IAM user-based authentication method supports using IAM user credentials to gain access to your AWS resources. For more information, see [IAM users](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users.html). #### Preparations[​](#preparations "Direct link to Preparations") First, find the IAM role associated with the EC2 instance on which your StarRocks cluster runs (that role is referred to as the EC2 instance role hereinafter in this topic), and obtain the role's ARN. You will need the EC2 instance role for instance profile-based authentication and need the EC2 instance role and its ARN for assumed role-based authentication. Next step, create an IAM policy based on the type of AWS resource you want to access and the specific operation scenario within StarRocks. A policy in AWS IAM declares a set of permissions on a specific AWS resource. After creating a policy, you need to attach it to an IAM role or user. As such, the IAM role or user is assigned the permissions declared in the policy to access the specified AWS resource. > **NOTICE** > > To make these preparations, you must have permission to sign in to the [AWS IAM console](https://us-east-1.console.aws.amazon.com/iamv2/home#/home) and edit IAM users and roles. For the IAM policy you will need to access a specific AWS resource, see the following sections: * [Batch load data from AWS S3](https://docs.starrocks.io/docs/sql-reference/aws_iam_policies.md#batch-load-data-from-aws-s3) * [Read/write AWS S3](https://docs.starrocks.io/docs/sql-reference/aws_iam_policies.md#readwrite-aws-s3) * [Integrate with AWS Glue](https://docs.starrocks.io/docs/sql-reference/aws_iam_policies.md#integrate-with-aws-glue) ##### Preparation for instance profile-based authentication[​](#preparation-for-instance-profile-based-authentication "Direct link to Preparation for instance profile-based authentication") Attach the [IAM policies](https://docs.starrocks.io/docs/sql-reference/aws_iam_policies.md) for accessing the required AWS resources to the EC2 instance role. ##### Preparation for assumed role-based authentication[​](#preparation-for-assumed-role-based-authentication "Direct link to Preparation for assumed role-based authentication") ###### Create IAM roles and attach policies to them[​](#create-iam-roles-and-attach-policies-to-them "Direct link to Create IAM roles and attach policies to them") Create one or more IAM roles, depending on the AWS resources you want to access. See [Creating IAM roles](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create.html). Then, attach the [IAM policies](https://docs.starrocks.io/docs/sql-reference/aws_iam_policies.md) for accessing the required AWS resources to the IAM roles you create. For example, you want your StarRocks cluster to access AWS S3 and AWS Glue. In this situation, you can choose to create one IAM role (for example, `s3_assumed_role`), and attach the policy for accessing AWS S3 and the policy for accessing AWS Glue both to that role. Alternatively, you can choose to create two different IAM roles (for example, `s3_assumed_role` and `glue_assumed_role`), and attach these policies to the two different roles respectively (that is, attach the policy for accessing AWS S3 to `s3_assumed_role` and the policy for accessing AWS Glue to `glue_assumed_role`). The IAM roles you create will be assumed by the EC2 instance role of the StarRocks cluster to access the specified AWS resources. This section assumes that you have created only one assumed role, `s3_assumed_role`, and have added the policy for accessing AWS S3 and the policy for accessing AWS Glue both to that role. ###### Configure a trust relationship[​](#configure-a-trust-relationship "Direct link to Configure a trust relationship") Configure your assumed role as follows: 1. Sign in to the [AWS IAM console](https://us-east-1.console.aws.amazon.com/iamv2/home#/home). 2. In the left-side navigation pane, choose **Access management** > **Roles**. 3. Find the assumed role (`s3_assumed_role`) and click its name. 4. On the role's details page, click the **Trust relationships** tab, and on the **Trust relationships** tab click **Edit trust policy**. 5. On the **Edit trust policy** page, delete the existing JSON policy document, and paste the following IAM policy, in which you must replace `` with the EC2 instance role's ARN you have obtained above. Then, click **Update policy**. ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "" }, "Action": "sts:AssumeRole" } ] } ``` If you have created different assumed roles for accessing different AWS resources, you need to repeat the preceding steps to configure your other assumed roles. For example, you have created `s3_assumed_role` and `glue_assumed_role` for accessing AWS S3 and AWS Glue respectively. In this situation, you need to repeat the preceding steps to configure `glue_assumed_role`. Configure your EC2 instance role as follows: 1. Sign in to the [AWS IAM console](https://us-east-1.console.aws.amazon.com/iamv2/home#/home). 2. In the left-side navigation pane, choose **Access management** > **Roles**. 3. Find the EC2 instance role and click its name. 4. In the **Permissions policies** section of the role's details page, click **Add permissions** and choose **Create inline policy**. 5. In the **Specify permissions** step, click the **JSON** tab, delete the existing JSON policy document, and paste the following IAM policy, in which you must replace `` with the ARN of the assumed role `s3_assumed_role`. Then, click **Review policy**. ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["sts:AssumeRole"], "Resource": [ "" ] } ] } ``` If you have created different assumed roles for accessing different AWS resources, you need to fill the ARNs of all these assumed roles in the **Resource** element of the preceding IAM policy and separate them with a comma (,). For example, you have created`s3_assumed_role` and `glue_assumed_role` for accessing AWS S3 and AWS Glue respectively. In this situation, you need to fill the ARN of `s3_assumed_role` and the ARN of `glue_assumed_role` in the **Resource** element by using the following format: `"",""`. 6. In the **Review Policy** step, enter a policy name and click **Create policy**. ##### Preparation for IAM user-based authentication[​](#preparation-for-iam-user-based-authentication "Direct link to Preparation for IAM user-based authentication") Create an IAM user. See [Creating an IAM user in your AWS account](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html). Then, attach the [IAM policies](https://docs.starrocks.io/docs/sql-reference/aws_iam_policies.md) for accessing the required AWS resources to the IAM user you create. #### Comparison between authentication methods[​](#comparison-between-authentication-methods "Direct link to Comparison between authentication methods") The following figure provides a high-level explanation of the differences in mechanism between instance profile-based authentication, assumed role-based authentication, and IAM user-based authentication in StarRocks. ![Comparison between authentication methods](/assets/images/authenticate_s3_credential_methods-1f1020387bebbf0fdadc38a85205940d.png) #### Build connections with AWS resources[​](#build-connections-with-aws-resources "Direct link to Build connections with AWS resources") ##### Authentication parameters for accessing AWS S3[​](#authentication-parameters-for-accessing-aws-s3 "Direct link to Authentication parameters for accessing AWS S3") In various scenarios in which StarRocks needs to integrate with AWS S3, for example, when you create external catalogs or file external tables or when you ingest, back up, or restore data from AWS S3, configure the authentication parameters for accessing AWS S3 as follows: * For instance profile-based authentication, set `aws.s3.use_instance_profile` to `true`. * For assumed role-based authentication, set `aws.s3.use_instance_profile` to `true` and configure `aws.s3.iam_role_arn` as the assumed role's ARN that you use to access AWS S3 (for example, the ARN of the assumed role `s3_assumed_role` you have created above). * For IAM user-based authentication, set `aws.s3.use_instance_profile` to `false` and configure `aws.s3.access_key` and `aws.s3.secret_key` as the access key and secret key of your AWS IAM user. The following table describes the parameters. | Parameter | Required | Description | | ----------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | aws.s3.use\_instance\_profile | Yes | Specifies whether to enable the instance profile-based authentication method and the assumed role-based authentication method. Valid values: `true` and `false`. Default value: `false`. | | aws.s3.iam\_role\_arn | No | The ARN of the IAM role that has privileges on your AWS S3 bucket. If you use the assumed role-based authentication method to access AWS S3, you must specify this parameter. | | aws.s3.access\_key | No | The access key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter. | | aws.s3.secret\_key | No | The secret key of your IAM user. If you use the IAM user-based authentication method to access AWS S3, you must specify this parameter. | ##### Authentication parameters for accessing AWS Glue[​](#authentication-parameters-for-accessing-aws-glue "Direct link to Authentication parameters for accessing AWS Glue") In various scenarios in which StarRocks needs to integrate with AWS Glue, for example, when you create external catalogs, configure the authentication parameters for accessing AWS Glue as follows: * For instance profile-based authentication, set `aws.glue.use_instance_profile` to `true`. * For assumed role-based authentication, set `aws.glue.use_instance_profile` to `true` and configure `aws.glue.iam_role_arn` as the assumed role's ARN that you use to access AWS Glue (for example, the ARN of the assumed role `glue_assumed_role` you have created above). * For IAM user-based authentication, set `aws.glue.use_instance_profile` to `false` and configure `aws.glue.access_key` and `aws.glue.secret_key` as the access key and secret key of your AWS IAM user. The following table describes the parameters. | Parameter | Required | Description | | ------------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | aws.glue.use\_instance\_profile | Yes | Specifies whether to enable the instance profile-based authentication method and the assumed role-based authentication. Valid values: `true` and `false`. Default value: `false`. | | aws.glue.iam\_role\_arn | No | The ARN of the IAM role that has privileges on your AWS Glue Data Catalog. If you use the assumed role-based authentication method to access AWS Glue, you must specify this parameter. | | aws.glue.access\_key | No | The access key of your AWS IAM user. If you use the IAM user-based authentication method to access AWS Glue, you must specify this parameter. | | aws.glue.secret\_key | No | The secret key of your AWS IAM user. If you use the IAM user-based authentication method to access AWS Glue, you must specify this parameter. | #### Integration examples[​](#integration-examples "Direct link to Integration examples") ##### External catalog[​](#external-catalog "Direct link to External catalog") Creating an external catalog in your StarRocks cluster means building integration with the target data lake system, which is composed of two key components: * File storage like AWS S3 to store table files * Metastore like Hive metastore or AWS Glue to store the metadata and locations of table files StarRocks supports the following types of catalogs: * [Hive catalog](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md) * [Iceberg catalog](https://docs.starrocks.io/docs/data_source/catalog/iceberg/iceberg_catalog.md) * [Hudi catalog](https://docs.starrocks.io/docs/data_source/catalog/hudi_catalog.md) * [Delta Lake catalog](https://docs.starrocks.io/docs/data_source/catalog/deltalake_catalog.md) The following examples create a Hive catalog named `hive_catalog_hms` or `hive_catalog_glue`, depending on the type of metastore you use, to query data from your Hive cluster. For detailed syntax and parameters, see [Hive catalog](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md). ###### Instance profile-based authentication[​](#instance-profile-based-authentication-1 "Direct link to Instance profile-based authentication") * If you use Hive metastore in your Hive cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "us-west-2", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083" ); ``` * If you use AWS Glue in your Amazon EMR Hive cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_glue PROPERTIES ( "type" = "hive", "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "us-west-2", "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.region" = "us-west-2" ); ``` ###### Assumed role-based authentication[​](#assumed-role-based-authentication-1 "Direct link to Assumed role-based authentication") * If you use Hive metastore in your Hive cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::081976408565:role/s3_assumed_role", "aws.s3.region" = "us-west-2", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083" ); ``` * If you use AWS Glue in your Amazon EMR Hive cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_glue PROPERTIES ( "type" = "hive", "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::081976408565:role/s3_assumed_role", "aws.s3.region" = "us-west-2", "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "true", "aws.glue.iam_role_arn" = "arn:aws:iam::081976408565:role/glue_assumed_role", "aws.glue.region" = "us-west-2" ); ``` ###### IAM user-based authentication[​](#iam-user-based-authentication-1 "Direct link to IAM user-based authentication") * If you use Hive metastore in your Hive cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_hms PROPERTIES ( "type" = "hive", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "us-west-2", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083" ); ``` * If you use AWS Glue in your Amazon EMR Hive cluster, run a command like below: ```sql CREATE EXTERNAL CATALOG hive_catalog_glue PROPERTIES ( "type" = "hive", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "us-west-2", "hive.metastore.type" = "glue", "aws.glue.use_instance_profile" = "false", "aws.glue.access_key" = "", "aws.glue.secret_key" = "", "aws.glue.region" = "us-west-2" ); ``` ##### File external table[​](#file-external-table "Direct link to File external table") File external tables must be created in your internal catalog named `default_catalog`. The following examples create a file external table named `file_table` on an existing database named `test_s3_db`. For detailed syntax and parameters, see [File external table](https://docs.starrocks.io/docs/data_source/file_external_table.md). ###### Instance profile-based authentication[​](#instance-profile-based-authentication-2 "Direct link to Instance profile-based authentication") Run a command like below: ```sql CREATE EXTERNAL TABLE test_s3_db.file_table ( id varchar(65500), attributes map ) ENGINE=FILE PROPERTIES ( "path" = "s3://starrocks-test/", "format" = "ORC", "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "us-west-2" ); ``` ###### Assumed role-based authentication[​](#assumed-role-based-authentication-2 "Direct link to Assumed role-based authentication") Run a command like below: ```sql CREATE EXTERNAL TABLE test_s3_db.file_table ( id varchar(65500), attributes map ) ENGINE=FILE PROPERTIES ( "path" = "s3://starrocks-test/", "format" = "ORC", "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::081976408565:role/s3_assumed_role", "aws.s3.region" = "us-west-2" ); ``` ###### IAM user-based authentication[​](#iam-user-based-authentication-2 "Direct link to IAM user-based authentication") Run a command like below: ```sql CREATE EXTERNAL TABLE test_s3_db.file_table ( id varchar(65500), attributes map ) ENGINE=FILE PROPERTIES ( "path" = "s3://starrocks-test/", "format" = "ORC", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "us-west-2" ); ``` ##### Ingestion[​](#ingestion "Direct link to Ingestion") You can use LOAD LABEL to load data from AWS S3. The following examples load the data from all Parquet data files stored in the `s3a://test-bucket/test_brokerload_ingestion` path into the `test_ingestion_2` table in an existing database named `test_s3_db`. For detailed syntax and parameters, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). ###### Instance profile-based authentication[​](#instance-profile-based-authentication-3 "Direct link to Instance profile-based authentication") Run a command like below: ```sql LOAD LABEL test_s3_db.test_credential_instanceprofile_7 ( DATA INFILE("s3a://test-bucket/test_brokerload_ingestion/*") INTO TABLE test_ingestion_2 FORMAT AS "parquet" ) WITH BROKER ( "aws.s3.use_instance_profile" = "true", "aws.s3.region" = "us-west-1" ) PROPERTIES ( "timeout" = "1200" ); ``` ###### Assumed role-based authentication[​](#assumed-role-based-authentication-3 "Direct link to Assumed role-based authentication") Run a command like below: ```sql LOAD LABEL test_s3_db.test_credential_instanceprofile_7 ( DATA INFILE("s3a://test-bucket/test_brokerload_ingestion/*") INTO TABLE test_ingestion_2 FORMAT AS "parquet" ) WITH BROKER ( "aws.s3.use_instance_profile" = "true", "aws.s3.iam_role_arn" = "arn:aws:iam::081976408565:role/s3_assumed_role", "aws.s3.region" = "us-west-1" ) PROPERTIES ( "timeout" = "1200" ); ``` ###### IAM user-based authentication[​](#iam-user-based-authentication-3 "Direct link to IAM user-based authentication") Run a command like below: ```sql LOAD LABEL test_s3_db.test_credential_instanceprofile_7 ( DATA INFILE("s3a://test-bucket/test_brokerload_ingestion/*") INTO TABLE test_ingestion_2 FORMAT AS "parquet" ) WITH BROKER ( "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "us-west-1" ) PROPERTIES ( "timeout" = "1200" ); ``` --- ### Authenticate to Microsoft Azure Storage From v3.0 onwards, StarRocks can integrate with Microsoft Azure Storage (Azure Blob Storage or Azure Data Lake Storage) in the following scenarios: * Batch load data from Azure Storage. * Back up data from and restore data to Azure Storage. * Query Parquet and ORC files in Azure Storage. * Query [Hive](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md), [Iceberg](https://docs.starrocks.io/docs/data_source/catalog/iceberg/iceberg_catalog.md), [Hudi](https://docs.starrocks.io/docs/data_source/catalog/hudi_catalog.md), and [Delta Lake](https://docs.starrocks.io/docs/data_source/catalog/deltalake_catalog.md) tables in Azure Storage. StarRocks supports the following types of Azure Storage accounts: * Azure Blob Storage * Azure Data Lake Storage Gen1 * Azure Data Lake Storage Gen2 In this topic, Hive catalog, file external table, and Broker Load are used as examples to show how StarRocks integrates with Azure Storage by using these types of Azure Storage accounts. For information about the parameters in the examples, see [Hive catalog](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md), [File external table](https://docs.starrocks.io/docs/data_source/file_external_table.md), and [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). #### Blob Storage[​](#blob-storage "Direct link to Blob Storage") StarRocks supports using one of the following authentication methods to access Blob Storage: * Shared Key * SAS Token > **NOTE** > > When you load data or directly query files from Blob Storage, you must use the wasb or wasbs protocol to access your data: > > * If your storage account allows access over HTTP, use the wasb protocol and write the file path as `wasb://@.blob.core.windows.net//`. > * If your storage account allows access over HTTPS, use the wasbs protocol and write the file path as `wasbs://@.blob.core.windows.net//`. ##### Shared Key[​](#shared-key "Direct link to Shared Key") ###### External catalog[​](#external-catalog "Direct link to External catalog") Configure `azure.blob.storage_account` and `azure.blob.shared_key` as follows in the [CREATE EXTERNAL CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/CREATE_EXTERNAL_CATALOG.md) statement: ```sql CREATE EXTERNAL CATALOG hive_catalog_azure PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.blob.storage_account" = "", "azure.blob.shared_key" = "" ); ``` ###### File external table[​](#file-external-table "Direct link to File external table") Configure `azure.blob.storage_account`, `azure.blob.shared_key`, and the file path (`path`) as follows in the [CREATE EXTERNAL TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) statement: ```sql CREATE EXTERNAL TABLE external_table_azure ( id varchar(65500), attributes map ) ENGINE=FILE PROPERTIES ( "path" = "wasb[s]://@.blob.core.windows.net//", "format" = "ORC", "azure.blob.storage_account" = "", "azure.blob.shared_key" = "" ); ``` ###### Broker Load[​](#broker-load "Direct link to Broker Load") Configure `azure.blob.storage_account`, `azure.blob.shared_key`, and the file path (`DATA INFILE`) as follows in the [LOAD LABEL](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) statement: ```sql LOAD LABEL test_db.label000 ( DATA INFILE("wasb[s]://@.blob.core.windows.net//") INTO TABLE test_ingestion_2 FORMAT AS "parquet" ) WITH BROKER ( "azure.blob.storage_account" = "", "azure.blob.shared_key" = "" ); ``` ##### SAS Token[​](#sas-token "Direct link to SAS Token") ###### External catalog[​](#external-catalog-1 "Direct link to External catalog") Configure `azure.blob.storage_account`, `azure.blob.container`, and `azure.blob.sas_token` as follows in the [CREATE EXTERNAL CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/CREATE_EXTERNAL_CATALOG.md) statement: ```sql CREATE EXTERNAL CATALOG hive_catalog_azure PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.blob.storage_account" = "", "azure.blob.container" = "", "azure.blob.sas_token" = "" ); ``` ###### File external table[​](#file-external-table-1 "Direct link to File external table") Configure `azure.blob.storage_account`, `azure.blob.container`, `azure.blob.sas_token`, and the file path (`path`) as follows in the [CREATE EXTERNAL TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) statement: ```sql CREATE EXTERNAL TABLE external_table_azure ( id varchar(65500), attributes map ) ENGINE=FILE PROPERTIES ( "path" = "wasb[s]://@.blob.core.windows.net//", "format" = "ORC", "azure.blob.storage_account" = "", "azure.blob.container" = "", "azure.blob.sas_token" = "" ); ``` ###### Broker load[​](#broker-load-1 "Direct link to Broker load") Configure `azure.blob.storage_account`, `azure.blob.container`, `azure.blob.sas_token`, and the file path (`DATA INFILE`) as follows in the [LOAD LABEL](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) statement: ```sql LOAD LABEL test_db.label000 ( DATA INFILE("wasb[s]://@.blob.core.windows.net//") INTO TABLE target_table FORMAT AS "parquet" ) WITH BROKER ( "azure.blob.storage_account" = "", "azure.blob.container" = "", "azure.blob.sas_token" = "" ); ``` #### Data Lake Storage Gen1[​](#data-lake-storage-gen1 "Direct link to Data Lake Storage Gen1") StarRocks supports using one of the following authentication methods to access Data Lake Storage Gen1: * Managed Service Identity * Service Principal > **NOTE** > > When you load data or query files from Data Lake Storage Gen1, you must use the adl protocol to access your data and write the file path as `adl://.azuredatalakestore.net//`. ##### Managed Service Identity[​](#managed-service-identity "Direct link to Managed Service Identity") ###### External catalog[​](#external-catalog-2 "Direct link to External catalog") Configure `azure.adls1.use_managed_service_identity` as follows in the [CREATE EXTERNAL CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/CREATE_EXTERNAL_CATALOG.md) statement: ```sql CREATE EXTERNAL CATALOG hive_catalog_azure PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls1.use_managed_service_identity" = "true" ); ``` ###### File external table[​](#file-external-table-2 "Direct link to File external table") Configure `azure.adls1.use_managed_service_identity` and the file path (`path`) as follows in the [CREATE EXTERNAL TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) statement: ```sql CREATE EXTERNAL TABLE external_table_azure ( id varchar(65500), attributes map ) ENGINE=FILE PROPERTIES ( "path" = "adl://.azuredatalakestore.net//", "format" = "ORC", "azure.adls1.use_managed_service_identity" = "true" ); ``` ###### Broker Load[​](#broker-load-2 "Direct link to Broker Load") Configure `azure.adls1.use_managed_service_identity` and the file path (`DATA INFILE`) as follows in the [LOAD LABEL](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) statement: ```sql LOAD LABEL test_db.label000 ( DATA INFILE("adl://.azuredatalakestore.net//") INTO TABLE target_table FORMAT AS "parquet" ) WITH BROKER ( "azure.adls1.use_managed_service_identity" = "true" ); ``` ##### Service Principal[​](#service-principal "Direct link to Service Principal") ###### External catalog[​](#external-catalog-3 "Direct link to External catalog") Configure `azure.adls1.oauth2_client_id`, `azure.adls1.oauth2_credential`, and `azure.adls1.oauth2_endpoint` as follows in the [CREATE EXTERNAL CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/CREATE_EXTERNAL_CATALOG.md) statement: ```sql CREATE EXTERNAL CATALOG hive_catalog_azure PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls1.oauth2_client_id" = "", "azure.adls1.oauth2_credential" = "", "azure.adls1.oauth2_endpoint" = "" ); ``` ###### File external table[​](#file-external-table-3 "Direct link to File external table") Configure `azure.adls1.oauth2_client_id`, `azure.adls1.oauth2_credential`, `azure.adls1.oauth2_endpoint`, and the file path (`path`) as follows in the [CREATE EXTERNAL TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) statement: ```sql CREATE EXTERNAL TABLE external_table_azure ( id varchar(65500), attributes map ) ENGINE=FILE PROPERTIES ( "path" = "adl://.azuredatalakestore.net//", "format" = "ORC", "azure.adls1.oauth2_client_id" = "", "azure.adls1.oauth2_credential" = "", "azure.adls1.oauth2_endpoint" = "" ); ``` ###### Broker Load[​](#broker-load-3 "Direct link to Broker Load") Configure `azure.adls1.oauth2_client_id`, `azure.adls1.oauth2_credential`, `azure.adls1.oauth2_endpoint`, and the file path (`DATA INFILE`) as follows in the [LOAD LABEL](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) statement: ```sql LOAD LABEL test_db.label000 ( DATA INFILE("adl://.azuredatalakestore.net//") INTO TABLE target_table FORMAT AS "parquet" ) WITH BROKER ( "azure.adls1.oauth2_client_id" = "", "azure.adls1.oauth2_credential" = "", "azure.adls1.oauth2_endpoint" = "" ); ``` #### Data Lake Storage Gen2[​](#data-lake-storage-gen2 "Direct link to Data Lake Storage Gen2") StarRocks supports using one of the following authentication methods to access Data Lake Storage Gen2: * Managed Identity * Shared Key * Service Principal > **NOTE** > > When you load data or query files from Data Lake Storage Gen2, you must use the abfs or abfss protocol to access your data: > > * If your storage account allows access over HTTP, use the abfs protocol and write the file path as `abfs://@.dfs.core.windows.net//`. > * If your storage account allows access over HTTPS, use the abfss protocol and write the file path as `abfss://@.dfs.core.windows.net//`. ##### Managed Identity[​](#managed-identity "Direct link to Managed Identity") Before you start, you need to make the following preparations: * Edit the virtual machines (VMs) on which your StarRocks cluster is deployed. * Add the managed identities to these VMs. * Make sure that the managed identities are associated with the role (**Storage Blob Data Reader**) authorized to read data in your storage account. ###### External catalog[​](#external-catalog-4 "Direct link to External catalog") Configure `azure.adls2.oauth2_use_managed_identity`, `azure.adls2.oauth2_tenant_id`, and `azure.adls2.oauth2_client_id` as follows in the [CREATE EXTERNAL CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/CREATE_EXTERNAL_CATALOG.md) statement: ```sql CREATE EXTERNAL CATALOG hive_catalog_azure PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` ###### File external table[​](#file-external-table-4 "Direct link to File external table") Configure `azure.adls2.oauth2_use_managed_identity`, `azure.adls2.oauth2_tenant_id`, `azure.adls2.oauth2_client_id`, and the file path (`path`) as follows in the [CREATE EXTERNAL TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) statement: ```sql CREATE EXTERNAL TABLE external_table_azure ( id varchar(65500), attributes map ) ENGINE=FILE PROPERTIES ( "path" = "abfs[s]://@.dfs.core.windows.net//", "format" = "ORC", "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` ###### Broker Load[​](#broker-load-4 "Direct link to Broker Load") Configure `azure.adls2.oauth2_use_managed_identity`, `azure.adls2.oauth2_tenant_id`, `azure.adls2.oauth2_client_id`, and the file path (`DATA INFILE`) as follows in the [LOAD LABEL](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) statement: ```sql LOAD LABEL test_db.label000 ( DATA INFILE("adfs[s]://@.dfs.core.windows.net//") INTO TABLE target_table FORMAT AS "parquet" ) WITH BROKER ( "azure.adls2.oauth2_use_managed_identity" = "true", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` ##### Shared Key[​](#shared-key-1 "Direct link to Shared Key") ###### External catalog[​](#external-catalog-5 "Direct link to External catalog") Configure `azure.adls2.storage_account` and `azure.adls2.shared_key` as follows in the [CREATE EXTERNAL CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/CREATE_EXTERNAL_CATALOG.md) statement: ```sql CREATE EXTERNAL CATALOG hive_catalog_azure PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.storage_account" = "", "azure.adls2.shared_key" = "" ); ``` ###### File external table[​](#file-external-table-5 "Direct link to File external table") Configure `azure.adls2.storage_account`, `azure.adls2.shared_key`, and the file path (`path`) as follows in the [CREATE EXTERNAL TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) statement: ```sql CREATE EXTERNAL TABLE external_table_azure ( id varchar(65500), attributes map ) ENGINE=FILE PROPERTIES ( "path" = "abfs[s]://@.dfs.core.windows.net//", "format" = "ORC", "azure.adls2.storage_account" = "", "azure.adls2.shared_key" = "" ); ``` ###### Broker Load[​](#broker-load-5 "Direct link to Broker Load") Configure `azure.adls2.storage_account`, `azure.adls2.shared_key`, and the file path (`DATA INFILE`) as follows in the [LOAD LABEL](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) statement: ```sql LOAD LABEL test_db.label000 ( DATA INFILE("adfs[s]://@.dfs.core.windows.net//") INTO TABLE target_table FORMAT AS "parquet" ) WITH BROKER ( "azure.adls2.storage_account" = "", "azure.adls2.shared_key" = "" ); ``` ##### Service Principal[​](#service-principal-1 "Direct link to Service Principal") Before you start, you need to create a service principal, create a role assignment to assign a role to the service principal, and then add the role assignment to your storage account. As such, you can make sure that this service principal can successfully access the data in your storage account. ###### External catalog[​](#external-catalog-6 "Direct link to External catalog") Configure `azure.adls2.oauth2_client_id`, `azure.adls2.oauth2_client_secret`, and `azure.adls2.oauth2_client_endpoint` as follows in the [CREATE EXTERNAL CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/CREATE_EXTERNAL_CATALOG.md) statement: ```sql CREATE EXTERNAL CATALOG hive_catalog_azure PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_client_secret" = "", "azure.adls2.oauth2_client_endpoint" = "" ); ``` ###### File external table[​](#file-external-table-6 "Direct link to File external table") Configure `azure.adls2.oauth2_client_id`, `azure.adls2.oauth2_client_secret`, `azure.adls2.oauth2_client_endpoint`, and the file path (`path`) as follows in the [CREATE EXTERNAL TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) statement: ```sql CREATE EXTERNAL TABLE external_table_azure ( id varchar(65500), attributes map ) ENGINE=FILE PROPERTIES ( "path" = "abfs[s]://@.dfs.core.windows.net//", "format" = "ORC", "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_client_secret" = "", "azure.adls2.oauth2_client_endpoint" = "" ); ``` ###### Broker Load[​](#broker-load-6 "Direct link to Broker Load") Configure `azure.adls2.oauth2_client_id`, `azure.adls2.oauth2_client_secret`, `azure.adls2.oauth2_client_endpoint`, and the file path (`DATA INFILE`) as follows in the [LOAD LABEL](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) statement: ```sql LOAD LABEL test_db.label000 ( DATA INFILE("adfs[s]://@.dfs.core.windows.net//") INTO TABLE target_table FORMAT AS "parquet" ) WITH BROKER ( "azure.adls2.oauth2_client_id" = "", "azure.adls2.oauth2_client_secret" = "", "azure.adls2.oauth2_client_endpoint" = "" ); ``` ##### Workload Identity[​](#workload-identity "Direct link to Workload Identity") From v3.5.10 onwards, StarRocks supports Azure Workload Identity as an authentication method when accessing Azure Data Lake Storage Gen2. This authentication method is designed for workloads running inside Azure (for example, AKS pods) where the compute identity is federated with an Azure AD application via a projected token file, avoiding the need to store long-lived credentials. To use Workload Identity authentication, the following Azure-side setup is required before proceeding: * Enable the Azure Workload Identity webhook on your AKS cluster (or equivalent). * Create a federated identity credential linking your Kubernetes service account to an Azure AD application or user-assigned managed identity. * Annotate the Kubernetes pod/service account so that the webhook injects the token projection. The token file will appear at the path configured in `azure.adls2.oauth2_token_file` (commonly `/var/run/secrets/azure/tokens/azure-identity-token`). * Grant the Azure AD identity the necessary RBAC role on the storage account (for example, Storage Blob Data Reader or Storage Blob Data Contributor). ###### External catalog[​](#external-catalog-7 "Direct link to External catalog") Configure `azure.adls2.oauth2_token_file`, `azure.adls2.oauth2_tenant_id`, and `azure.adls2.oauth2_client_id` as follows in the [CREATE EXTERNAL CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/CREATE_EXTERNAL_CATALOG.md) statement: ```sql CREATE EXTERNAL CATALOG hive_catalog_azure PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", "azure.adls2.oauth2_token_file" = "/var/run/secrets/azure/tokens/azure-identity-token", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` ###### File external table[​](#file-external-table-7 "Direct link to File external table") Configure `azure.adls2.oauth2_token_file`, `azure.adls2.oauth2_tenant_id`, `azure.adls2.oauth2_client_id`, and the file path (`path`) as follows in the [CREATE EXTERNAL TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) statement: ```sql CREATE EXTERNAL TABLE external_table_azure ( id varchar(65500), attributes map ) ENGINE=FILE PROPERTIES ( "path" = "abfs[s]://@.dfs.core.windows.net//", "format" = "ORC", "azure.adls2.oauth2_token_file" = "/var/run/secrets/azure/tokens/azure-identity-token", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` ###### Broker Load[​](#broker-load-7 "Direct link to Broker Load") Configure `azure.adls2.oauth2_token_file`, `azure.adls2.oauth2_tenant_id`, `azure.adls2.oauth2_client_id`, and the file path (`DATA INFILE`) as follows in the [LOAD LABEL](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) statement: ```sql LOAD LABEL test_db.label000 ( DATA INFILE("adfs[s]://@.dfs.core.windows.net//") INTO TABLE target_table FORMAT AS "parquet" ) WITH BROKER ( "azure.adls2.oauth2_token_file" = "/var/run/secrets/azure/tokens/azure-identity-token", "azure.adls2.oauth2_tenant_id" = "", "azure.adls2.oauth2_client_id" = "" ); ``` --- ### Authenticate to Google Cloud Storage #### Authentication methods[​](#authentication-methods "Direct link to Authentication methods") From v3.0 onwards, StarRocks supports using one of the following authentication methods to access Google Cloud Storage (GCS): * VM-based authentication Use the credential attached to Google Cloud Compute Engine to authenticate GCS. * Service account-based authentication Use a service account to authenticate GCS. * Impersonation-based authentication Make a service account or virtual machine (VM) instance impersonate another service account. #### Scenarios[​](#scenarios "Direct link to Scenarios") StarRocks can authenticate to GCS in the following scenarios: * Batch load data from GCS. * Back up data from and restore data to GCS. * Query Parquet and ORC files in GCS. * Query [Hive](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md), [Iceberg](https://docs.starrocks.io/docs/data_source/catalog/iceberg/iceberg_catalog.md), [Hudi](https://docs.starrocks.io/docs/data_source/catalog/hudi_catalog.md), and [Delta Lake](https://docs.starrocks.io/docs/data_source/catalog/deltalake_catalog.md) tables in GCS. In this topic, [Hive catalog](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md), [file external table](https://docs.starrocks.io/docs/data_source/file_external_table.md), and [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) are used as examples to show how StarRocks integrates with GCS in different scenarios. For information about `StorageCredentialParams` in the examples, see the "[Parameters](https://docs.starrocks.io/docs/integrations/authenticate_to_gcs.md#parameters)" section of this topic. > **NOTE** > > StarRocks supports loading data or directly querying files from GCS only according to the gs protocol. Therefore, when you load data or query files from GCS, you must include `gs` as a prefix in the file path. ##### External catalog[​](#external-catalog "Direct link to External catalog") Use the [CREATE EXTERNAL CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/CREATE_EXTERNAL_CATALOG.md) statement to create a Hive catalog named `hive_catalog_gcs` as follows, in order to query files from GCS: ```sql CREATE EXTERNAL CATALOG hive_catalog_gcs PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:9083", StorageCredentialParams ); ``` ##### File external table[​](#file-external-table "Direct link to File external table") Use the [CREATE EXTERNAL TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md) statement to create a file external table named `external_table_gcs` as follows, in order to query a data file named `test_file_external_tbl` from GCS without any metastore: ```sql CREATE EXTERNAL TABLE external_table_gcs ( id varchar(65500), attributes map ) ENGINE=FILE PROPERTIES ( "path" = "gs:////test-gcs/test_file_external_tbl", "format" = "ORC", StorageCredentialParams ); ``` ##### Broker load[​](#broker-load "Direct link to Broker load") Use the [LOAD LABEL](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) statement to create a Broker Load job whose label is `test_db.label000`, in order to batch load data from GCS into the StarRocks table `target_table`: ```sql LOAD LABEL test_db.label000 ( DATA INFILE("gs://bucket_gcs/test_brokerload_ingestion/*") INTO TABLE target_table FORMAT AS "parquet" ) WITH BROKER ( StorageCredentialParams ); ``` #### Parameters[​](#parameters "Direct link to Parameters") `StorageCredentialParams` represents a parameter set that describes how to authenticate to GCS with different authentication methods. ##### VM-based authentication[​](#vm-based-authentication "Direct link to VM-based authentication") If your StarRocks cluster is deployed on a VM instance hosted on Google Cloud Platform (GCP) and you want to use that VM instance to authenticate GCS, configure `StorageCredentialParams` as follows: ```plain "gcp.gcs.use_compute_engine_service_account" = "true" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ---------------------------------------------- | ----------------- | --------------------- | ------------------------------------------------------------------------------------------- | | gcp.gcs.use\_compute\_engine\_service\_account | false | true | Specifies whether to directly use the service account that is bound to your Compute Engine. | ##### Service account-based authentication[​](#service-account-based-authentication "Direct link to Service account-based authentication") If you directly use a service account to authenticate GCS, configure `StorageCredentialParams` as follows: ```plain "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ------------------------------------------ | ----------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------- | | gcp.gcs.service\_account\_email | "" | "`user@hello.iam.gserviceaccount.com`" | The email address in the JSON file generated at the creation of the service account. | | gcp.gcs.service\_account\_private\_key\_id | "" | "61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea" | The private key ID in the JSON file generated at the creation of the service account. | | gcp.gcs.service\_account\_private\_key | "" | "-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n" | The private key in the JSON file generated at the creation of the service account. | ##### Impersonation-based authentication[​](#impersonation-based-authentication "Direct link to Impersonation-based authentication") ###### Make a VM instance impersonate a service account[​](#make-a-vm-instance-impersonate-a-service-account "Direct link to Make a VM instance impersonate a service account") If your StarRocks cluster is deployed on a VM instance hosted on GCP and you want to make that VM instance impersonate a service account, so as to make StarRocks inherit the privileges from the service account to access GCS, configure `StorageCredentialParams` as follows: ```plain "gcp.gcs.use_compute_engine_service_account" = "true", "gcp.gcs.impersonation_service_account" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ---------------------------------------------- | ----------------- | --------------------- | ------------------------------------------------------------------------------------------- | | gcp.gcs.use\_compute\_engine\_service\_account | false | true | Specifies whether to directly use the service account that is bound to your Compute Engine. | | gcp.gcs.impersonation\_service\_account | "" | "hello" | The service account that you want to impersonate. | ###### Make a service account impersonate another service account[​](#make-a-service-account-impersonate-another-service-account "Direct link to Make a service account impersonate another service account") If you want to make a service account (temporarily named as meta service account) impersonate another service account (temporarily named as data service account) and make StarRocks inherit the privileges from the data service account to access GCS, configure `StorageCredentialParams` as follows: ```plain "gcp.gcs.service_account_email" = "", "gcp.gcs.service_account_private_key_id" = "", "gcp.gcs.service_account_private_key" = "", "gcp.gcs.impersonation_service_account" = "" ``` The following table describes the parameters you need to configure in `StorageCredentialParams`. | **Parameter** | **Default value** | **Value** **example** | **Description** | | ------------------------------------------ | ----------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------ | | gcp.gcs.service\_account\_email | "" | "`user@hello.iam.gserviceaccount.com`" | The email address in the JSON file generated at the creation of the meta service account. | | gcp.gcs.service\_account\_private\_key\_id | "" | "61d257bd8479547cb3e04f0b9b6b9ca07af3b7ea" | The private key ID in the JSON file generated at the creation of the meta service account. | | gcp.gcs.service\_account\_private\_key | "" | "-----BEGIN PRIVATE KEY----xxxx-----END PRIVATE KEY-----\n" | The private key in the JSON file generated at the creation of the meta service account. | | gcp.gcs.impersonation\_service\_account | "" | "hello" | The data service account that you want to impersonate. | --- ### StarRocks Spark Connector ### Load data using Spark connector (recommended) StarRocks provides a self-developed connector named StarRocks Connector for Apache Spark™ (Spark connector for short) to help you load data into a StarRocks table by using Spark. The basic principle is to accumulate the data and then load it all at a time into StarRocks through [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). The Spark connector is implemented based on Spark DataSource V2. A DataSource can be created by using Spark DataFrames or Spark SQL. And both batch and structured streaming modes are supported. > **NOTICE** > > Only users with the SELECT and INSERT privileges on a StarRocks table can load data into this table. You can follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant these privileges to a user. #### Version requirements[​](#version-requirements "Direct link to Version requirements") | Spark connector | Spark | StarRocks | Java | Scala | | --------------- | ------------------ | ------------- | ---- | ----- | | 1.1.2 | 3.2, 3.3, 3.4, 3.5 | 2.5 and later | 8 | 2.12 | | 1.1.1 | 3.2, 3.3, or 3.4 | 2.5 and later | 8 | 2.12 | | 1.1.0 | 3.2, 3.3, or 3.4 | 2.5 and later | 8 | 2.12 | > **NOTICE** > > * Please see [Upgrade Spark connector](#upgrade-spark-connector) for behavior changes among different versions of the Spark connector. > * The Spark connector does not provide MySQL JDBC driver since version 1.1.1, and you need import the driver to the spark classpath manually. You can find the driver on [MySQL site](https://dev.mysql.com/downloads/connector/j/) or [Maven Central](https://repo1.maven.org/maven2/mysql/mysql-connector-java/). #### Obtain Spark connector[​](#obtain-spark-connector "Direct link to Obtain Spark connector") You can obtain the Spark connector JAR file in the following ways: * Directly download the compiled Spark Connector JAR file. * Add the Spark connector as a dependency in your Maven project and then download the JAR file. * Compile the source code of the Spark Connector into a JAR file by yourself. The naming format of the Spark connector JAR file is `starrocks-spark-connector-${spark_version}_${scala_version}-${connector_version}.jar`. For example, if you install Spark 3.2 and Scala 2.12 in your environment and you want to use Spark connector 1.1.0, you can use `starrocks-spark-connector-3.2_2.12-1.1.0.jar`. > **NOTICE** > > In general, the latest version of the Spark connector only maintains compatibility with the three most recent versions of Spark. ##### Download the compiled Jar file[​](#download-the-compiled-jar-file "Direct link to Download the compiled Jar file") Directly download the corresponding version of the Spark connector JAR from the [Maven Central Repository](https://repo1.maven.org/maven2/com/starrocks). ##### Maven Dependency[​](#maven-dependency "Direct link to Maven Dependency") 1. In your Maven project's `pom.xml` file, add the Spark connector as a dependency according to the following format. Replace `spark_version`, `scala_version`, and `connector_version` with the respective versions. ```xml com.starrocks starrocks-spark-connector-${spark_version}_${scala_version} ${connector_version} ``` 2. For example, if the version of Spark in your environment is 3.2, the version of Scala is 2.12, and you choose Spark connector 1.1.0, you need to add the following dependency: ```xml com.starrocks starrocks-spark-connector-3.2_2.12 1.1.0 ``` ##### Compile by yourself[​](#compile-by-yourself "Direct link to Compile by yourself") 1. Download the [Spark connector package](https://github.com/StarRocks/starrocks-connector-for-apache-spark). 2. Execute the following command to compile the source code of Spark connector into a JAR file. Note that `spark_version` is replaced with the corresponding Spark version. ```bash sh build.sh ``` For example, if the Spark version in your environment is 3.2, you need to execute the following command: ```bash sh build.sh 3.2 ``` 3. Go to the `target/` directory to find the Spark connector JAR file, such as `starrocks-spark-connector-3.2_2.12-1.1.0-SNAPSHOT.jar` , generated upon compilation. > **NOTE** > > The name of Spark connector which is not formally released contains the `SNAPSHOT` suffix. #### Parameters[​](#parameters "Direct link to Parameters") ##### starrocks.fe.http.url[​](#starrocksfehttpurl "Direct link to starrocks.fe.http.url") **Required**: YES
**Default value**: None
**Description**: The HTTP URL of the FE in your StarRocks cluster. You can specify multiple URLs, which must be separated by a comma (,). Format: `:,:`. Since version 1.1.1, you can also add `http://` prefix to the URL, such as `http://:,http://:`. ##### starrocks.fe.jdbc.url[​](#starrocksfejdbcurl "Direct link to starrocks.fe.jdbc.url") **Required**: YES
**Default value**: None
**Description**: The address that is used to connect to the MySQL server of the FE. Format: `jdbc:mysql://:`. ##### starrocks.table.identifier[​](#starrockstableidentifier "Direct link to starrocks.table.identifier") **Required**: YES
**Default value**: None
**Description**: The name of the StarRocks table. Format: `.`. ##### starrocks.user[​](#starrocksuser "Direct link to starrocks.user") **Required**: YES
**Default value**: None
**Description**: The username of your StarRocks cluster account. The user needs the [SELECT and INSERT privileges](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) on the StarRocks table. ##### starrocks.password[​](#starrockspassword "Direct link to starrocks.password") **Required**: YES
**Default value**: None
**Description**: The password of your StarRocks cluster account. ##### starrocks.write.label.prefix[​](#starrockswritelabelprefix "Direct link to starrocks.write.label.prefix") **Required**: NO
**Default value**: spark-
**Description**: The label prefix used by Stream Load. ##### starrocks.write.enable.transaction-stream-load[​](#starrockswriteenabletransaction-stream-load "Direct link to starrocks.write.enable.transaction-stream-load") **Required**: NO
**Default value**: TRUE
**Description**: Whether to use [Stream Load transaction interface](https://docs.starrocks.io/docs/loading/Stream_Load_transaction_interface.md) to load data. It requires StarRocks v2.5 or later. This feature can load more data in a transaction with less memory usage, and improve performance.
**NOTICE:** Since 1.1.1, this parameter takes effect only when the value of `starrocks.write.max.retries` is non-positive because Stream Load transaction interface does not support retry. ##### starrocks.write.buffer.size[​](#starrockswritebuffersize "Direct link to starrocks.write.buffer.size") **Required**: NO
**Default value**: 104857600
**Description**: The maximum size of data that can be accumulated in memory before being sent to StarRocks at a time. Setting this parameter to a larger value can improve loading performance but may increase loading latency. ##### starrocks.write.buffer.rows[​](#starrockswritebufferrows "Direct link to starrocks.write.buffer.rows") **Required**: NO
**Default value**: Integer.MAX\_VALUE
**Description**: Supported since version 1.1.1. The maximum number of rows that can be accumulated in memory before being sent to StarRocks at a time. ##### starrocks.write.flush.interval.ms[​](#starrockswriteflushintervalms "Direct link to starrocks.write.flush.interval.ms") **Required**: NO
**Default value**: 300000
**Description**: The interval at which data is sent to StarRocks. This parameter is used to control the loading latency. ##### starrocks.write.max.retries[​](#starrockswritemaxretries "Direct link to starrocks.write.max.retries") **Required**: NO
**Default value**: 3
**Description**: Supported since version 1.1.1. The number of times that the connector retries to perform the Stream Load for the same batch of data if the load fails.
**NOTICE:** Because Stream Load transaction interface does not support retry. If this parameter is positive, the connector always use Stream Load interface and ignore the value of `starrocks.write.enable.transaction-stream-load`. ##### starrocks.write.retry.interval.ms[​](#starrockswriteretryintervalms "Direct link to starrocks.write.retry.interval.ms") **Required**: NO
**Default value**: 10000
**Description**: Supported since version 1.1.1. The interval to retry the Stream Load for the same batch of data if the load fails. ##### starrocks.columns[​](#starrockscolumns "Direct link to starrocks.columns") **Required**: NO
**Default value**: None
**Description**: The StarRocks table column into which you want to load data. You can specify multiple columns, which must be separated by commas (,), for example, `"col0,col1,col2"`. ##### starrocks.column.types[​](#starrockscolumntypes "Direct link to starrocks.column.types") **Required**: NO
**Default value**: None
**Description**: Supported since version 1.1.1. Customize the column data types for Spark instead of using the defaults inferred from the StarRocks table and the [default mapping](#data-type-mapping-between-spark-and-starrocks). The parameter value is a schema in DDL format same as the output of Spark [StructType#toDDL](https://github.com/apache/spark/blob/master/sql/api/src/main/scala/org/apache/spark/sql/types/StructType.scala#L449) , such as `col0 INT, col1 STRING, col2 BIGINT`. Note that you only need to specify columns that need customization. One use case is to load data into columns of [BITMAP](#load-data-into-columns-of-bitmap-type) or [HLL](#load-data-into-columns-of-hll-type) type. ##### starrocks.write.properties.\*[​](#starrockswriteproperties "Direct link to starrocks.write.properties.*") **Required**: NO
**Default value**: None
**Description**: The parameters that are used to control Stream Load behavior. For example, the parameter `starrocks.write.properties.format` specifies the format of the data to be loaded, such as CSV or JSON. For a list of supported parameters and their descriptions, see [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). ##### starrocks.write.properties.format[​](#starrockswritepropertiesformat "Direct link to starrocks.write.properties.format") **Required**: NO
**Default value**: CSV
**Description**: The file format based on which the Spark connector transforms each batch of data before the data is sent to StarRocks. Valid values: CSV and JSON. ##### starrocks.write.properties.row\_delimiter[​](#starrockswritepropertiesrow_delimiter "Direct link to starrocks.write.properties.row_delimiter") **Required**: NO
**Default value**: \n
**Description**: The row delimiter for CSV-formatted data. ##### starrocks.write.properties.column\_separator[​](#starrockswritepropertiescolumn_separator "Direct link to starrocks.write.properties.column_separator") **Required**: NO
**Default value**: \t
**Description**: The column separator for CSV-formatted data. ##### starrocks.write.properties.partial\_update[​](#starrockswritepropertiespartial_update "Direct link to starrocks.write.properties.partial_update") **Required**: NO
**Default value**: `FALSE`
**Description**: Whether to use partial updates. Valid values: `TRUE` and `FALSE`. Default value: `FALSE`, indicating to disable this feature. ##### starrocks.write.properties.partial\_update\_mode[​](#starrockswritepropertiespartial_update_mode "Direct link to starrocks.write.properties.partial_update_mode") **Required**: NO
**Default value**: `row`
**Description**: Specifies the mode for partial updates. Valid values: `row` and `column`. * The value `row` (default) means partial updates in row mode, which is more suitable for real-time updates with many columns and small batches. * The value `column` means partial updates in column mode, which is more suitable for batch updates with few columns and many rows. In such scenarios, enabling the column mode offers faster update speeds. For example, in a table with 100 columns, if only 10 columns (10% of the total) are updated for all rows, the update speed of the column mode is 10 times faster. ##### starrocks.write.num.partitions[​](#starrockswritenumpartitions "Direct link to starrocks.write.num.partitions") **Required**: NO
**Default value**: None
**Description**: The number of partitions into which Spark can write data in parallel. When the data volume is small, you can reduce the number of partitions to lower the loading concurrency and frequency. The default value for this parameter is determined by Spark. However, this method may cause Spark Shuffle cost. ##### starrocks.write.partition.columns[​](#starrockswritepartitioncolumns "Direct link to starrocks.write.partition.columns") **Required**: NO
**Default value**: None
**Description**: The partitioning columns in Spark. The parameter takes effect only when `starrocks.write.num.partitions` is specified. If this parameter is not specified, all columns being written are used for partitioning. ##### starrocks.timezone[​](#starrockstimezone "Direct link to starrocks.timezone") **Required**: NO
**Default value**: Default timezone of JVM
**Description**: Supported since 1.1.1. The timezone used to convert Spark `TimestampType` to StarRocks `DATETIME`. The default is the timezone of JVM returned by `ZoneId#systemDefault()`. The format can be a timezone name such as `Asia/Shanghai`, or a zone offset such as `+08:00`. #### Data type mapping between Spark and StarRocks[​](#data-type-mapping-between-spark-and-starrocks "Direct link to Data type mapping between Spark and StarRocks") * The default data type mapping is as follows: | Spark data type | StarRocks data type | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | BooleanType | BOOLEAN | | ByteType | TINYINT | | ShortType | SMALLINT | | IntegerType | INT | | LongType | BIGINT | | StringType | LARGEINT | | FloatType | FLOAT | | DoubleType | DOUBLE | | DecimalType | DECIMAL | | StringType | CHAR | | StringType | VARCHAR | | StringType | STRING | | StringType | JSON | | DateType | DATE | | TimestampType | DATETIME | | ArrayType | ARRAY
**NOTE:**
**Supported since version 1.1.1**. For detailed steps, see [Load data into columns of ARRAY type](#load-data-into-columns-of-array-type). | * You can also customize the data type mapping. For example, a StarRocks table contains BITMAP and HLL columns, but Spark does not support the two data types. You need to customize the corresponding data types in Spark. For detailed steps, see load data into [BITMAP](#load-data-into-columns-of-bitmap-type) and [HLL](#load-data-into-columns-of-hll-type) columns. **BITMAP and HLL are supported since version 1.1.1**. #### Upgrade Spark connector[​](#upgrade-spark-connector "Direct link to Upgrade Spark connector") ##### Upgrade from version 1.1.0 to 1.1.1[​](#upgrade-from-version-110-to-111 "Direct link to Upgrade from version 1.1.0 to 1.1.1") * Since 1.1.1, the Spark connector does not provide `mysql-connector-java` which is the official JDBC driver for MySQL, because of the limitations of the GPL license used by `mysql-connector-java`. However, the Spark connector still needs the MySQL JDBC driver to connect to StarRocks for the table metadata, so you need to add the driver to the Spark classpath manually. You can find the driver on [MySQL site](https://dev.mysql.com/downloads/connector/j/) or [Maven Central](https://repo1.maven.org/maven2/mysql/mysql-connector-java/). * Since 1.1.1, the connector uses Stream Load interface by default rather than Stream Load transaction interface in version 1.1.0. If you still want to use Stream Load transaction interface, you can set the option `starrocks.write.max.retries` to `0`. Please see the description of `starrocks.write.enable.transaction-stream-load` and `starrocks.write.max.retries` for details. #### Examples[​](#examples "Direct link to Examples") The following examples show how to use the Spark connector to load data into a StarRocks table with Spark DataFrames or Spark SQL. The Spark DataFrames supports both Batch and Structured Streaming modes. For more examples, see [Spark Connector Examples](https://github.com/StarRocks/starrocks-connector-for-apache-spark/tree/main/src/test/java/com/starrocks/connector/spark/examples). ##### Preparations[​](#preparations "Direct link to Preparations") ###### Create a StarRocks table[​](#create-a-starrocks-table "Direct link to Create a StarRocks table") Create a database `test` and create a Primary Key table `score_board`. ```sql CREATE DATABASE `test`; CREATE TABLE `test`.`score_board` ( `id` int(11) NOT NULL COMMENT "", `name` varchar(65533) NULL DEFAULT "" COMMENT "", `score` int(11) NOT NULL DEFAULT "0" COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`id`) COMMENT "OLAP" DISTRIBUTED BY HASH(`id`); ``` ###### Network configuration[​](#network-configuration "Direct link to Network configuration") Ensure that the machine where Spark is located can access the FE nodes of the StarRocks cluster via the [`http_port`](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#http_port) (default: `8030`) and [`query_port`](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#query_port) (default: `9030`), and the BE nodes via the [`be_http_port`](https://docs.starrocks.io/docs/administration/management/BE_configuration.md#be_http_port) (default: `8040`). ###### Set up your Spark environment[​](#set-up-your-spark-environment "Direct link to Set up your Spark environment") Note that the following examples are run in Spark 3.2.4 and use `spark-shell`, `pyspark` and `spark-sql`. Before running the examples, make sure to place the Spark connector JAR file in the `$SPARK_HOME/jars` directory. ##### Load data with Spark DataFrames[​](#load-data-with-spark-dataframes "Direct link to Load data with Spark DataFrames") The following two examples explain how to load data with Spark DataFrames Batch or Structured Streaming mode. ###### Batch[​](#batch "Direct link to Batch") Construct data in memory and load data into the StarRocks table. 1. You can write the spark application using Scala or Python. For Scala, run the following code snippet in `spark-shell`: ```scala // 1. Create a DataFrame from a sequence. val data = Seq((1, "starrocks", 100), (2, "spark", 100)) val df = data.toDF("id", "name", "score") // 2. Write to StarRocks by configuring the format as "starrocks" and the following options. // You need to modify the options according your own environment. df.write.format("starrocks") .option("starrocks.fe.http.url", "127.0.0.1:8030") .option("starrocks.fe.jdbc.url", "jdbc:mysql://127.0.0.1:9030") .option("starrocks.table.identifier", "test.score_board") .option("starrocks.user", "root") .option("starrocks.password", "") .mode("append") .save() ``` For Python, run the following code snippet in `pyspark`: ```python from pyspark.sql import SparkSession spark = SparkSession \ .builder \ .appName("StarRocks Example") \ .getOrCreate() # 1. Create a DataFrame from a sequence. data = [(1, "starrocks", 100), (2, "spark", 100)] df = spark.sparkContext.parallelize(data) \ .toDF(["id", "name", "score"]) # 2. Write to StarRocks by configuring the format as "starrocks" and the following options. # You need to modify the options according your own environment. df.write.format("starrocks") \ .option("starrocks.fe.http.url", "127.0.0.1:8030") \ .option("starrocks.fe.jdbc.url", "jdbc:mysql://127.0.0.1:9030") \ .option("starrocks.table.identifier", "test.score_board") \ .option("starrocks.user", "root") \ .option("starrocks.password", "") \ .mode("append") \ .save() ``` 2. Query data in the StarRocks table. ```sql MySQL [test]> SELECT * FROM `score_board`; +------+-----------+-------+ | id | name | score | +------+-----------+-------+ | 1 | starrocks | 100 | | 2 | spark | 100 | +------+-----------+-------+ 2 rows in set (0.00 sec) ``` ###### Structured Streaming[​](#structured-streaming "Direct link to Structured Streaming") Construct a streaming read of data from a CSV file and load data into the StarRocks table. 1. In the directory `csv-data`, create a CSV file `test.csv` with the following data: ```csv 3,starrocks,100 4,spark,100 ``` 2. You can write the Spark application using Scala or Python. For Scala, run the following code snippet in `spark-shell`: ```scala import org.apache.spark.sql.types.StructType // 1. Create a DataFrame from CSV. val schema = (new StructType() .add("id", "integer") .add("name", "string") .add("score", "integer") ) val df = (spark.readStream .option("sep", ",") .schema(schema) .format("csv") // Replace it with your path to the directory "csv-data". .load("/path/to/csv-data") ) // 2. Write to StarRocks by configuring the format as "starrocks" and the following options. // You need to modify the options according your own environment. val query = (df.writeStream.format("starrocks") .option("starrocks.fe.http.url", "127.0.0.1:8030") .option("starrocks.fe.jdbc.url", "jdbc:mysql://127.0.0.1:9030") .option("starrocks.table.identifier", "test.score_board") .option("starrocks.user", "root") .option("starrocks.password", "") // replace it with your checkpoint directory .option("checkpointLocation", "/path/to/checkpoint") .outputMode("append") .start() ) ``` For Python, run the following code snippet in `pyspark`: ```python from pyspark.sql import SparkSession from pyspark.sql.types import IntegerType, StringType, StructType, StructField spark = SparkSession \ .builder \ .appName("StarRocks SS Example") \ .getOrCreate() # 1. Create a DataFrame from CSV. schema = StructType([ StructField("id", IntegerType()), StructField("name", StringType()), StructField("score", IntegerType()) ]) df = ( spark.readStream .option("sep", ",") .schema(schema) .format("csv") # Replace it with your path to the directory "csv-data". .load("/path/to/csv-data") ) # 2. Write to StarRocks by configuring the format as "starrocks" and the following options. # You need to modify the options according your own environment. query = ( df.writeStream.format("starrocks") .option("starrocks.fe.http.url", "127.0.0.1:8030") .option("starrocks.fe.jdbc.url", "jdbc:mysql://127.0.0.1:9030") .option("starrocks.table.identifier", "test.score_board") .option("starrocks.user", "root") .option("starrocks.password", "") # replace it with your checkpoint directory .option("checkpointLocation", "/path/to/checkpoint") .outputMode("append") .start() ) ``` 3. Query data in the StarRocks table. ```sql MySQL [test]> select * from score_board; +------+-----------+-------+ | id | name | score | +------+-----------+-------+ | 4 | spark | 100 | | 3 | starrocks | 100 | +------+-----------+-------+ 2 rows in set (0.67 sec) ``` ##### Load data with Spark SQL[​](#load-data-with-spark-sql "Direct link to Load data with Spark SQL") The following example explains how to load data with Spark SQL by using the `INSERT INTO` statement in the [Spark SQL CLI](https://spark.apache.org/docs/latest/sql-distributed-sql-engine-spark-sql-cli.html). 1. Execute the following SQL statement in the `spark-sql`: ```sql -- 1. Create a table by configuring the data source as `starrocks` and the following options. -- You need to modify the options according your own environment. CREATE TABLE `score_board` USING starrocks OPTIONS( "starrocks.fe.http.url"="127.0.0.1:8030", "starrocks.fe.jdbc.url"="jdbc:mysql://127.0.0.1:9030", "starrocks.table.identifier"="test.score_board", "starrocks.user"="root", "starrocks.password"="" ); -- 2. Insert two rows into the table. INSERT INTO `score_board` VALUES (5, "starrocks", 100), (6, "spark", 100); ``` 2. Query data in the StarRocks table. ```sql MySQL [test]> select * from score_board; +------+-----------+-------+ | id | name | score | +------+-----------+-------+ | 6 | spark | 100 | | 5 | starrocks | 100 | +------+-----------+-------+ 2 rows in set (0.00 sec) ``` #### Best Practices[​](#best-practices "Direct link to Best Practices") ##### Load data to Primary Key table[​](#load-data-to-primary-key-table "Direct link to Load data to Primary Key table") This section will show how to load data to StarRocks Primary Key table to achieve partial updates, and conditional updates. You can see [Change data through loading](https://docs.starrocks.io/docs/loading/Load_to_Primary_Key_tables.md) for the detailed introduction of these features. These examples use Spark SQL. ###### Preparations[​](#preparations-1 "Direct link to Preparations") Create a database `test` and create a Primary Key table `score_board` in StarRocks. ```sql CREATE DATABASE `test`; CREATE TABLE `test`.`score_board` ( `id` int(11) NOT NULL COMMENT "", `name` varchar(65533) NULL DEFAULT "" COMMENT "", `score` int(11) NOT NULL DEFAULT "0" COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`id`) COMMENT "OLAP" DISTRIBUTED BY HASH(`id`); ``` ###### Partial updates[​](#partial-updates "Direct link to Partial updates") This example will show how to only update data in the column `name` through loading: 1. Insert initial data to StarRocks table in MySQL client. ```sql mysql> INSERT INTO `score_board` VALUES (1, 'starrocks', 100), (2, 'spark', 100); mysql> select * from score_board; +------+-----------+-------+ | id | name | score | +------+-----------+-------+ | 1 | starrocks | 100 | | 2 | spark | 100 | +------+-----------+-------+ 2 rows in set (0.02 sec) ``` 2. Create a Spark table `score_board` in Spark SQL client. * Set the option `starrocks.write.properties.partial_update` to `true` which tells the connector to do partial update. * Set the option `starrocks.columns` to `"id,name"` to tell the connector which columns to write. ```sql CREATE TABLE `score_board` USING starrocks OPTIONS( "starrocks.fe.http.url"="127.0.0.1:8030", "starrocks.fe.jdbc.url"="jdbc:mysql://127.0.0.1:9030", "starrocks.table.identifier"="test.score_board", "starrocks.user"="root", "starrocks.password"="", "starrocks.write.properties.partial_update"="true", "starrocks.columns"="id,name" ); ``` 3. Insert data into the table in Spark SQL client, and only update the column `name`. ```sql INSERT INTO `score_board` VALUES (1, 'starrocks-update'), (2, 'spark-update'); ``` 4. Query the StarRocks table in MySQL client. You can see that only values for `name` change, and the values for `score` does not change. ```sql mysql> select * from score_board; +------+------------------+-------+ | id | name | score | +------+------------------+-------+ | 1 | starrocks-update | 100 | | 2 | spark-update | 100 | +------+------------------+-------+ 2 rows in set (0.02 sec) ``` ###### Conditional updates[​](#conditional-updates "Direct link to Conditional updates") This example will show how to do conditional updates according to the values of column `score`. The update for an `id` takes effect only when the new value for `score` is has a greater or equal to the old value. 1. Insert initial data to StarRocks table in MySQL client. ```sql mysql> INSERT INTO `score_board` VALUES (1, 'starrocks', 100), (2, 'spark', 100); mysql> select * from score_board; +------+-----------+-------+ | id | name | score | +------+-----------+-------+ | 1 | starrocks | 100 | | 2 | spark | 100 | +------+-----------+-------+ 2 rows in set (0.02 sec) ``` 2. Create a Spark table `score_board` in the following ways. * Set the option `starrocks.write.properties.merge_condition` to `score` which tells the connector to use the column `score` as the condition. * Make sure that the Spark connector use Stream Load interface to load data, rather than Stream Load transaction interface, because the latter does not support this feature. ```sql CREATE TABLE `score_board` USING starrocks OPTIONS( "starrocks.fe.http.url"="127.0.0.1:8030", "starrocks.fe.jdbc.url"="jdbc:mysql://127.0.0.1:9030", "starrocks.table.identifier"="test.score_board", "starrocks.user"="root", "starrocks.password"="", "starrocks.write.properties.merge_condition"="score" ); ``` 3. Insert data to the table in Spark SQL client, and update the row whose `id` is 1 with a smaller score value, and the row whose `id` is 2 with a larger score value. ```sql INSERT INTO `score_board` VALUES (1, 'starrocks-update', 99), (2, 'spark-update', 101); ``` 4. Query the StarRocks table in MySQL client. You can see that only the row whose `id` is 2 changes, and the row whose `id` is 1 does not change. ```sql mysql> select * from score_board; +------+--------------+-------+ | id | name | score | +------+--------------+-------+ | 1 | starrocks | 100 | | 2 | spark-update | 101 | +------+--------------+-------+ 2 rows in set (0.03 sec) ``` ##### Load data into columns of BITMAP type[​](#load-data-into-columns-of-bitmap-type "Direct link to Load data into columns of BITMAP type") [`BITMAP`](https://docs.starrocks.io/docs/sql-reference/data-types/other-data-types/BITMAP.md) is often used to accelerate count distinct, such as counting UV, see [Use Bitmap for exact Count Distinct](https://docs.starrocks.io/docs/using_starrocks/distinct_values/Using_bitmap.md). Here we take the counting of UV as an example to show how to load data into columns of the `BITMAP` type. **`BITMAP` is supported since version 1.1.1**. 1. Create a StarRocks Aggregate table. In the database `test`, create an Aggregate table `page_uv` where the column `visit_users` is defined as the `BITMAP` type and configured with the aggregate function `BITMAP_UNION`. ```sql CREATE TABLE `test`.`page_uv` ( `page_id` INT NOT NULL COMMENT 'page ID', `visit_date` datetime NOT NULL COMMENT 'access time', `visit_users` BITMAP BITMAP_UNION NOT NULL COMMENT 'user ID' ) ENGINE=OLAP AGGREGATE KEY(`page_id`, `visit_date`) DISTRIBUTED BY HASH(`page_id`); ``` 2. Create a Spark table. The schema of the Spark table is inferred from the StarRocks table, and the Spark does not support the `BITMAP` type. So you need to customize the corresponding column data type in Spark, for example as `BIGINT`, by configuring the option `"starrocks.column.types"="visit_users BIGINT"`. When using Stream Load to ingest data, the connector uses the [`to_bitmap`](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/to_bitmap.md) function to convert the data of `BIGINT` type into `BITMAP` type. Run the following DDL in `spark-sql`: ```sql CREATE TABLE `page_uv` USING starrocks OPTIONS( "starrocks.fe.http.url"="127.0.0.1:8030", "starrocks.fe.jdbc.url"="jdbc:mysql://127.0.0.1:9030", "starrocks.table.identifier"="test.page_uv", "starrocks.user"="root", "starrocks.password"="", "starrocks.column.types"="visit_users BIGINT" ); ``` 3. Load data into StarRocks table. Run the following DML in `spark-sql`: ```sql INSERT INTO `page_uv` VALUES (1, CAST('2020-06-23 01:30:30' AS TIMESTAMP), 13), (1, CAST('2020-06-23 01:30:30' AS TIMESTAMP), 23), (1, CAST('2020-06-23 01:30:30' AS TIMESTAMP), 33), (1, CAST('2020-06-23 02:30:30' AS TIMESTAMP), 13), (2, CAST('2020-06-23 01:30:30' AS TIMESTAMP), 23); ``` 4. Calculate page UVs from the StarRocks table. ```sql MySQL [test]> SELECT `page_id`, COUNT(DISTINCT `visit_users`) FROM `page_uv` GROUP BY `page_id`; +---------+-----------------------------+ | page_id | count(DISTINCT visit_users) | +---------+-----------------------------+ | 2 | 1 | | 1 | 3 | +---------+-----------------------------+ 2 rows in set (0.01 sec) ``` > **NOTICE:** > > The connector uses [`to_bitmap`](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/to_bitmap.md) function to convert data of the `TINYINT`, `SMALLINT`, `INTEGER`, and `BIGINT` types in Spark to the `BITMAP` type in StarRocks, and uses [`bitmap_hash`](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_hash.md) or [`bitmap_hash64`](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_hash64.md) function for other Spark data types. ##### Load data into columns of HLL type[​](#load-data-into-columns-of-hll-type "Direct link to Load data into columns of HLL type") [`HLL`](https://docs.starrocks.io/docs/sql-reference/data-types/other-data-types/HLL.md) can be used for approximate count distinct, see [Use HLL for approximate count distinct](https://docs.starrocks.io/docs/using_starrocks/distinct_values/Using_HLL.md). Here we take the counting of UV as an example to show how to load data into columns of the `HLL` type. **`HLL` is supported since version 1.1.1**. 1. Create a StarRocks Aggregate table. In the database `test`, create an Aggregate table `hll_uv` where the column `visit_users` is defined as the `HLL` type and configured with the aggregate function `HLL_UNION`. ```sql CREATE TABLE `hll_uv` ( `page_id` INT NOT NULL COMMENT 'page ID', `visit_date` datetime NOT NULL COMMENT 'access time', `visit_users` HLL HLL_UNION NOT NULL COMMENT 'user ID' ) ENGINE=OLAP AGGREGATE KEY(`page_id`, `visit_date`) DISTRIBUTED BY HASH(`page_id`); ``` 2. Create a Spark table. The schema of the Spark table is inferred from the StarRocks table, and the Spark does not support the `HLL` type. So you need to customize the corresponding column data type in Spark, for example as `BIGINT`, by configuring the option `"starrocks.column.types"="visit_users BIGINT"`. When using Stream Load to ingest data, the connector uses the [`hll_hash`](https://docs.starrocks.io/docs/sql-reference/sql-functions/scalar-functions/hll_hash.md) function to convert the data of `BIGINT` type into `HLL` type. Run the following DDL in `spark-sql`: ```sql CREATE TABLE `hll_uv` USING starrocks OPTIONS( "starrocks.fe.http.url"="127.0.0.1:8030", "starrocks.fe.jdbc.url"="jdbc:mysql://127.0.0.1:9030", "starrocks.table.identifier"="test.hll_uv", "starrocks.user"="root", "starrocks.password"="", "starrocks.column.types"="visit_users BIGINT" ); ``` 3. Load data into StarRocks table. Run the following DML in `spark-sql`: ```sql INSERT INTO `hll_uv` VALUES (3, CAST('2023-07-24 12:00:00' AS TIMESTAMP), 78), (4, CAST('2023-07-24 13:20:10' AS TIMESTAMP), 2), (3, CAST('2023-07-24 12:30:00' AS TIMESTAMP), 674); ``` 4. Calculate page UVs from the StarRocks table. ```sql MySQL [test]> SELECT `page_id`, COUNT(DISTINCT `visit_users`) FROM `hll_uv` GROUP BY `page_id`; +---------+-----------------------------+ | page_id | count(DISTINCT visit_users) | +---------+-----------------------------+ | 4 | 1 | | 3 | 2 | +---------+-----------------------------+ 2 rows in set (0.01 sec) ``` ##### Load data into columns of ARRAY type[​](#load-data-into-columns-of-array-type "Direct link to Load data into columns of ARRAY type") The following example explains how to load data into columns of the [`ARRAY`](https://docs.starrocks.io/docs/sql-reference/data-types/semi_structured/Array.md) type. 1. Create a StarRocks table. In the database `test`, create a Primary Key table `array_tbl` that includes one `INT` column and two `ARRAY` columns. ```sql CREATE TABLE `array_tbl` ( `id` INT NOT NULL, `a0` ARRAY, `a1` ARRAY> ) ENGINE=OLAP PRIMARY KEY(`id`) DISTRIBUTED BY HASH(`id`) ; ``` 2. Write data to StarRocks. Because some versions of StarRocks does not provide the metadata of `ARRAY` column, the connector can not infer the corresponding Spark data type for this column. However, you can explicitly specify the corresponding Spark data type of the column in the option `starrocks.column.types`. In this example, you can configure the option as `a0 ARRAY,a1 ARRAY>`. Run the following codes in `spark-shell`: ```scala val data = Seq( | (1, Seq("hello", "starrocks"), Seq(Seq(1, 2), Seq(3, 4))), | (2, Seq("hello", "spark"), Seq(Seq(5, 6, 7), Seq(8, 9, 10))) | ) val df = data.toDF("id", "a0", "a1") df.write .format("starrocks") .option("starrocks.fe.http.url", "127.0.0.1:8030") .option("starrocks.fe.jdbc.url", "jdbc:mysql://127.0.0.1:9030") .option("starrocks.table.identifier", "test.array_tbl") .option("starrocks.user", "root") .option("starrocks.password", "") .option("starrocks.column.types", "a0 ARRAY,a1 ARRAY>") .mode("append") .save() ``` 3. Query data in the StarRocks table. ```sql MySQL [test]> SELECT * FROM `array_tbl`; +------+-----------------------+--------------------+ | id | a0 | a1 | +------+-----------------------+--------------------+ | 1 | ["hello","starrocks"] | [[1,2],[3,4]] | | 2 | ["hello","spark"] | [[5,6,7],[8,9,10]] | +------+-----------------------+--------------------+ 2 rows in set (0.01 sec) ``` --- ### Apache Spark Load ### Load data in bulk using Spark Load This load uses external Apache Spark™ resources to pre-process imported data, which improves import performance and saves compute resources. It is mainly used for **initial migration** and **large data import** into StarRocks (data volume up to TB level). Spark load is an **asynchronous** import method that requires users to create Spark-type import jobs via the MySQL protocol and view the import results using `SHOW LOAD`. > **NOTICE** > > * Only users with the INSERT privilege on a StarRocks table can load data into this table. You can follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the required privilege. > * Spark Load can not be used to load data into a Primary Key table. #### Terminology explanation[​](#terminology-explanation "Direct link to Terminology explanation") * **Spark ETL**: Mainly responsible for ETL of data in the import process, including global dictionary construction (BITMAP type), partitioning, sorting, aggregation, etc. * **Broker**: Broker is an independent stateless process. It encapsulates the file system interface and provides StarRocks with the ability to read files from remote storage systems. * **Global Dictionary**: Saves the data structure that maps data from the original value to the encoded value. The original value can be any data type, while the encoded value is an integer. The global dictionary is mainly used in scenarios where exact count distinct is precomputed. #### Fundamentals[​](#fundamentals "Direct link to Fundamentals") The user submits a Spark type import job through the MySQL client;the FE records the metadata and returns the submission result. The execution of the spark load task is divided into the following main phases. 1. The user submits the spark load job to the FE. 2. The FE schedules the submission of the ETL task to the Apache Spark™ cluster for execution. 3. The Apache Spark™ cluster executes the ETL task that includes global dictionary construction (BITMAP type), partitioning, sorting, aggregation, etc. 4. After the ETL task is completed, the FE gets the data path of each preprocessed slice and schedules the relevant BE to execute the Push task. 5. The BE reads data through Broker process from HDFS and converts it into StarRocks storage format. > If you choose not to use Broker process, the BE reads data from HDFS directly. 6. The FE schedules the effective version and completes the import job. The following diagram illustrates the main flow of spark load. ![Spark load](/assets/images/4.3.2-1-7104f83d68f711e7dd819b5f11391eb7.png) *** #### Global Dictionary[​](#global-dictionary "Direct link to Global Dictionary") ##### Applicable Scenarios[​](#applicable-scenarios "Direct link to Applicable Scenarios") Currently, the BITMAP column in StarRocks is implemented using the Roaringbitmap, which only has integer to be the input data type. So if you want to implement precomputation for the BITMAP column in the import process, then you need to convert the input data type to integer. In the existing import process of StarRocks, the data structure of the global dictionary is implemented based on the Hive table, which saves the mapping from the original value to the encoded value. ##### Build Process[​](#build-process "Direct link to Build Process") 1. Read the data from the upstream data source and generate a temporary Hive table, named `hive-table`. 2. Extract the values of the de-emphasized fields of `hive-table` to generate a new Hive table named `distinct-value-table`. 3. Create a new global dictionary table named `dict-table` with one column for the original values and one column for the encoded values. 4. Left join between `distinct-value-table` and `dict-table`, and then use the window function to encode this set. Finally both the original value and the encoded value of the de-duplicated column are written back to `dict-table`. 5. Join between `dict-table` and `hive-table` to finish the job of replacing the original value in `hive-table` with the integer encoded value. 6. `hive-table` will be read by the next time data pre-processing, and then imported into StarRocks after calculation. #### Data Pre-processing[​](#data-pre-processing "Direct link to Data Pre-processing") The basic process of data pre-processing is as follows: 1. Read data from the upstream data source (HDFS file or Hive table). 2. Complete field mapping and calculation for the read data, then generate `bucket-id` based on the partition information. 3. Generate RollupTree based on the Rollup metadata of StarRocks table. 4. Iterate through the RollupTree and perform hierarchical aggregation operations. The Rollup of the next hierarchy can be calculated from the Rollup of the previous hierarchy. 5. Each time the aggregation calculation is completed, the data is bucketed according to `bucket-id` and then written to HDFS. 6. The subsequent Broker process will pull the files from HDFS and import them into the StarRocks BE node. #### Basic Operations[​](#basic-operations "Direct link to Basic Operations") ##### Configuring ETL Clusters[​](#configuring-etl-clusters "Direct link to Configuring ETL Clusters") Apache Spark™ is used as an external computational resource in StarRocks for ETL work. There may be other external resources added to StarRocks, such as Spark/GPU for query, HDFS/S3 for external storage, MapReduce for ETL, etc. Therefore, we introduce `Resource Management` to manage these external resources used by StarRocks. Before submitting a Apache Spark™ import job, configure the Apache Spark™ cluster for performing ETL tasks. The syntax for operation is as follows: ```sql -- create Apache Spark™ resource CREATE EXTERNAL RESOURCE resource_name PROPERTIES ( type = spark, spark_conf_key = spark_conf_value, working_dir = path, broker = broker_name, broker.property_key = property_value ); -- drop Apache Spark™ resource DROP RESOURCE resource_name; -- show resources SHOW RESOURCES SHOW PROC "/resources"; -- privileges GRANT USAGE_PRIV ON RESOURCE resource_name TO user_identityGRANT USAGE_PRIV ON RESOURCE resource_name TO ROLE role_name; REVOKE USAGE_PRIV ON RESOURCE resource_name FROM user_identityREVOKE USAGE_PRIV ON RESOURCE resource_name FROM ROLE role_name; ``` * Create resource **For example**: ```sql -- yarn cluster mode CREATE EXTERNAL RESOURCE "spark0" PROPERTIES ( "type" = "spark", "spark.master" = "yarn", "spark.submit.deployMode" = "cluster", "spark.jars" = "xxx.jar,yyy.jar", "spark.files" = "/tmp/aaa,/tmp/bbb", "spark.executor.memory" = "1g", "spark.yarn.queue" = "queue0", "spark.hadoop.yarn.resourcemanager.address" = "127.0.0.1:9999", "spark.hadoop.fs.defaultFS" = "hdfs://127.0.0.1:10000", "working_dir" = "hdfs://127.0.0.1:10000/tmp/starrocks", "broker" = "broker0", "broker.username" = "user0", "broker.password" = "password0" ); -- yarn HA cluster mode CREATE EXTERNAL RESOURCE "spark1" PROPERTIES ( "type" = "spark", "spark.master" = "yarn", "spark.submit.deployMode" = "cluster", "spark.hadoop.yarn.resourcemanager.ha.enabled" = "true", "spark.hadoop.yarn.resourcemanager.ha.rm-ids" = "rm1,rm2", "spark.hadoop.yarn.resourcemanager.hostname.rm1" = "host1", "spark.hadoop.yarn.resourcemanager.hostname.rm2" = "host2", "spark.hadoop.fs.defaultFS" = "hdfs://127.0.0.1:10000", "working_dir" = "hdfs://127.0.0.1:10000/tmp/starrocks", "broker" = "broker1" ); ``` `resource-name` is the name of the Apache Spark™ resource configured in StarRocks. `PROPERTIES` includes parameters relating to the Apache Spark™ resource, as follows: > **Note** > > For detailed description of Apache Spark™ resource PROPERTIES, please see [CREATE RESOURCE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Resource/CREATE_RESOURCE.md) * Spark related parameters: * `type`: Resource type, required, currently only supports `spark`. * `spark.master`: Required, currently only supports `yarn`. * `spark.submit.deployMode`: The deployment mode of the Apache Spark™ program, required, currently supports both `cluster` and `client`. * `spark.hadoop.fs.defaultFS`: Required if master is yarn. * Parameters related to yarn resource manager, required. * one ResourceManager on a single node `spark.hadoop.yarn.resourcemanager.address`: Address of the single point resource manager. * ResourceManager HA > You can choose to specify ResourceManager's hostname or address. * `spark.hadoop.yarn.resourcemanager.ha.enabled`: Enable the resource manager HA, set to `true`. * `spark.hadoop.yarn.resourcemanager.ha.rm-ids`: list of resource manager logical ids. * `spark.hadoop.yarn.resourcemanager.hostname.rm-id`: For each rm-id, specify the hostname corresponding to the resource manager. * `spark.hadoop.yarn.resourcemanager.address.rm-id`: For each rm-id, specify `host:port` for the client to submit jobs to. * `*working_dir`: The directory used by ETL. Required if Apache Spark™ is used as an ETL resource. For example: `hdfs://host:port/tmp/starrocks`. * Broker related parameters: * `broker`: Broker name. Required if Apache Spark™ is used as an ETL resource. You need to use the `ALTER SYSTEM ADD BROKER` command to complete the configuration in advance. * `broker.property_key`: Information (e.g.authentication information) to be specified when Broker process reads the intermediate file generated by the ETL. **Precaution**: The above is a description of parameters for loading through Broker process. If you intend to load data without Broker process, the following should be noted. * You do not need to specify `broker`. * If you need to configure user authentication, and HA for NameNode nodes, you need to configure the parameters in the hdfs-site.xml file in the HDFS cluster, see [broker\_properties](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md#hdfs) for descriptions of parameters. and you need to move the **hdfs-site.xml** file under **$FE\_HOME/conf** for each FE and **$BE\_HOME/conf** for each BE. > Note > > If the HDFS file can only be accessed by a specific user, you still need to specify the HDFS username in `broker.name` and the user password in `broker.password`. * View resources Regular accounts can only view resources to which they have `USAGE-PRIV` access. The root and admin accounts can view all resources. * Resource Permissions Resource permissions are managed through `GRANT REVOKE`, which currently only supports `USAGE-PRIV` permissions. You can give `USAGE-PRIV` permissions to a user or a role. ```sql -- Grant access to spark0 resources to user0 GRANT USAGE_PRIV ON RESOURCE "spark0" TO "user0"@"%"; -- Grant access to spark0 resources to role0 GRANT USAGE_PRIV ON RESOURCE "spark0" TO ROLE "role0"; -- Grant access to all resources to user0 GRANT USAGE_PRIV ON RESOURCE* TO "user0"@"%"; -- Grant access to all resources to role0 GRANT USAGE_PRIV ON RESOURCE* TO ROLE "role0"; -- Revoke the use privileges of spark0 resources from user user0 REVOKE USAGE_PRIV ON RESOURCE "spark0" FROM "user0"@"%"; ``` ##### Configuring Spark Client[​](#configuring-spark-client "Direct link to Configuring Spark Client") Configure the Spark client for FE so that the latter can submit Spark tasks by executing the `spark-submit` command. It is recommended to use the official version of Spark2 2.4.5 or above [spark download address](https://archive.apache.org/dist/spark/). After downloading, please use the following steps to complete the configuration. * Configure `SPARK-HOME` Place the Spark client in a directory on the same machine as the FE, and configure `spark_home_default_dir` in the FE configuration file to this directory, which by default is the `lib/spark2x` path in the FE root directory, and cannot be empty. * **Configure SPARK dependency package** To configure the dependency package, zip and archive all jar files in the jars folder under the Spark client, and configure the `spark_resource_path` item in the FE configuration to this zip file. If this configuration is empty, the FE will try to find the `lib/spark2x/jars/spark-2x.zip` file in the FE root directory. If the FE fails to find it, it will report an error. When the spark load job is submitted, the archived dependency files will be uploaded to the remote repository. The default repository path is under the `working_dir/{cluster_id}` directory named with `--spark-repository--{resource-name}`, which means that a resource in the cluster corresponds to a remote repository. The directory structure is referenced as follows: ```bash ---spark-repository--spark0/ |---archive-1.0.0/ | |\---lib-990325d2c0d1d5e45bf675e54e44fb16-spark-dpp-1.0.0\-jar-with-dependencies.jar | |\---lib-7670c29daf535efe3c9b923f778f61fc-spark-2x.zip |---archive-1.1.0/ | |\---lib-64d5696f99c379af2bee28c1c84271d5-spark-dpp-1.1.0\-jar-with-dependencies.jar | |\---lib-1bbb74bb6b264a270bc7fca3e964160f-spark-2x.zip |---archive-1.2.0/ | |-... ``` In addition to the spark dependencies (named `spark-2x.zip` by default), the FE also uploads the DPP dependencies to the remote repository. If all the dependencies submitted by the spark load already exist in the remote repository, then there is no need to upload the dependencies again, saving the time of repeatedly uploading a large number of files each time. ##### Configuring YARN Client[​](#configuring-yarn-client "Direct link to Configuring YARN Client") Configure the yarn client for FE so that the FE can execute yarn commands to get the status of the running application or kill it.It is recommended to use the official version of Hadoop2 2.5.2 or above ([hadoop download address](https://archive.apache.org/dist/hadoop/common/)). After downloading, please use the following steps to complete the configuration: * **Configure the YARN executable path** Place the downloaded yarn client in a directory on the same machine as the FE, and configure the `yarn_client_path` item in the FE configuration file to the binary executable file of yarn, which by default is the `lib/yarn-client/hadoop/bin/yarn` path in the FE root directory. * **Configure the path to the configuration file needed to generate YARN (optional)** When the FE goes through the yarn client to get the status of the application, or to kill the application, by default StarRocks generates the configuration file required to execute the yarn command in the `lib/yarn-config` path of the FE root directory This path can be modified by configuring the `yarn_config_dir` entry in the FE configuration file, which currently includes `core-site.xml` and `yarn-site.xml`. ##### Create Import Job[​](#create-import-job "Direct link to Create Import Job") **Syntax:** ```sql LOAD LABEL load_label (data_desc, ...) WITH RESOURCE resource_name [resource_properties] [PROPERTIES (key1=value1, ... )] * load_label: db_name.label_name * data_desc: DATA INFILE ('file_path', ...) [NEGATIVE] INTO TABLE tbl_name [PARTITION (p1, p2)] [COLUMNS TERMINATED BY separator ] [(col1, ...)] [COLUMNS FROM PATH AS (col2, ...)] [SET (k1=f1(xx), k2=f2(xx))] [WHERE predicate] DATA FROM TABLE hive_external_tbl [NEGATIVE] INTO TABLE tbl_name [PARTITION (p1, p2)] [SET (k1=f1(xx), k2=f2(xx))] [WHERE predicate] * resource_properties: (key2=value2, ...) ``` **Example 1**: The case where the upstream data source is HDFS ```sql LOAD LABEL db1.label1 ( DATA INFILE("hdfs://abc.com:8888/user/starrocks/test/ml/file1") INTO TABLE tbl1 COLUMNS TERMINATED BY "," (tmp_c1,tmp_c2) SET ( id=tmp_c2, name=tmp_c1 ), DATA INFILE("hdfs://abc.com:8888/user/starrocks/test/ml/file2") INTO TABLE tbl2 COLUMNS TERMINATED BY "," (col1, col2) where col1 > 1 ) WITH RESOURCE 'spark0' ( "spark.executor.memory" = "2g", "spark.shuffle.compress" = "true" ) PROPERTIES ( "timeout" = "3600" ); ``` **Example 2**: The case where the upstream data source is Hive. * Step 1: Create a new hive resource ```sql CREATE EXTERNAL RESOURCE hive0 PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:8080" ); ``` * Step 2: Create a new hive external table ```sql CREATE EXTERNAL TABLE hive_t1 ( k1 INT, K2 SMALLINT, k3 varchar(50), uuid varchar(100) ) ENGINE=hive PROPERTIES ( "resource" = "hive0", "database" = "tmp", "table" = "t1" ); ``` * Step 3: Submit the load command, requiring that the columns in the imported StarRocks table exist in the hive external table. ```sql LOAD LABEL db1.label1 ( DATA FROM TABLE hive_t1 INTO TABLE tbl1 SET ( uuid=bitmap_dict(uuid) ) ) WITH RESOURCE 'spark0' ( "spark.executor.memory" = "2g", "spark.shuffle.compress" = "true" ) PROPERTIES ( "timeout" = "3600" ); ``` Introduction to the parameters in the Spark load: * **Label** Label of the import job. Each import job has a Label that is unique within the database, following the same rules as broker load. * **Data description class parameters** Currently, supported data sources are CSV and Hive table. Other rules are the same as broker load. * **Import Job Parameters** Import job parameters refer to the parameters belonging to the `opt_properties` section of the import statement. These parameters are applicable to the entire import job. The rules are the same as broker load. * **Spark Resource Parameters** Spark resources need to be configured into StarRocks in advance and users need to be given USAGE-PRIV permissions before they can apply the resources to Spark load. Spark resource parameters can be set when the user has a temporary need, such as adding resources for a job and modifying Spark configs. The setting only takes effect on this job and does not affect the existing configurations in the StarRocks cluster. ```sql WITH RESOURCE 'spark0' ( "spark.driver.memory" = "1g", "spark.executor.memory" = "3g" ) ``` * **Import when the data source is Hive** Currently, to use a Hive table in the import process, you need to create an external table of the `Hive` type and then specify its name when submitting the import command. * **Import process to build a global dictionary** In the load command, you can specify the required fields for building the global dictionary in the following format: `StarRocks field name=bitmap_dict(hive table field name)` Note that currently **the global dictionary is only supported when the upstream data source is a Hive table**. * **Load binary type data** Since v2.5.17, Spark Load supports the bitmap\_from\_binary function, which can convert binary data into bitmap data. If the column type of the Hive table or HDFS file is binary and the corresponding column in the StarRocks table is a bitmap-type aggregate column, you can specify the fields in the load command in the following format, `StarRocks field name=bitmap_from_binary(Hive table field name)`. This eliminates the need for building a global dictionary. #### Viewing Import Jobs[​](#viewing-import-jobs "Direct link to Viewing Import Jobs") The Spark load import is asynchronous, as is the broker load. The user must record the label of the import job and use it in the `SHOW LOAD` command to view the import results. The command to view the import is common to all import methods. The example is as follows. Refer to Broker Load for a detailed explanation of returned parameters.The differences are as follows. ```sql mysql> show load order by createtime desc limit 1\G *************************** 1. row *************************** JobId: 76391 Label: label1 State: FINISHED Progress: ETL:100%; LOAD:100% Type: SPARK EtlInfo: unselected.rows=4; dpp.abnorm.ALL=15; dpp.norm.ALL=28133376 TaskInfo: cluster:cluster0; timeout(s):10800; max_filter_ratio:5.0E-5 ErrorMsg: N/A CreateTime: 2019-07-27 11:46:42 EtlStartTime: 2019-07-27 11:46:44 EtlFinishTime: 2019-07-27 11:49:44 LoadStartTime: 2019-07-27 11:49:44 LoadFinishTime: 2019-07-27 11:50:16 URL: http://1.1.1.1:8089/proxy/application_1586619723848_0035/ JobDetails: {"ScannedRows":28133395,"TaskNumber":1,"FileNumber":1,"FileSize":200000} ``` * **State** The current stage of the imported job. PENDING: The job is committed. ETL: Spark ETL is committed. LOADING: The FE schedule an BE to execute push operation. FINISHED: The push is completed and the version is effective. There are two final stages of the import job – `CANCELLED` and `FINISHED`, both indicating the load job is completed. `CANCELLED` indicates import failure and `FINISHED` indicates import success. * **Progress** Description of the import job progress. There are two types of progress –ETL and LOAD, which correspond to the two phases of the import process, ETL and LOADING. * The range of progress for LOAD is 0~100%. `LOAD progress = the number of currently completed tablets of all replications imports / the total number of tablets of this import job * 100%`. * If all tables have been imported, the LOAD progress is 99%, and changes to 100% when the import enters the final validation phase. * The import progress is not linear. If there is no change in progress for a period of time, it does not mean that the import is not executing. * **Type** The type of the import job. SPARK for spark load. * **CreateTime/EtlStartTime/EtlFinishTime/LoadStartTime/LoadFinishTime** These values represent the time when the import was created, when the ETL phase started, when the ETL phase completed, when the LOADING phase started, and when the entire import job was completed. * **JobDetails** Displays the detailed running status of the job, including the number of imported files, total size (in bytes), number of subtasks, number of raw rows being processed, etc. For example: ```json {"ScannedRows":139264,"TaskNumber":1,"FileNumber":1,"FileSize":940754064} ``` * **URL** You can copy the input to your browser to access the web interface of the corresponding application. ##### View Apache Spark™ Launcher commit logs[​](#view-apache-spark-launcher-commit-logs "Direct link to View Apache Spark™ Launcher commit logs") Sometimes users need to view the detailed logs generated during a Apache Spark™ job commit. By default, the logs are saved in the path `log/spark_launcher_log` in the FE root directory named as `spark-launcher-{load-job-id}-{label}.log`. The logs are saved in this directory for a period of time and will be erased when the import information in FE metadata is cleaned up. The default retention time is 3 days. ##### Cancel Import[​](#cancel-import "Direct link to Cancel Import") When the Spark load job status is not `CANCELLED` or `FINISHED`, it can be cancelled manually by the user by specifying the Label of the import job. *** #### Related System Configurations[​](#related-system-configurations "Direct link to Related System Configurations") **FE Configuration:** The following configuration is the system-level configuration of Spark load, which applies to all Spark load import jobs. The configuration values can be adjusted mainly by modifying `fe.conf`. * enable-spark-load: Enable Spark load and resource creation with a default value of false. * spark-load-default-timeout-second: The default timeout for the job is 259200 seconds (3 days). * spark-home-default-dir: The Spark client path (`fe/lib/spark2x`). * spark-resource-path: The path to the packaged S park dependency file (empty by default). * spark-launcher-log-dir: The directory where the commit log of the Spark client is stored (`fe/log/spark-launcher-log`). * yarn-client-path: The path to the yarn binary executable (`fe/lib/yarn-client/hadoop/bin/yarn`). * yarn-config-dir: Yarn's configuration file path (`fe/lib/yarn-config`). *** #### Best Practices[​](#best-practices "Direct link to Best Practices") The most suitable scenario for using Spark load is when the raw data is in the file system (HDFS) and the data volume is in the tens of GB to TB level. Use Stream Load or Broker Load for smaller data volumes. For the full spark load import example, refer to the demo on github: #### FAQs[​](#faqs "Direct link to FAQs") * `Error: When running with master 'yarn' either HADOOP-CONF-DIR or YARN-CONF-DIR must be set in the environment.` Using Spark Load without configuring the `HADOOP-CONF-DIR` environment variable in `spark-env.sh` of the Spark client. * `Error: Cannot run program "xxx/bin/spark-submit": error=2, No such file or directory` The `spark_home_default_dir` configuration item does not specify the Spark client root directory when using Spark Load. * `Error: File xxx/jars/spark-2x.zip does not exist.` The `spark-resource-path` configuration item does not point to the packed zip file when using Spark load. * `Error: yarn client does not exist in path: xxx/yarn-client/hadoop/bin/yarn` The yarn-client-path configuration item does not specify the yarn executable when using Spark load. * `ERROR: Cannot execute hadoop-yarn/bin/... /libexec/yarn-config.sh` When using Hadoop with CDH, you need to configure the `HADOOP_LIBEXEC_DIR` environment variable. Since `hadoop-yarn` and hadoop directories are different, the default `libexec` directory will look for `hadoop-yarn/bin/... /libexec`, while `libexec` is in the hadoop directory. The \`\`\`yarn application status\`\` command to get the Spark task status reported an error causing the import job to fail. --- ### FineBI This topic describes how to connect FineBI to StarRocks to perform visualized analysis of StarRocks data on FineBI. For details about how to connect to StarRocks on FineBI, see [StarRocks data connection](https://help.fanruan.com/finebi-en/doc-view-5857.html). --- ### Hex Hex supports querying and visualizing both internal data and external data in StarRocks. Add a data connection in Hex. Note that you must select MySQL as the connection type. ![Hex](/assets/images/BI_hex_1-b380296e12a2537f7d48d3f36fc4e094.png) The parameters that you need to configure are described as follows: * **Name**: the name of the data connection. * **Host & port**: the FE host IP address and FE query port of your StarRocks cluster. An example query port is `9030`. * **Database**: the data source that you want to access in your StarRocks cluster. The value of this parameter is in the `.` format. * `catalog_name`: the name of the target catalog in your StarRocks cluster. Both internal and external catalogs are supported. * `database_name`: the name of the target database in your StarRocks cluster. Both internal and external databases are supported. * **Type**: the authentication method that you want to use. Select **Password**. * **User**: the username that is used to log in to your StarRocks cluster, for example, `admin`. * **Password**: the password that is used to log in to your StarRocks cluster. --- ### Metabase Metabase supports querying and visualizing both internal data and external data in StarRocks. Start Metabase and do as follows: 1. In the upper-right corner of the Metabase homepage, click the **Settings** icon and choose **Admin settings**. ![Metabase - Admin settings](/assets/images/Metabase_1-2f1158158723a03d42120e9de4768f3f.png) 2. Choose **Databases** in the top menu bar. 3. On the **Databases** page, click **Add database**. ![Metabase - Add database](/assets/images/Metabase_2-2e0e966419eaa3c520d7b02f9de4eeae.png) 4. On the page that appears, configure the database parameters and click **Save**. * **Database type**: Select **MySQL**. * **Host** and **Port**: Enter the host and port information appropriate for your use case. * **Database name**: Enter a database name in the `.` format. In StarRocks versions earlier than v3.2, you can integrate only the internal catalog of your StarRocks cluster with Metabase. From StarRocks v3.2 onwards, you can integrate both the internal catalog and external catalogs of your StarRocks cluster with Metabase. * **Username** and **Password**: Enter the username and password of your StarRocks cluster user. * **Additional JDBC connection string options**: You must add the property `tinyInt1isBit=false` in this field. Otherwise, there may be an error. The other parameters do not involve StarRocks. Configure them based on your business needs. ![Metabase - Configure database](/assets/images/Metabase_3-c7994bd1edc973e10c1ccce5b63a0883.png) NOTE: Please avoid using DECIMAL data types as Metabase does not understand this StarRocks specific column data type --- ### Querybook Querybook supports querying and visualizing both internal data and external data in StarRocks. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") Make sure that you have finished the following preparations: 1. Clone and download the Querybook repository. ```sql git clone git@github.com:pinterest/querybook.git cd querybook ``` 2. Create a file named `local.txt` under the `requirements` folder in the project's root directory. ```sql touch requirements/local.txt ``` 3. Add the required packages. ```sql echo -e "starrocks\nmysqlclient" > requirements/local.txt ``` 4. Start the container. ```sql make ``` #### Integration[​](#integration "Direct link to Integration") Visit the following URL and add a new query engine: ```plain https://localhost:10001/admin/query_engine/ ``` ![Querybook](/assets/images/BI_querybook_1-4805d379448a5c6df3b524aad977f499.png) Take note of the following points: * For **Language**, select **Starrocks**. * For **Executor**, select **sqlalchemy**. * For **Connection\_string**, enter a URI in the StarRocks SQLAlchemy URI format as below: ```sql starrocks://:@:/. ``` The parameters in the URI are described as follows: * `User`: the username that is used to log in to your StarRocks cluster, for example, `admin`. * `Password`: the password that is used to log in to your StarRocks cluster. * `Host`: the FE host IP address of your StarRocks cluster. * `Port`: the FE query port of your StarRocks cluster, for example, `9030`. * `Catalog`: the target catalog in your StarRocks cluster. Both internal and external catalogs are supported. * `Database`: the target database in your StarRocks cluster. Both internal and external databases are supported. --- ### QuickBI This topic describes how to connect QuickBI to StarRocks to perform visualized analysis of StarRocks data on QuickBI. For details about how to connect to StarRocks on QuickBI, see [Add a user-created StarRocks data source](https://www.alibabacloud.com/help/en/quick-bi/user-guide/add-a-user-created-starrocks-data-source). --- ### Rill Rill supports connecting to StarRocks as an OLAP connector to power Rill dashboards with external tables. Rill can query and visualize both internal data and external data in StarRocks. #### Connection[​](#connection "Direct link to Connection") Rill connects to StarRocks using the MySQL protocol. You can configure the connection using either connection parameters or a DSN connection string. ##### Connection Parameters[​](#connection-parameters "Direct link to Connection Parameters") When adding a data source in Rill, select **StarRocks** and configure the following parameters: * **Host**: The FE host IP address or hostname of your StarRocks cluster * **Port**: The MySQL protocol port of the StarRocks FE (default: `9030`) * **Username**: The username for authentication (default: `root`) * **Password**: The password for authentication * **Catalog**: The StarRocks catalog name (default: `default_catalog`). Supports both internal and external catalogs (e.g., Iceberg, Hive) * **Database**: The StarRocks database name * **SSL**: Enable SSL/TLS encryption (default: `false`) ##### Connection String (DSN)[​](#connection-string-dsn "Direct link to Connection String (DSN)") Alternatively, you can use a MySQL-format DSN connection string: ```text user:password@tcp(host:9030)/database?parseTime=true ``` For external catalogs, specify the catalog and database as separate properties. #### External Catalogs[​](#external-catalogs "Direct link to External Catalogs") Rill supports querying data from external catalogs in StarRocks, including Hive, Iceberg, Delta Lake, and other external data sources. Set the `catalog` property to your external catalog name (e.g., `iceberg_catalog`) and the `database` property to the database within that catalog. #### More Information[​](#more-information "Direct link to More Information") For detailed configuration options, examples, troubleshooting, and the most up-to-date information, see the [Rill Data StarRocks connector documentation](https://docs.rilldata.com/developers/build/connectors/olap/starrocks). --- ### Apache Superset Apache Superset supports querying and visualizing both internal data and external data in StarRocks. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") Make sure that you have finished the following installations: 1. Install the Python client for StarRocks on your Apache Superset server. ```sql pip install starrocks ``` 2. Install the latest version of Apache Superset. For more information, see [Installing Superset](https://superset.apache.org/docs/intro). #### Integration[​](#integration "Direct link to Integration") Create a database in Apache Superset: ![Apache Superset - 1](/assets/images/BI_superset_1-eff12bff07257eed12c79806812c0da6.png) ![Apache Superset - 2](/assets/images/BI_superset_2-0f8ec523d0d808a9e157ab1a0e91c69a.png) Take note of the following points: * For **SUPPORTED DATABASES**, select **StarRocks**, which will be used as the data source. * For **SQLALCHEMY** **URI**, enter a URI in the StarRocks SQLAlchemy URI format as below: ```sql starrocks://:@:/. ``` The parameters in the URI are described as follows: * `User`: the username that is used to log in to your StarRocks cluster, for example, `admin`. * `Password`: the password that is used to log in to your StarRocks cluster. * `Host`: the FE host IP address of your StarRocks cluster. * `Port`: the FE query port of your StarRocks cluster, for example, `9030`. * `Catalog`: the target catalog in your StarRocks cluster. Both internal and external catalogs are supported. * `Database`: the target database in your StarRocks cluster. Both internal and external databases are supported. --- ### Tableau This topic describes how to connect StarRocks to Tableau Desktop and Tableau Server with StarRocks Tableau JDBC Connector. #### Overview[​](#overview "Direct link to Overview") The StarRocks Tableau JDBC Connector is a custom extension for Tableau Desktop and Tableau Server. It simplifies the process of connecting Tableau to StarRocks and enhances support for standard Tableau functionality, outperforming the default Generic ODBC/JDBC connection. ##### Key Features[​](#key-features "Direct link to Key Features") * LDAP Support: Enables LDAP login with password prompts for secure authentication. * High Compatibility: Achieves 99.99% compatibility in TDVT (Tableau Design Verification Tool) testing, with only one minor failure case. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before proceeding, make sure the following requirements are met: * Tableau Version: Tableau 2020.4 and later * StarRocks Version: v3.2 and later #### Install Connector for Tableau Desktop[​](#install-connector-for-tableau-desktop "Direct link to Install Connector for Tableau Desktop") 1. Download the [MySQL JDBC Driver 8.0.33](https://downloads.mysql.com/archives/c-j/). 2. Store the driver file in the following directory (create the directory if it does not exist): * macOS: `~/Library/Tableau/Drivers` * Windows: `C:\Program Files\Tableau\Drivers` 3. Download the [StarRocks Tableau JDBC Connector file](https://exchange.tableau.com/products/1079). 4. Store the connector file in the following directory: * macOS: `~/Documents/My Tableau Repository/Connectors` * Windows: `C:\Users\[Windows User]\Documents\My Tableau Repository\Connectors` 5. Launch Tableau Desktop. 6. Navigate to **Connect** -> **To a Server** -> **StarRocks JDBC by CelerData**. #### Install Connector for Tableau Server[​](#install-connector-for-tableau-server "Direct link to Install Connector for Tableau Server") 1. Download the [MySQL JDBC Driver 8.0.33](https://downloads.mysql.com/archives/c-j/). 2. Store the driver file in the following directory (create the directory if it does not exist): * Linux: `/opt/tableau/tableau_driver/jdbc` * Windows: `C:\Program Files\Tableau\Drivers` info On Linux, you must permit the "Tableau" user to access the directory. Follow these steps: 1. Create the directory and copy the driver file to the directory: ```bash sudo mkdir -p /opt/tableau/tableau_driver/jdbc # Replace with the absolute path of the driver file. sudo cp /.jar /opt/tableau/tableau_driver/jdbc ``` 2. Grant permission to the the "Tableau" user. ```bash # Replace with the name of the driver file. sudo chmod 755 /opt/tableau/tableau_driver/jdbc/.jar ``` 3. Download the [StarRocks Tableau JDBC Connector file](https://exchange.tableau.com/products/1079). 4. Store the connector file in the following directory of each node: * Linux: `/opt/tableau/connectors` * Windows: `C:\Program Files\Tableau\Connectors` 5. Restart Tableau Server. ```bash tsm restart ``` info You must restart Tableau Server to apply the changes whenever you add, remove, or update a connector. #### Usage notes[​](#usage-notes "Direct link to Usage notes") If LDAP login support is required, you can tick the **Enable LDAP** switch in the **Advanced** tab during configuration. --- ### Data Lakehouse ![DLA](/assets/images/1.1-8-dla-c67d601d709b092317fa11eb64ac1783.png) In addition to efficient analytics of local data, StarRocks can work as the compute engine to analyze data stored in data lakes such as Apache Hudi, Apache Iceberg, and Delta Lake. One of the key features of StarRocks is its external catalog, which acts as the linkage to an externally maintained metastore. This functionality provides users with the capability to query external data sources seamlessly, eliminating the need for data migration. As such, users can analyze data from different systems such as HDFS and Amazon S3, in various file formats such as Parquet, ORC, and CSV, etc. The preceding figure shows a data lake analytics scenario where StarRocks is responsible for data computing and analysis, and the data lake is responsible for data storage, organization, and maintenance. Data lakes allow users to store data in open storage formats and use flexible schemas to produce reports on "single source of truth" for various BI, AI, ad-hoc, and reporting use cases. StarRocks fully leverages the advantages of its vectorization engine and CBO, significantly improving the performance of data lake analytics. #### Key ideas[​](#key-ideas "Direct link to Key ideas") * Open Data Formats: Supports a variety of data types, including JSON, Parquet, and Avro, facilitating the storage and processing of both structured and unstructured data. * Metadata Management: Implements a shared metadata layer, often utilizing formats like the Iceberg table format, to organize and govern data efficiently. * Governance and Security: Features robust built-in mechanisms for data security, privacy, and compliance, ensuring data integrity and trustworthiness. #### Advantages of Data Lakehouse architecture[​](#advantages-of-data-lakehouse-architecture "Direct link to Advantages of Data Lakehouse architecture") * Flexibility and Scalability: Seamlessly manages diverse data types and scales with the organization’s needs. * Cost-Effectiveness: Offers an economical alternative for data storage and processing, compared to traditional methods. * Enhanced Data Governance: Improves data control, management, and integrity, ensuring reliable and secure data handling. * AI and Analytics Readiness: Perfectly suited for complex analytical tasks, including machine learning and AI-driven data processing. #### StarRocks approach[​](#starrocks-approach "Direct link to StarRocks approach") The key things to consider are: * Standardizing the integration with catalog, or metadata services * Elastic scalability of compute nodes * Flexible caching mechanisms *** #### Catalogs[​](#catalogs "Direct link to Catalogs") StarRocks has two types of catalogs, internal and external. The internal catalog contains metadata for data stored within StarRocks databases. External catalogs are used to work with data stored externally, including the data managed by Hive, Iceberg, Delta Lake, and Hudi. There are many other external systems, links are in the More Information section at the bottom of the page. #### Compute node (CN) scaling[​](#compute-node-cn-scaling "Direct link to Compute node (CN) scaling") Separation of storage and compute reduces the complexity of scaling. Since the StarRocks compute nodes are only storing local cache, nodes can be added or removed based on load. #### Data cache[​](#data-cache "Direct link to Data cache") Cache on the compute nodes is optional. If your compute nodes are spinning up and down quickly based on quickly changing load patterns or your queries are often only on the most recent data it might not make sense to cache data. More information is in the [Catalog docs](https://docs.starrocks.io/docs/data_source/catalog/catalog_intro). --- ### dbt `dbt-starrocks` enables the use of `dbt` to transform data in StarRocks using dbt's modeling patterns and best practices. `dbt-starrocks` [GitHub repo](https://github.com/StarRocks/dbt-starrocks). Experimental feature [Advice on use of experimental features](https://docs.starrocks.io/docs/introduction/maturity.md) #### Supported features[​](#supported-features "Direct link to Supported features") | StarRocks >= 3.1 | StarRocks >= 3.4 | Feature | | ---------------- | ---------------- | --------------------------------- | | ✅ | ✅ | Table materialization | | ✅ | ✅ | View materialization | | ✅ | ✅ | Materialized View materialization | | ✅ | ✅ | Incremental materialization | | ✅ | ✅ | Primary Key Model | | ✅ | ✅ | Sources | | ✅ | ✅ | Custom data tests | | ✅ | ✅ | Docs generate | | ✅ | ✅ | Expression Partition | | ❌ | ❌ | Kafka | | ❌ | ✅ | Dynamic Overwrite | | `*` | ✅ | Submit task | | ✅ | ✅ | Microbatch (Insert Overwrite) | | ❌ | ✅ | Microbatch (Dynamic Overwrite) | `*` Verify the specific `submit task` support for your version, see [SUBMIT TASK](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/ETL/SUBMIT_TASK.md) #### Installation[​](#installation "Direct link to Installation") Install the StarRocks DBT adapter using pip: ```sh pip install dbt-starrocks ``` #### Verify Installation[​](#verify-installation "Direct link to Verify Installation") Verify the installation by checking the version: ```sh dbt --version ``` This should list `starrocks` under plugins. #### Configuration[​](#configuration "Direct link to Configuration") ##### Profiles[​](#profiles "Direct link to Profiles") Create or update `profiles.yml` with StarRocks-specific settings. ```yaml starrocks_project: target: dev outputs: dev: type: starrocks host: your-starrocks-host.com port: 9030 schema: your_database username: your_username password: your_password catalog: test_catalog ``` ##### Parameters[​](#parameters "Direct link to Parameters") ###### `type`[​](#type "Direct link to type") **Description**: The specific adapter to use, this must be set to `starrocks`
**Required?**: Required
**Example**: `starrocks` ###### `host`[​](#host "Direct link to host") **Description**: The hostname to connect to
**Required?**: Required
**Example**: `192.168.100.28` ###### `port`[​](#port "Direct link to port") **Description**: The port to use
**Required?**: Required
**Example**: `9030` ###### `catalog`[​](#catalog "Direct link to catalog") **Description**: Specify the catalog to build models into
**Required?**: Optional
**Example**: `default_catalog` ###### `schema`[​](#schema "Direct link to schema") **Description**: Specify the schema (database in StarRocks) to build models into
**Required?**: Required
**Example**: `analytics` ###### `username`[​](#username "Direct link to username") **Description**: The username to use to connect to the server
**Required?**: Required
**Example**: `dbt_admin` ###### `password`[​](#password "Direct link to password") **Description**: The password to use for authenticating to the server
**Required?**: Required
**Example**: `correct-horse-battery-staple` ###### `version`[​](#version "Direct link to version") **Description**: Let Plugin try to go to a compatible starrocks version
**Required?**: Optional
**Example**: `3.1.0` ###### `use_pure`[​](#use_pure "Direct link to use_pure") **Description**: set to "true" to use C extensions
**Required?**: Optional
**Example**: `true` ###### `is_async`[​](#is_async "Direct link to is_async") **Description**: "true" to submit suitable tasks as etl tasks.
**Required?**: Optional
**Example**: `true` ###### `async_query_timeout`[​](#async_query_timeout "Direct link to async_query_timeout") **Description**: Sets the `query_timeout` value when submitting a task to StarRocks
**Required?**: Optional
**Example**: `300` ##### Sources[​](#sources "Direct link to Sources") Create or update `sources.yml` ```yml sources: - name: your_source database: your_sr_catalog schema: your_sr_database tables: - name: your_table ``` If the catalog is not specified in the schema, it will default to the catalog defined in the profile. Using the profile from earlier, if catalog is not defined, the model will assume the source is located at `test_catalog.your_sr_database`. #### Materializations[​](#materializations "Direct link to Materializations") ##### Table[​](#table "Direct link to Table") Basic Table Configuration ```sql {{ config( materialized='table', engine='OLAP', keys=['id', 'name', 'created_date'], table_type='PRIMARY', distributed_by=['id'], buckets=3, partition_by=['created_date'], properties=[ {"replication_num": "1"} ] ) }} SELECT id, name, email, created_date, last_modified_date FROM {{ source('your_source', 'users') }} ``` #### Configuration Options[​](#configuration-options "Direct link to Configuration Options") * engine: Storage engine (default: `OLAP`) * keys: Columns that define the sort key * table\_type: Table model type * `PRIMARY`: Primary key model (supports upserts and deletes) * `DUPLICATE`: Duplicate key model (allows duplicate rows) * `UNIQUE`: Unique key model (enforces uniqueness) * `distributed_by`: Columns for hash distribution * `buckets`: Number of buckets for data distribution (leave empty for auto bucketing) * `partition_by`: Columns for table partitioning * `partition_by_init`: Initial partition definitions * `properties`: Additional StarRocks table properties #### Tables in External Catalogs[​](#tables-in-external-catalogs "Direct link to Tables in External Catalogs") ##### Read from External into StarRocks[​](#read-from-external-into-starrocks "Direct link to Read from External into StarRocks") This example creates a materialized table in StarRocks containing aggregated data from an external Hive catalog. tip Configure the external catalog if it does not already exist: ```sql CREATE EXTERNAL CATALOG `hive_external` PROPERTIES ( "hive.metastore.uris" = "thrift://127.0.0.1:8087", "type"="hive" ); ``` ```sql {{ config( materialized='table', keys=['product_id', 'order_date'], distributed_by=['product_id'], partition_by=['order_date'] ) }} -- Aggregate data from Hive external catalog into StarRocks table SELECT h.product_id, h.order_date, COUNT(*) as order_count, SUM(h.amount) as total_amount, MAX(h.last_updated) as last_updated FROM {{ source('hive_external', 'orders') }} h GROUP BY h.product_id, h.order_date ``` ##### Write to External[​](#write-to-external "Direct link to Write to External") ```sql {{ config( materialized='table', on_table_exists = 'replace', partition_by=['order_date'], properties={}, catalog='external_catalog', database='test_db' ) }} SELECT * FROM {{ source('iceberg_external', 'orders') }} ``` The configuration for materialization to external catalogs supports fewer options. `on_table_exist`s, `partition_by`, and `properties` are supported. If `catalog` and `database` are not set, the defaults from the profile will be used. ##### Incremental[​](#incremental "Direct link to Incremental") Incremental materializations are supported in StarRocks as well: ```sql {{ config( materialized='incremental', unique_key='id', table_type='PRIMARY', keys=['id'], distributed_by=['id'], incremental_strategy='default' ) }} SELECT id, user_id, event_name, event_timestamp, properties FROM {{ source('raw', 'events') }} {% if is_incremental() %} WHERE event_timestamp > (SELECT MAX(event_timestamp) FROM {{ this }}) {% endif %} ``` ###### Incremental Strategies[​](#incremental-strategies "Direct link to Incremental Strategies") `dbt-starrocks` supports multiple incremental strategies: 1. `append` (default): Simply appends new records without deduplication 2. `insert_overwrite`: Overwrites table partitions with insertion 3. `dynamic_overwrite`: Overwrites, creates, and writes table partitions For more information about which overwrite strategy to use, see the [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md) documentation. note Currently, incremental merge is not supported. #### Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") * Before using external catalogs in dbt, you must create them in StarRocks.[Catalog overview](https://docs.starrocks.io/docs/data_source/catalog/catalog_overview.md). * External sources should be accessed using the `{{ source('external_source_name', 'table_name' }}` macro. * `dbt seed` was not tested for external catalogs and is not currently supported. * In order for `dbt` to create models in external databases that do not currently exist, the location of the models must be set through properties. * External models need to define the location they are stored at. This location will be defined if the destination database exists and sets the location property. Otherwise, the location needs to be set. * We will currently only support creating external models in databases that already exist. --- ### DataGrip DataGrip supports querying both internal data and external data in StarRocks. tip [DataGrip docs](https://www.jetbrains.com/help/datagrip/getting-started.html) You can connect to StarRocks from DataGrip using either the native StarRocks JDBC driver (recommended) or the MySQL driver. #### Connect using the StarRocks JDBC driver (recommended)[​](#connect-using-the-starrocks-jdbc-driver-recommended "Direct link to Connect using the StarRocks JDBC driver (recommended)") The StarRocks JDBC driver provides accurate metadata discovery, which enables schema browsing, auto-complete, and table introspection in DataGrip. ##### Prerequisites[​](#prerequisites "Direct link to Prerequisites") Download the StarRocks JDBC driver JAR. See [StarRocks JDBC Driver](https://docs.starrocks.io/docs/integrations/JDBC_driver.md) for download instructions. ##### Steps[​](#steps "Direct link to Steps") 1. In DataGrip, go to **File** > **Data Sources** (or click the **Database** icon in the toolbar). 2. Click **+** and select **Driver and Data Source**. 3. In the **Drivers** tab, click **+** to create a new driver. * Set the **Name** to `StarRocks`. * Under **Driver Files**, click **+** and add the StarRocks JDBC driver JAR you downloaded. * Set the **Class** to `com.starrocks.cj.jdbc.Driver`. * Set the **URL template** to one of: ```text jdbc:starrocks://{host}:{port} jdbc:starrocks://{host}:{port}/{database} ``` * Click **OK** to save the driver. ![DataGrip - StarRocks driver configuration](/assets/images/IDE_datagrip_starrocks_driver-aaa1e7f2239f055d5b94bf51e1bd6c33.png) 4. Back in the **Data Sources** tab, click **+** and select the **StarRocks** driver you just created. 5. Configure the connection settings: * **Host**: the FE host IP address of your StarRocks cluster. * **Port**: the FE query port of your StarRocks cluster, for example, `9030`. * **Database**: the database to connect to, in the format `[{catalog_name}.]{database_name}`. Both internal and external catalogs are supported. If the catalog is omitted, `default_catalog` is used. * `catalog_name`: the name of the target catalog in your StarRocks cluster. * `database_name`: the name of the target database in your StarRocks cluster. * **User**: the username to log in to your StarRocks cluster, for example, `admin`. * **Password**: the password to log in to your StarRocks cluster. 6. Click **Test Connection** to verify the settings, then click **OK**. note After connecting, DataGrip only loads the database list. Tables are not fetched until you right-click the database in the sidebar and select **Refresh**. This is expected behavior. ##### Browse catalogs and databases in the sidebar[​](#browse-catalogs-and-databases-in-the-sidebar "Direct link to Browse catalogs and databases in the sidebar") After connecting, there are two ways to control what appears in the left sidebar: **Option 1 — Use the URL to scope directly to one catalog/database (simplest)** Edit the **URL** field to include `catalog.database` as shown in the tip above. DataGrip will display that catalog and database in the sidebar immediately, with no further configuration. **Option 2 — Select schemas in data source properties (multiple catalogs/databases)** 1. Double-click the data source (or right-click > **Properties**). 2. Go to the **Schemas** tab. 3. Check the catalogs and databases you want visible in the sidebar. 4. Click **OK**. DataGrip will refresh and display your selection. #### Connect using the MySQL driver[​](#connect-using-the-mysql-driver "Direct link to Connect using the MySQL driver") The MySQL driver is a fallback for the StarRocks JDBC driver. Create a data source in DataGrip. Note that you must select MySQL as the data source. ![DataGrip - 1](/assets/images/BI_datagrip_1-c1fed7e5af5d1cc0a678f2ccba06cac1.png) ![DataGrip - 2](/assets/images/BI_datagrip_2-b4dc9f2ad1041a911ff1baa18c2bdbdd.png) The parameters that you need to configure are described as follows: * **Host**: the FE host IP address of your StarRocks cluster. * **Port**: the FE query port of your StarRocks cluster, for example, `9030`. * **Authentication**: the authentication method that you want to use. Select **Username & Password**. * **User**: the username that is used to log in to your StarRocks cluster, for example, `admin`. * **Password**: the password that is used to log in to your StarRocks cluster. * **Database**: the data source that you want to access in your StarRocks cluster. The value of this parameter is in the `.` format. * `catalog_name`: the name of the target catalog in your StarRocks cluster. Both internal and external catalogs are supported. * `database_name`: the name of the target database in your StarRocks cluster. Both internal and external databases are supported. --- ### Dataphin Dataphin is a cloud-based output of the internal practices of Alibaba Group's OneData data governance methodology. It provides one-stop solution of data integration, construction, management and utilization throughout the entire lifecycle of big data, aming to helping enterprises significantly improve the level of data governance and build an enterprise-level data middle platform that is of high and reliable quality, convenience consumption, and safe and economical production. Dataphin provides a variety of computing platform support and scalable open capabilities to meet the platform technical architecture and specific requirements of enterprises in various industries. There are several ways to integrate Dataphin with StarRocks: * As the source or destination data source for data integration. Data can be read from StarRocks and push into other data sources, or data can be pull from other data sources and written into StarRocks. * As a source table (unbounded scan), dimension table (bounded scan), or result table (streaming sink and batch sink) of flink SQL and datastram development. * As a data warehouse or a datamart. StarRocks can be registered as a compute source, which can be used for SQL scripts development, scheduling, data quality detection, security identification, and other data research and governance tasks. #### Data integration[​](#data-integration "Direct link to Data integration") You can create StarRocks data sources and use StarRocks data sources as source databases or destination databases in offline integration tasks. The procedure is as follows: ##### Create a StarRocks data source[​](#create-a-starrocks-data-source "Direct link to Create a StarRocks data source") ###### Basic information[​](#basic-information "Direct link to Basic information") ![Create a StarRocks data source - 1](/assets/images/create_sr_datasource_1-7bb418ec67fade24f7247048233bae66.png) * **Name**: Required. Enter a data source name. It can only contain Chinese characters, letters, numbers, underscores (\_), and hyphens (-). It cannot exceed 64 characters in length. * **Data source code**: Optional. After you configure the data source code, you can use the `data source code.table` or `data source code.schema.table` format to reference the Flink SQL in the data source. If you want to automatically access the data source in the corresponding environment, use `${data source code}.table` or `${data source code}.schema.table` format access. > **NOTE** > > Currently, only MySQL, Hologres, and MaxCompute data sources are supported. * **Support scenerios**: The scenerios that data source can be applied in. * **Description**: Optional. You can enter a brief description of the data source. A maximum of 128 characters are allowed. * **Environment**: If the business data source distinguishes between production data source and development data source, choose **Prod and Dev**. If the business data source does not distinguish between production and development data sources, choose **Prod**. * **Tags**: You can select tags to label data sources. ###### Configuration information[​](#configuration-information "Direct link to Configuration information") ![Create a StarRocks data source - 2](/assets/images/create_sr_datasource_2-bfe94f501c2d905494c63d768f62c9c1.png) * **JDBC URL**: Required. The format is `jdbc:mysql://:/`. `host` is the IP address of the FE (Front End) host in the StarRocks cluster, `port` is the query port of FE, and `dbname` is the database name. * **Load URL**: Required. The format is `fe_ip:http_port;fe_ip:http_port`. `fe_ip` is the host of the FE (Front End), and `http_port` is the port of the FE. * **Username**: Required. The username of the database. * **Password**: Required. The password of the database. ###### Advanced settings[​](#advanced-settings "Direct link to Advanced settings") ![Create a StarRocks data source - 3](/assets/images/create_sr_datasource_3-873e78fb773c2c2c7a639482954d8250.png) * **connectTimeout**: the connectTimeout (in ms) of the database. The default value is 900000 milliseconds (15 minutes). * **socketTimeout**: the socketTimeout (in ms) of the database. The default value is 1800000 milliseconds (30 minutes). ##### Read data from StarRocks data sources and write data to other data sources[​](#read-data-from-starrocks-data-sources-and-write-data-to-other-data-sources "Direct link to Read data from StarRocks data sources and write data to other data sources") ###### Drag the StarRocks input component to the offline integration task canvas[​](#drag-the-starrocks-input-component-to-the-offline-integration-task-canvas "Direct link to Drag the StarRocks input component to the offline integration task canvas") ![Read data from StarRocks - 1](/assets/images/read_from_sr_datasource_1-7b39a62322439c4213c7ba1e35b7a2e9.png) ###### StarRocks input component configuration[​](#starrocks-input-component-configuration "Direct link to StarRocks input component configuration") ![Read data from StarRocks - 2](/assets/images/read_from_sr_datasource_2-4c7776a751f027d3666ab4533202a508.png) * **Step name**: Enter an appropriate name based on the scenario and location of the current component. * **Data source**: Select the StarRocks data source or project created on Dataphin. Read permission of the data source is required. If there is no satisfied data source, you can add a data source or apply for relevant permissions. * **Source table**: Select a single table or multiple tables with the same table structure as the input. * **Table**: Select the table in the StarRocks data source from the drop-down list. * **Split key**: Used with concurrency configuration. You can use a column in the source data table as the split key. It is recommended to use a primary key or an indexed column as the split key. * **Batch number**: The number of data records extracted in a batch. * **Input Filtering**: Optional. In the following two cases, you need to fill in the filter information: * If you want to filter a certain part of data. * If you need to incrementally append data on a daily basis or obtain full data, you need to fill in the date whose value is set as the system time of Dataphin console. For example, a transaction table in the StarRocks and the transaction creation date of it is set as `${bizdate}`. * **Output fields**: List the related fields based on the input table information. You can rename, remove, add, and move the fields again. In general, fields are renamed to increase the readability of downstream data or facilitate mapping of fields during output. Fields can be removed during the input stage because relevant fields are not needed in application scenarios. The order of fields is changed to ensure that you can effectively merge data or map output data by mapping fields with different names in the same line when multiple input data are merged or output at the downstream side. ###### Select and configure an output component as the destination data source[​](#select-and-configure-an-output-component-as-the-destination-data-source "Direct link to Select and configure an output component as the destination data source") ![Read data from StarRocks - 3](/assets/images/read_from_sr_datasource_3-6b37b3ae1c5dfcea15015c59d61f1f6f.png) ##### Read data from other data sources and write data to StarRocks data sources[​](#read-data-from-other-data-sources-and-write-data-to-starrocks-data-sources "Direct link to Read data from other data sources and write data to StarRocks data sources") ###### Configure the input component in the offline integration task, and select and configure the StarRocks output component as the destination data source[​](#configure-the-input-component-in-the-offline-integration-task-and-select-and-configure-the-starrocks-output-component-as-the-destination-data-source "Direct link to Configure the input component in the offline integration task, and select and configure the StarRocks output component as the destination data source") ![Write data to StarRocks - 1](/assets/images/write_to_sr_datasource_1-8e19eef8d08e3d0e50e62d022e91a3f2.png) ###### Configure StarRocks output component[​](#configure-starrocks-output-component "Direct link to Configure StarRocks output component") ![Write data to StarRocks - 2](/assets/images/write_to_sr_datasource_2-2553b62350851e43640343c494c78947.png) * **Step Name**: enter an appropriate name based on the scenario and location of the current component. * **Data Source**: Select the Dataphin data source or project created in the StarRocks. The data source that the configuration personnel have the synchronous write permission. If the data source is not satisfied, you can add a data source or apply for relevant permissions. * **Table**: Select the table in the StarRocks data source from the drop-down list. * **Generate Target Table by One Click**: If you have not created a target table in StarRocks data source, you can automatically obtain the name, type, and remarks of the fields read from the upstream, and generate a table creation statement. Click to generate a target table with one click. * **CSV import column delimiter**: Use StreamLoad CSV to import. You can configure the CSV import column delimiter. Default value `\t`. Do not specify the default value here. If the data itself contains `\t`, you must use other characters as delimiters. * **CSV import row delimiter**: Use StreamLoad CSV to import. You can configure the CSV import row delimiter. Default value: `\n`. Do not specify the default value here. If the data itself contains `\n`, you must use other characters as delimiters. * **Parse Solution**: Optional. It is some special processing before or after the data is written. The preparation statement is executed before the data is written to the StarRocks data source, and the completion Statement is executed after the data is written. * **Field Mapping**: You can manually select fields for mapping, or use name-based or position-based mapping to process multiple fields at a time based on the fields from upstream input and those in the destination table. #### Real-time Development[​](#real-time-development "Direct link to Real-time Development") ##### Brief introduction[​](#brief-introduction "Direct link to Brief introduction") StarRocks is a fast and scalable real-time analysis Database. It is commonly used in real-time computing to read and write data to meet the needs of real-time data analysis and query. It is widely used in enterprise real-time computing scenarios. It can be used in real-time business monitoring and analysis, real-time user behavior analysis, real-time advertising bidding system, real-time risk control, anti-fraud, real-time monitoring and early warning and other application scenarios. By analyzing and querying data in real time, enterprises can quickly understand business conditions, optimize decisions, provide better services and protect their interests. ##### StarRocks Connector[​](#starrocks-connector "Direct link to StarRocks Connector") The StarRocks connector supports the following information: | **Category** | **Facts and figures** | | ------------------------------------------------------ | ------------------------------------------- | | Supported types | Source Table, dimension table, result table | | Running mode | Stream mode and batch mode | | Data format | JSON and CSV | | Special Metrics | None | | API type | Datastream and SQL | | Support updating or deleting data in the result table? | Yes | ##### How to use it?[​](#how-to-use-it "Direct link to How to use it?") Dataphin supports StarRocks data sources as read and write targets for realtime compute. You can create StarRocks meta tables and use them for realtime compute tasks: ###### Create StarRocks meta table[​](#create-starrocks-meta-table "Direct link to Create StarRocks meta table") 1. Go to **Dataphin** > **R & D** > **Develop** > **Tables**. 2. Click **Create** to select a real-time compute table. ![Create StarRocks meta table - 1](/assets/images/create_sr_metatable_1-4d12085e290b9653a921ef8c71f4a824.png) * **Table type**: Select **Metatable**. * **Metatable**: Input the name of the meta table. The name immutable. * **Datasource**: Select a StarRocks data source. * **Directory**: Select the directory where you want to create a table. * **Description**: Optional. ![Create StarRocks meta table - 2](/assets/images/create_sr_metatable_2-3a638b34be7589e62d3250478a17c032.png) 3. After creating a meta table, you can edit the meta table, including modifying data sources, source tables, meta table fields, and configuring meta table parameters. ![Edit StarRocks meta table](/assets/images/edit_sr_metatable_1-f1098e912569b1f58fd020bfe32a06b0.png) 4. Submit the meta table. ###### Create Flink SQL task to write data from Kafka to StarRocks in real time[​](#create-flink-sql-task-to-write-data-from-kafka-to-starrocks-in-real-time "Direct link to Create Flink SQL task to write data from Kafka to StarRocks in real time") 1. Go to **Dataphin** > **R & D** > **Develop** > **Computing Tasks**. 2. Click **Create Flink SQL task**. ![Create Flink SQL task - Step 2](/assets/images/create_flink_task_step2-679d17bc320ed44c45b106bc05acbb7c.png) 3. Edit Flink SQL code and precompile It. Kafka meta table is used as an input table and StarRocks meta table as an output table. ![Create Flink SQL task - Step 3 - 1](/assets/images/create_flink_task_step3-1-0524f78767a6e4bf233f19e5020f7a34.png) ​ ![Create Flink SQL task - Step 3 - 2](/assets/images/create_flink_task_step3-2-2484f52fca424ae0f8981fa8fec01587.png) 4. After the precompilation is successful, you can debug and submit the code. 5. Testing in the development environment can be performed by printing logs and writing test tables. The test tables can be set in Meta Tables > Properties > debugging test configurations. ![Create Flink SQL task - Step 5 - 1](/assets/images/create_flink_task_step5-1-5a658adaaaa0bd9341dc172f4cf90254.png) ![Create Flink SQL task - Step 5 - 2](/assets/images/create_flink_task_step5-2-7db3c3caf5a4af70894defecb9b28013.png) 6. After a task in the development environment runs normally, you can publish the task and the meta table used to the production environment. ![Create Flink SQL task - Step 6](/assets/images/create_flink_task_step6-97dfc5403fe84086b4eb54248a51b694.png) 7. Start a task in the production environment to write data from Kafka to StarRocks in real time. You can view the status and logs of each metric in the running analysis to learn about the task running status, or configure monitoring alerts for the task. ![Create Flink SQL task - Step 7 - 1](/assets/images/create_flink_task_step7-1-8637be96949330a35d93af698be5f865.png) ![Create Flink SQL task - Step 7 - 2](/assets/images/create_flink_task_step7-2-9bd69161cb2c4e153bdfc9e1f28fd254.png) #### Data warehouse or data mart[​](#data-warehouse-or-data-mart "Direct link to Data warehouse or data mart") ##### Prerequisites[​](#prerequisites "Direct link to Prerequisites") * The StarRocks version is 3.0.6 or later. * Dataphin is installed and the Dataphin version is 3.12 or later. * Statistics collection must be enabled. After the StarRocks is installed, collection is enabled by default. For more information, see [Gather statistics for CBO](https://docs.starrocks.io/docs/using_starrocks/Cost_based_optimizer.md). * StarRocks internal catalog (default catalog) is supported, and external catalog is not supported. ##### Connection configuration[​](#connection-configuration "Direct link to Connection configuration") ###### Metadata warehouse settings[​](#metadata-warehouse-settings "Direct link to Metadata warehouse settings") Dataphin can present and display information based on metadata, including table usage information and metadata changes. You can use StarRocks to process and calculate metadata. Therefore, you need to initialize the metadata Computing Engine (metadata warehouse) before using it. The procedure is as follows: 1. Use an administrator account to log on to Dataphin metadata warehouse tenant 2. Go to Administration > System > Metadata Warehouse Configuration a. Click Start b. Select StarRocks c. Configure the parameters. After passing the test connection, click next. d. Complete meta warehouse initialization ![Metadata warehouse settings](/assets/images/metadata_warehouse_settings_1-bd97bd9aecba5212b07bb2e9432bdf59.png) Parameters are described as follows: * **JDBC URL**: JDBC connection string, which is divided into two parts: * Part I: The format is `jdbc:mysql://:/`. `Host` is the IP address of the FE host in the StarRocks cluster. `Port` is the query Port of FE. Default value: `9030`. * Part Two: format is `database? key1 = value1 & key2 = value2`, where `database` is the name of the StarRocks database used for metadata calculation, which is required. The parameter after '?' is optional. * **Load URL**: The format is `fe_ip:http_port;fe_ip:http_port`. `fe_ip` is host of FE (Front End), and `http_port` is th port of FE. * **Username**: The username used to connect to the StarRocks. The user need to have read and write permissions on the database specified in the JDBC URL, and must have access permissions on the following databases and tables: * All table in Information Schema * *statistics*.column\_statistics * *statistics*.table\_statistic\_v1 * **Password**: the password of the link of StarRocks. * **Meta Project**: The name of the project used for metadata processing in Dataphin. It is only used within the Dataphin system. We recommend that you use `dataphin_meta` as the project name. ###### Create StarRocks project and start data development[​](#create-starrocks-project-and-start-data-development "Direct link to Create StarRocks project and start data development") To start data development, follow these steps: 1. Computing settings. 2. Create StarRocks computing source. 3. Create a project. 4. Create StarRocks SQL task. ###### Computing settings[​](#computing-settings "Direct link to Computing settings") The computing settings set the computing engine type and cluster address of the tenant. The detailed steps are as follows: 1. Log on to the Dataphin as a System Administrator or Super Administrator. 2. Go to **Administration** > **System** > **Computation Configuration**. 3. Select **StarRocks** and click **Next**. 4. Enter the JDBC URL and verify it. The format of the JDBC URL is `jdbc:mysql://:/`. `Host` is the IP address of the FE Host in the StarRocks cluster. `Port` is the query Port of FE. Default value: `9030`. ###### StarRocks Computing source[​](#starrocks-computing-source "Direct link to StarRocks Computing source") Computing source is a concept of Dataphin. Its main purpose is to bind and register Dataphin project space with StarRocks storage computing space (database). You must create a computing source for each project. The detailed steps are as follows: 1. Log on to the Dataphin as a system administrator or Super Administrator. 2. Go to **Planning** > **Engine**. 3. Click **Add Computing Engine** in the upper-right corner to create a computing source. The detailed configuration information is as follows: 1. **Essential information** ![Create compute engine - 1](/assets/images/create_compute_engine_1-94a980eb79c9379f42d84c51235c1620.png) * **Computing Engine Type**: Select **StarRocks**. * **Computing Engine Name**: We recommend that you use the same name as the project to be created. For development projects, add the suffix `_dev`. * **Description**: Optional. Enter the description of the computing source. 2. **Configuration information** ![Create compute engine - 2](/assets/images/create_compute_engine_2-47f0deea152a1809e302e97315803c43.png) * **JDBC URL**: The format is `jdbc:mysql://:/`. `Host` is the IP address of the FE Host in the StarRocks cluster. `Port` is the query Port of FE. Default value: `9030`. * **Load URL**: The format is `fe_ip:http_port;fe_ip:http_port`. `fe_ip` is host of FE (Front End), and `http_port` is th port of FE. * **Username**: The username used to connect to the StarRocks. * **Password**: The password of StarRocks. * **Task Resource Group**: you can specify different StarRocks resource groups for tasks with different priorities. When you select do not specify Resource Group, the StarRocks engine determines the resource group to be executed. When you select specify Resource Group, tasks with different priorities are assigned to the specified resource group by the Dataphin.If a resource group is specified in the code of an SQL task or in the materialized configuration of a logical table, the configuration of the Resource Group of the compute source task is ignored when the task is executed. ![Create compute engine - 3](/assets/images/create_compute_engine_3-df002c75045eb66b55afd092cff0da66.png) ###### Dataphin project[​](#dataphin-project "Direct link to Dataphin project") After you create a computing source, you can bind it to a Dataphin project. Dataphin project manages project members, StarRocks storage and computing space, and manages and maintains computing tasks. To create a Dataphin project, follow these steps: 1. Log on to the Dataphin as a System Administrator or Super Administrator. 2. Go to **Planning** > **Project Management**. 3. Click **Create project** in the upper-right corner to create a project. 4. Enter the basic information and select the StarRocks engine created in the previous step from the offline engine. 5. Click **Create**. ###### StarRocks SQL[​](#starrocks-sql "Direct link to StarRocks SQL") After you create a project, you can create a StarRocks SQL task to perform DDL or DML operations on the StarRocks. The detailed steps are as follows: 1. Go to **R & D** > **Develop**. 2. Click '+' in the upper-right corner to create StarRocks SQL task. ![Configure Dataphin project - 1](/assets/images/configure_dataphin_project_1-06d5728dff9a48b8fb5829e0e4f613c2.png) 3. Enter the name and scheduling type to create an SQL task. 4. Enter SQL in the editor to start DDL and DML operations on StarRock. ![Configure Dataphin project - 2](/assets/images/configure_dataphin_project_2-f6f531faebb06c4c0416385b1b8e2968.png) --- ### DBeaver DBeaver is a SQL client software application and a database administration tool, which offers a helpful assistant that walks you through the process of connecting to a database. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") Make sure that you have installed DBeaver. You can download the DBeaver Community edition at [https://dbeaver.io](https://dbeaver.io/) or the DBeaver PRO edition at [https://dbeaver.com](https://dbeaver.com/). #### Integration[​](#integration "Direct link to Integration") Follow these steps to connect to a database: 1. Launch DBeaver. 2. Click the plus sign (**+**) icon in the upper-left corner of the DBeaver window or choose **Database** > **New Database Connection** in the menu bar to access the assistant. ![DBeaver - Access the assistant](/assets/images/IDE_dbeaver_1-0cdfa5e0fbf065c8520b49fb33113c84.png) ![DBeaver - Access the assistant](/assets/images/IDE_dbeaver_2-0d9ee4b2c44a326772aa84f8e2ee2a36.png) 3. Select the StarRocks driver. In the **Select your database** step, you are presented with a list of available drivers. Search for **StarRocks** in the search bar, or click **Analytical** in the left-side pane to locate it. Then, double-click the **StarRocks** icon. note If your version of DBeaver does not include the StarRocks driver, you can use the **MySQL** driver as a fallback. ![DBeaver - Select your database](/assets/images/IDE_dbeaver_3-0c417b72eca77a125da29f31e1a1a826.png) 4. Configure the connection to the database. In the **Connection Settings** step, go to the **Main** tab and configure the following essential connection settings: * **Host**: the FE host IP address of your StarRocks cluster. * **Port**: the FE query port of your StarRocks cluster, for example, `9030`. * **Database/Schema**: the target database in your StarRocks cluster. * **Username**: the username that is used to log in to your StarRocks cluster, for example, `admin`. * **Password**: the password that is used to log in to your StarRocks cluster. note Starting with DBeaver 26.0.5, multi-catalog browsing is supported when using the StarRocks driver, allowing you to explore all catalogs in your cluster without specifying a database. ![DBeaver - Connection Settings - Main tab](/assets/images/IDE_dbeaver_4-334f8e0c979e76a1a61724373b39d598.png) You can also view and edit the properties of the StarRocks driver on the **Driver properties** tab if necessary. To edit a specific property, click the row in the **Value** column for that property. ![DBeaver - Connection Settings - Driver properties tab](/assets/images/IDE_dbeaver_5-9d57615f98709513e124a2b075dd867c.png) 5. Test the connection to the database. Click **Test Connection** to verify the accuracy of the connection settings. A dialog box displaying the StarRocks driver's information appears. Click **OK** in the dialog box to confirm the information. After you have successfully configured the connection settings, click **Finish** to complete the process. ![DBeaver - Test Connection](/assets/images/IDE_dbeaver_6-9b86f85c2e17114132b9571cca316ee8.png) 6. Connect to the database. After the connection is established, you can view it in the left-side database connection tree and DBeaver can effectively connect to the database. ![DBeaver - Connect database](/assets/images/IDE_dbeaver_7-e1a369e03c409939f48ea4ce85944ca0.png) --- ### Jupyter This guide describes how to integrate your StarRocks cluster with [Jupyter](https://jupyter.org/), the latest web-based interactive development environment for notebooks, code, and data. All of this is made possible via [JupySQL](https://jupysql.readthedocs.io/) which allows you to run SQL and plot large datasets in Jupyter via a %sql, %%sql, and %sqlplot magics. You can use JupySQL on top of Jupyter to run queries on top of StarRocks. Once the data is loaded into the cluster, you can query and visualize it via SQL plotting. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before getting started, you must have the following software installed locally: * [JupySQL](https://jupysql.readthedocs.io/en/latest/quick-start.html): `pip install jupysql` * Jupyterlab: `pip install jupyterlab` * [SKlearn Evaluation](https://github.com/ploomber/sklearn-evaluation): `pip install sklearn-evaluation` * Python * pymysql: `pip install pymysql` > **NOTE** > > Once you have the above requirements fulfilled, you can open Jupyter lab simply by calling `jupyterlab` - this will open the notebook interface. If Jupyter lab is already running in a notebook, you can simply run the cell bellow to get the dependencies. ```python # Install required packages. %pip install --quiet jupysql sklearn-evaluation pymysql ``` > **NOTE** > > You may need to restart the kernel to use updated packages. ```python import pandas as pd from sklearn_evaluation import plot # Import JupySQL Jupyter extension to create SQL cells. %load_ext sql %config SqlMagic.autocommit=False ``` **You will need to make sure your StarRocks instance is up and reachable for the next stages.** > **NOTE** > > You will need to adjust the connection string according to the instance type you are trying to connect to (url, user, and password). The example below uses a local instance. #### Connecting to StarRocks via JupySQL[​](#connecting-to-starrocks-via-jupysql "Direct link to Connecting to StarRocks via JupySQL") In this example, a docker instance is used, and that is reflecting the data in the connection string. The `root` user is used to connect to the local StarRocks instance, create a database, and check that data can actually be read from and written into the table. ```python %sql mysql+pymysql://root:@localhost:9030 ``` Create and use that JupySQL database: ```python %sql CREATE DATABASE jupysql; ``` ```python %sql USE jupysql; ``` Create a table: ```python %%sql CREATE TABLE tbl(c1 int, c2 int) distributed by hash(c1) properties ("replication_num" = "1"); INSERT INTO tbl VALUES (1, 10), (2, 20), (3, 30); SELECT * FROM tbl; ``` #### Saving and loading queries[​](#saving-and-loading-queries "Direct link to Saving and loading queries") Now after you create a database, you can write some sample data into it and query it. JupySQL allows you to break queries into multiple cells, simplifying the process of building large queries. You can write complex queries, save them, and execute them when needed, in a similar manner to CTEs in SQL. ```python # This is pending for the next JupySQL release. %%sql --save initialize-table --no-execute CREATE TABLE tbl(c1 int, c2 int) distributed by hash(c1) properties ("replication_num" = "1"); INSERT INTO tbl VALUES (1, 1), (2, 2), (3, 3); SELECT * FROM tbl; ``` > **NOTE** > > `--save` stores the query, not the data. Note that we are using `--with;`, this will retrieve previously saved queries, and prepend them (using CTEs). Then, we save the query in `track_fav`. #### Plotting directly on StarRocks[​](#plotting-directly-on-starrocks "Direct link to Plotting directly on StarRocks") JupySQL comes with a few plots by default, allowing you to visualize the data directly in SQL. You can use a bar plot to visualize the data in your newly created table: ```python top_artist = %sql SELECT * FROM tbl top_artist.bar() ``` Now you have a new bar plot without any extra code. You can run SQL directly from your notebook via JupySQL (by ploomber). This adds lots of possibilities around StarRocks for data scientists and engineers. In case that you got stuck or need any support, please reach out to us via Slack. --- ### Marimo Integrate your StarRocks cluster with [Marimo](https://marimo.io/), a reactive Python notebook built for reproducibility and interactivity. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") First, start by installing Marimo and setting up a notebook according to the [Marimo quickstart documentation](https://github.com/marimo-team/marimo#quickstart). You will also need the following packages: ```bash pip install starrocks sqlalchemy pandas ``` #### Connecting to StarRocks[​](#connecting-to-starrocks "Direct link to Connecting to StarRocks") Use [SQLAlchemy](https://www.sqlalchemy.org/) to create a connection engine. The connection string format is: ```text starrocks://username:password@host:port/database ``` ```python import marimo as mo import sqlalchemy as sa engine = sa.create_engine("starrocks://username:password@:9030") ``` Replace `` with your StarRocks FE host. #### Using Marimo UI for credentials[​](#using-marimo-ui-for-credentials "Direct link to Using Marimo UI for credentials") To avoid hardcoding credentials, use Marimo's interactive UI elements to collect them at runtime. **Cell 1** — render input fields: ```python user = mo.ui.text(label="Username") pw = mo.ui.text(label="Password", kind="password") mo.hstack([user, pw]) ``` **Cell 2** — create the engine using the entered values: ```python engine = sa.create_engine( f"starrocks://{user.value}:{pw.value}@:9030" ) ``` #### Querying StarRocks[​](#querying-starrocks "Direct link to Querying StarRocks") With the engine established, use pandas to run queries: ```python import pandas as pd df = pd.read_sql("SELECT * FROM my_database.my_table LIMIT 100", engine) mo.ui.table(df) ``` ![Marimo notebook connected to StarRocks](/assets/images/marimo_starrocks-5e392b2109c487ad9d31bf31b0d8a1e2.png) note Multi-catalog support requires Marimo version 0.22.5 or later. --- ### StarRocks JDBC Driver StarRocks provides a native JDBC driver that enables direct connectivity from any JDBC-compatible client, IDE, or application. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") * Java 8 or later * A running StarRocks cluster #### Download[​](#download "Direct link to Download") The StarRocks JDBC driver is available on [Maven Central](https://central.sonatype.com/artifact/com.starrocks/starrocks-connector-j). You can download the JAR directly from Maven Central, or add it as a dependency in your project using the instructions below. ##### Download via Maven CLI[​](#download-via-maven-cli "Direct link to Download via Maven CLI") If you have Maven installed, you can download the JAR without creating a project: ```bash mvn dependency:get -Dartifact=com.starrocks:starrocks-connector-j:1.1.1 ``` The JAR will be saved to your local Maven repository at: ```text ~/.m2/repository/com/starrocks/starrocks-connector-j/1.1.1/starrocks-connector-j-1.1.1.jar ``` #### Use the JAR in your project[​](#use-the-jar-in-your-project "Direct link to Use the JAR in your project") ##### Maven[​](#maven "Direct link to Maven") Add the following dependency to your `pom.xml`: ```xml com.starrocks starrocks-connector-j 1.1.1 ``` ##### Gradle[​](#gradle "Direct link to Gradle") Add the following dependency to your `build.gradle`: ```groovy implementation 'com.starrocks:starrocks-connector-j:1.1.1' ``` ##### Plain Java[​](#plain-java "Direct link to Plain Java") Download the JAR from [Maven Central](https://central.sonatype.com/artifact/com.starrocks/starrocks-connector-j) and add it to the classpath when compiling and running: ```bash javac -cp starrocks-connector-j-.jar MyApp.java java -cp .:starrocks-connector-j-.jar MyApp ``` #### Connection URL format[​](#connection-url-format "Direct link to Connection URL format") ```text jdbc:starrocks://:/. ``` | Parameter | Description | | --------------- | --------------------------------------------------------------------------------------------------------- | | `fe_host` | The FE host IP address of your StarRocks cluster. | | `fe_query_port` | The FE query port, default `9030`. | | `catalog` | The catalog to connect to. Use `default_catalog` for internal tables, or the name of an external catalog. | | `database` | The database within the catalog. | **Example:** ```text jdbc:starrocks://192.168.1.1:9030/default_catalog.my_database ``` #### Connection properties[​](#connection-properties "Direct link to Connection properties") | Property | Description | | ---------- | ---------------------------------------------------------- | | `user` | The username to log in to StarRocks, for example, `admin`. | | `password` | The password to log in to StarRocks. | #### Metadata discovery[​](#metadata-discovery "Direct link to Metadata discovery") The StarRocks JDBC driver supports standard JDBC metadata APIs (`DatabaseMetaData`), which allow tools to introspect catalogs, schemas, tables, and columns. This enables IDE features such as schema browsing, auto-complete, and table introspection to work out of the box. #### Example: connect from Java[​](#example-connect-from-java "Direct link to Example: connect from Java") ```java import java.sql.Connection; import java.sql.DriverManager; import java.sql.ResultSet; import java.sql.Statement; public class StarRocksExample { public static void main(String[] args) throws Exception { String url = "jdbc:starrocks://192.168.1.1:9030/default_catalog.my_database"; Connection conn = DriverManager.getConnection(url, "admin", "password"); try (Statement stmt = conn.createStatement(); ResultSet rs = stmt.executeQuery("SELECT * FROM my_table LIMIT 10")) { while (rs.next()) { System.out.println(rs.getString(1)); } } conn.close(); } } ``` --- ### BladePipe #### Introduction[​](#introduction "Direct link to Introduction") BladePipe is a **real-time end-to-end data replication tool**, moving data between **30+** databases, message queues, search engines, caching systems, real-time data warehouses, data lakes and more, with **ultra-low latency**. It features efficiency, stability and scalability, compatibility with diverse database engines, one-stop management, enhanced security, and complex data transformation. BladePipe helps to break down data silos, increasing the value of data. ![image.png](/assets/images/3.11-1-7e9c0c4be79fb00b9fa73f997d79dbf1.png) #### Features[​](#features "Direct link to Features") BladePipe presents a visualized management interface, allowing you to easily create DataJobs to achieve **schema migration, data migration, synchronization, verification and correction**, and more. In addition, it supports more refined and customized configurations by parameter settings. Now BladePipe supports data movement from the following source DataSources to StarRocks: | Source DataSource | Schema Migration | Data Migration | Data Sync | Verification & Correction | | --------------------------- | ---------------- | -------------- | --------- | ------------------------- | | MySQL/MariaDB/AuroraMySQL | Yes | Yes | Yes | Yes | | Oracle | Yes | Yes | Yes | Yes | | PostgreSQL/AuroraPostgreSQL | Yes | Yes | Yes | Yes | | SQL Server | Yes | Yes | Yes | Yes | | Kafka | No | No | Yes | No | | AutoMQ | No | No | Yes | No | | TiDB | Yes | Yes | Yes | Yes | | Hana | Yes | Yes | Yes | Yes | | PolarDB for MySQL | Yes | Yes | Yes | Yes | | Db2 | Yes | Yes | Yes | Yes | info For more information on supported functions and parameter settings, refer to [BladePipe Connections](https://www.bladepipe.com/docs/dataMigrationAndSync/connection/mysql2/?target=StarRocks). #### Installation[​](#installation "Direct link to Installation") [BladePipe](https://www.bladepipe.com/docs/quick/quick_start/) #### Example[​](#example "Direct link to Example") Taking a MySQL instance as an example, the following section describes how to migrate data from MySQL to StarRocks. ##### Add DataSources[​](#add-datasources "Direct link to Add DataSources") 1. Log in to the [BladePipe Cloud](https://cloud.bladepipe.com/). Click **DataSource** > **Add DataSource**. 2. Select **StarRocks** as the **Type**, and fill in the setup form. * **Client Address**：The port StarRocks provided to MySQL Client. BladePipe queries the metadata in databases via it. * **Account**: The user name of the StarRocks database. The INSERT privilege is required to write data to StarRocks. Follow the instruction provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the database user the INSERT privilege. * **Http Address**：The port used to receive the request from BladePipe to write data to StarRocks. ![image.png](/assets/images/3.11-2-33c064f79fe30ea39142c3766da5f0e0.png) 3. Click **Test Connection**. After the connection is successful, click **Add DataSource** to add the DataSource. 4. Add a MySQL DataSource following the above steps. ##### Create DataJob[​](#create-datajob "Direct link to Create DataJob") 1. Click **DataJob** > [**Create DataJob**](https://www.bladepipe.com/docs/operation/job_manage/create_job/create_full_incre_task/). 2. Select the source and target DataSources, and click **Test Connection** to ensure the connection to the source and target DataSources are both successful. ![image.png](/assets/images/3.11-3-a4b87453ed4bcc144b6f21b1aa4ed831.png) 3. Select **Incremental** for DataJob Type, together with the **Full Data** option. ![image.png](/assets/images/3.11-4-2c50b329100f9ee87f65566814faedd4.png) 4. Select the tables to be replicated. **Note that the target StarRocks tables automatically created after Schema Migration have primary keys, so source tables without primary keys are not supported currently**. ![image.png](/assets/images/3.11-5-12f941b5b9ada9b6abc103dea10c5a92.png) 5. Select the columns to be replicated. ![image.png](/assets/images/3.11-6-781944480bf1b87d1f4250b6591efe15.png) 6. Confirm the DataJob creation. ![image.png](/assets/images/3.11-7-dfe068ebe9474e54bbeed00ba926f6a6.png) 7. The DataJob runs automatically. BladePipe will automatically run the following DataTasks: * **Schema Migration**: The schemas of the source tables will be migrated to the target instance. * **Full Data**: All existing data of the source tables will be fully migrated to the target instance. * **Incremental**: Ongoing data changes will be continuously synchronized to the target instance (with latency less than a minute). ![image.png](/assets/images/3.11-8-9c11d378ec4ac2112b33039d3430c931.png) --- ### DataX writer #### Introduction[​](#introduction "Direct link to Introduction") The StarRocksWriter plugin allows writing data to StarRocks' destination table. Specifically,StarRocksWriter imports data to StarRocks in CSV or JSON format via [Stream Load](https://docs.starrocks.io/docs/loading/StreamLoad.md), and internally caches and bulk imports the data read by `reader` to StarRocks for better write performance. The overall data flow is `source -> Reader -> DataX channel -> Writer -> StarRocks`. [Download the plugin](https://github.com/StarRocks/DataX/releases) Please go to `https://github.com/alibaba/DataX` to download the full package of DataX, and put the starrockswriter plugin into the `datax/plugin/writer/` directory. Use the following command to test: `python datax.py --jvm="-Xms6G -Xmx6G" --loglevel=debug job.json` #### Function Description[​](#function-description "Direct link to Function Description") ##### Sample Configuration[​](#sample-configuration "Direct link to Sample Configuration") Here is a configuration file for reading data from MySQL and loading it to StarRocks. ```json { "job": { "setting": { "speed": { "channel": 1 }, "errorLimit": { "record": 0, "percentage": 0 } }, "content": [ { "reader": { "name": "mysqlreader", "parameter": { "username": "xxxx", "password": "xxxx", "column": [ "k1", "k2", "v1", "v2" ], "connection": [ { "table": [ "table1", "table2" ], "jdbcUrl": [ "jdbc:mysql://127.0.0.1:3306/datax_test1" ] }, { "table": [ "table3", "table4" ], "jdbcUrl": [ "jdbc:mysql://127.0.0.1:3306/datax_test2" ] } ] } }, "writer": { "name": "starrockswriter", "parameter": { "username": "xxxx", "password": "xxxx", "database": "xxxx", "table": "xxxx", "column": ["k1", "k2", "v1", "v2"], "preSql": [], "postSql": [], "jdbcUrl": "jdbc:mysql://172.28.17.100:9030/", "loadUrl": ["172.28.17.100:8030", "172.28.17.100:8030"], "loadProps": {} } } } ] } } ``` #### Starrockswriter Parameter Description[​](#starrockswriter-parameter-description "Direct link to Starrockswriter Parameter Description") * **username** * Description: The user name of the StarRocks database * Required: Yes * Default value: none * **password** * Description: The password for the StarRocks database * Required: Yes * Default: None * **database** * Description: The name of the database for the StarRocks table. * Required: Yes * Default: None * **table** * Description: The name of the table for the StarRocks table. * Required: Yes * Default: None * **loadUrl** * Description: The address of the StarRocks FE for stream load, can be multiple FE addresses, in the form of `fe_ip:fe_http_port`. * Required: yes * Default value: none * **column** * Description: The fields of the destination table **that need to be written to the data**, with the columns separated by commas. Example: "column": \["id", "name", "age"]. > **column configuration item must be specified and cannot be left blank.** Note: We strongly discourage you from leaving it empty, because your job may run incorrectly or fail when you change the number of columns, type, etc. of the destination table. The configuration items must be in the same order as the querySQL or column in the reader. * Required: Yes * Default value: No * **preSql** * Description: The standard statement will be executed before writing data to the destination table. * Required: No * Default: No * **jdbcUrl** * Description: JDBC connection information of the destination database for executing `preSql` and `postSql`. * Required: No * Default: No * **loadProps** * Description: Request parameters for StreamLoad, refer to the StreamLoad introduction page for details. * Required: No * Default value: No #### Type conversion[​](#type-conversion "Direct link to Type conversion") By default, incoming data is converted to strings, with `t` as column separator and `n` as row separator, to form `csv` files for StreamLoad import . To change the column separator, configure `loadProps` properly. ```json "loadProps": { "column_separator": "\\x01", "row_delimiter": "\\x02" } ``` To change the import format to `json`, configure `loadProps` properly. ```json "loadProps": { "format": "json", "strip_outer_array": true } ``` > The `json` format is for the writer to import data to StarRocks in JSON format. #### About time zone[​](#about-time-zone "Direct link to About time zone") If the source tp library is in another time zone, when executing datax.py, add the following parameter after the command line ```json "-Duser.timezone=xx" ``` e.g. If DataX imports Postgrest data and the source library is in UTC time, add the parameter "-Duser.timezone=GMT+0" to startup. --- ### loading_tools_integration ### Load data using tools StarRocks and its ecosystem partners offer the following tools to help you seamlessly integrate StarRocks with external databases. #### [SMT](https://docs.starrocks.io/docs/integrations/loading_tools/SMT.md)[​](#smt "Direct link to smt") SMT (StarRocks Migration Tool) is a data migration tool provided by StarRocks, designed to optimize complex data loading pipelines: source databases (such as MySQL, Oracle, PostgreSQL) ---> Flink ---> destination StarRocks clusters. Its main functions are as follows: * Simplifies table creation in StarRocks: Generates statements to create tables in StarRocks based on information from external databases and the target StarRocks cluster. * Simplifies the full or incremental data synchronization process in the data pipeline: Generates SQL statements that can be run in Flink's SQL client to submit Flink jobs for synchronizing data. The following flowchart illustrates the process of loading data from the source database MySQL through Flink into StarRocks. ![img](/assets/images/load_tools-8e88eab0e5c9d2a228fcd668c63bc3e1.png) #### [DataX](https://docs.starrocks.io/docs/integrations/loading_tools/DataX-starrocks-writer.md)[​](#datax "Direct link to datax") DataX is a tool for offline data synchronization, and is open-sourced by Alibaba. DataX can synchronize data between various heterogeneous data sources, including relational databases (MySQL, Oracle, etc.), HDFS, and Hive. DataX provides the StarRocks Writer plugin to synchronize data from data sources supported by DataX to StarRocks. #### [CloudCanal](https://docs.starrocks.io/docs/integrations/loading_tools/CloudCanal.md)[​](#cloudcanal "Direct link to cloudcanal") CloudCanal Community Edition is a free data migration and synchronization platform published by [ClouGence Co., Ltd](https://www.bladepipe.com/) that integrates Schema Migration, Full Data Migration, verification, Correction, and real-time Incremental Synchronization. You can directly add StarRocks as a data source in CloudCanal's visual interface and create tasks to automatically migrate or synchronize data from source databases (e.g., MySQL, Oracle, PostgreSQL) to StarRocks. #### [Kettle connector](https://github.com/StarRocks/starrocks-connector-for-kettle)[​](#kettle-connector "Direct link to kettle-connector") Kettle is an ETL (Extract, Transform, Load) tool with a visual graphical interface, which allows users to build data processing workflows by dragging components and configuring parameters. This intuitive method greatly simplifies the process of data processing and loading, enabling users to handle data more conveniently. Additionally, Kettle provides a rich library of components, allowing users to select suitable components according to their needs and perform various complex data processing tasks. StarRocks offers the Kettle Connector to integrate with Kettle. By combining Kettle's robust data processing and transformation capabilities with StarRocks's high-performance data storage and analytical abilities, more flexible and efficient data processing workflows can be achieved. --- ### StarRocks Migration Tool (SMT) StarRocks Migration Tool (SMT) is a data migration tool provided by StarRocks to load data from source databases through Flink into StarRocks. SMT mainly can: * Generate statements to create tables in StarRocks based on information of the source database and the target StarRocks cluster. * Generate SQL statements that can be executed in Flink's SQL client to submit Flink jobs for synchronizing data, which simplifies full or incremental data synchronization in the pipeline. Currently, SMT supports the following source databases: | Source database | Generate statement to create table in StarRocks | Full data synchronization | Incremental data synchronization | | --------------- | ----------------------------------------------- | ------------------------- | -------------------------------- | | MySQL | Supported | Supported | Supported | | PostgreSQL | Supported | Supported | Supported | | Oracle | Supported | Supported | Supported | | Hive | Supported | Supported | Not supported | | ClickHouse | Supported | Supported | Not supported | | SQL Server | Supported | Supported | Supported | | TiDB | Supported | Supported | Supported | Download link: #### Steps to use SMT[​](#steps-to-use-smt "Direct link to Steps to use SMT") The generally involved steps are as follows: 1. Configure the **conf/config\_prod.conf** file. 2. Execute **starrocks-migration-tool**. 3. After execution, the SQL scripts are generated in the **result** directory by default. You can then use the SQL scripts in the **result** directory for metadata or data synchronization. #### Configurations of SMT[​](#configurations-of-smt "Direct link to Configurations of SMT") * `[db]`: information to connect the data source. Configure the information to connect the data source corresponding to the database type specified in the `type` patameter. * `[other]`: additional configurations. It is recommended to specify the actual number of BE nodes in the `be_num` patameter. * `flink.starrocks.sink.*`: configurations of flink-connector-starrocks. For detailed configurations and description, see [configuration description](https://github.com/StarRocks/flink-connector-starrocks#sink-options). * `[table-rule.1]`: rule to match tables in the data source. The CREATE TABLE statement is generated based on the regular expressions configured in the rule to match the names of databases and tables in data source. Multiple rules can be configured, and each rule generates a corresponding result file, for example: * `[table-rule.1]` -> `result/starrocks-create.1.sql` * `[table-rule.2]` -> `result/starrocks-create.2.sql` Each rule needs to contain the configurations of database, table, and flink-connector-starrocks. ```bash [table-rule.1] # pattern to match databases for setting properties database = ^ database1.*$ # pattern to match tables for setting properties table = ^.*$ schema = ^.*$ ############################################ ### flink sink configurations ### DO NOT set `connector`, `table-name`, `database-name`, they are auto-generated ############################################ flink.starrocks.jdbc-url=jdbc:mysql://192.168.1.1:9030 flink.starrocks.load-url= 192.168.1.1:8030 flink.starrocks.username=root flink.starrocks.password= flink.starrocks.sink.max-retries=10 flink.starrocks.sink.buffer-flush.interval-ms=15000 flink.starrocks.sink.properties.format=json flink.starrocks.sink.properties.strip_outer_array=true [table-rule.2] # pattern to match databases for setting properties database = ^database2.*$ # pattern to match tables for setting properties table = ^.*$ schema = ^.*$ ############################################ ### flink sink configurations ### DO NOT set `connector`, `table-name`, `database-name`, they are auto-generated ############################################ flink.starrocks.jdbc-url=jdbc:mysql://192.168.1.1:9030 flink.starrocks.load-url= 192.168.1.1:8030 flink.starrocks.username=root flink.starrocks.password= flink.starrocks.sink.max-retries=10 flink.starrocks.sink.buffer-flush.interval-ms=15000 flink.starrocks.sink.properties.format=json flink.starrocks.sink.properties.strip_outer_array=true ``` * A separate rule can be configured for a large table that is split into shards in databases. For example, suppose that the two databases `edu_db_1` and `edu_db_2` contain the tables `course_1` and `course_2` respectively, and these two tables have the same structure. You can use the following rule to load data from these two tables into one StarRocks table for analysis. ```bash [table-rule.3] # pattern to match databases for setting properties database = ^edu_db_[0-9]*$ # pattern to match tables for setting properties table = ^course_[0-9]*$ schema = ^.*$ ``` This rule will automatically form a many-to-one loading relationship. The default name of the table that will be generated in StarRocks is `course__auto_shard`, and you can also modify the table name in the related SOL script, such as `result/starrocks-create.3.sql`. #### Synchronize MySQL to StarRocks[​](#synchronize-mysql-to-starrocks "Direct link to Synchronize MySQL to StarRocks") ##### Introduction[​](#introduction "Direct link to Introduction") Flink CDC connector and SMT can synchronize data from MySQL within subsecond. ![img](/assets/images/load_tools-8e88eab0e5c9d2a228fcd668c63bc3e1.png) As shown in the image, SMT can automatically generate CREATE TABLE statements of the Flink's source and sink tables based on the cluster information and table structure of MySQL and StarRocks. Flink CDC connector reads the MySQL Binlog and Flink-connector-starrocks writes data to StarRocks . ##### Steps[​](#steps "Direct link to Steps") | Dependency | Package name | Download link | | ------------------------- | ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | | Flink | flink-x.x.x-bin-scala\_2.12.tgz | [Flink](https://flink.apache.org/downloads/) | | Flink CDC connector | flink-sql-connector-mysql-cdc-x.x.x.jar | [Flink CDC connector](https://github.com/apache/flink-cdc/releases) | | Flink-connector-starrocks | flink-connector-starrocks-x.x.x\_flink-x.x.jar | [Flink connector for StarRocks](https://github.com/StarRocks/starrocks-connector-for-apache-flink/releases) | | SMT | smt.tar.gz | [SMT](https://cdn-thirdparty.starrocks.com/smt.tar.gz?r=2) | 1. Download [Flink](https://flink.apache.org/downloads/). Flink 1.11 or later is supported. 2. Download [Flink CDC connector](https://github.com/ververica/flink-cdc-connectors/releases). Make sure that you download the `flink-sql-connector-mysql-cdc-xxx.jar` corresponding to the version of Flink. 3. Download [Flink-connector-starrocks](https://github.com/StarRocks/starrocks-connector-for-apache-flink/releases). 4. Copy **flink-sql-connector-mysql-cdc-xxx.jar** and **flink-connector-starrocks-xxx.jar** to **flink-xxx/lib/**. 5. Download [smt.tar.gz](https://cdn-thirdparty.starrocks.com/smt.tar.gz?r=2). 6. Extract and modify the configuration file of SMT. ```bash [db] host = 192.168.1.1 port = 3306 user = root password = type = mysql [other] # number of backends in StarRocks be_num = 3 # `decimal_v3` is supported since StarRocks-1.18.1 use_decimal_v3 = false # directory to save the converted DDL SQL output_dir = ./result [table-rule.1] # pattern to match databases for setting properties database = ^db$ # pattern to match tables for setting properties table = ^table$ schema = ^.*$ ############################################ ### flink sink configurations ### DO NOT set `connector`, `table-name`, `database-name`, they are auto-generated ############################################ flink.starrocks.jdbc-url=jdbc:mysql://192.168.1.1:9030 flink.starrocks.load-url= 192.168.1.1:8030 flink.starrocks.username=root flink.starrocks.password= flink.starrocks.sink.max-retries=10 flink.starrocks.sink.buffer-flush.interval-ms=15000 flink.starrocks.sink.properties.format=json flink.starrocks.sink.properties.strip_outer_array=true ``` 7. Execute **starrocks-migrate-tool**. All SQL scripts are generated in the result directory. ```bash $./starrocks-migrate-tool $ls result flink-create.1.sql smt.tar.gz starrocks-create.all.sql flink-create.all.sql starrocks-create.1.sql starrocks-external-create.all.sql ``` 8. Use a SQL script whose prefix is **starrocks-create** to generate the table in StarRocks. ```bash mysql -hxx.xx.xx.x -P9030 -uroot -p < starrocks-create.all.sql ``` 9. Use a SQL script whose prefix is **flink-create** to generate the Flink source and sink tables, and start a Flink job to synchronize data. ```bash bin/sql-client.sh embedded < flink-create.all.sql ``` After the above command is successfully executed, the Flink job to synchronize data keeps running. 10. Observe the status of the Flink job. ```bash bin/flink list ``` If the job execution encounters an error, you can view detailed error information in the Flink logs. Also, you can modify the Flink configurations in the file **conf/flink-conf.yaml**, such as memory and slot. ##### Notes[​](#notes "Direct link to Notes") * How to enable MySQL binlog? 1. Modify /etc/my.cnf: ```plaintext # Enable binlog log-bin=/var/lib/mysql/mysql-bin #log_bin=ON ## Base name of binlog files #log_bin_basename=/var/lib/mysql/mysql-bin ## Index file for binlog files, managing all binlog files #log_bin_index=/var/lib/mysql/mysql-bin.index # Configure server id server-id=1 binlog_format = row ``` 2. Restart mysqld. You can check whether MySQL binlog is enabled by executing `SHOW VARIABLES LIKE 'log_bin';`. #### Synchronize PostgreSQL to StarRocks[​](#synchronize-postgresql-to-starrocks "Direct link to Synchronize PostgreSQL to StarRocks") ##### Introduction[​](#introduction-1 "Direct link to Introduction") Flink CDC connector and SMT can synchronize data from PostgreSQL within subsecond. SMT can automatically generate CREATE TABLE statements of the Flink's source and sink tables based on the cluster information and table structure of PostgreSQL and StarRocks. Flink CDC connector reads the WAL of PostgreSQL and Flink-connector-starrocks writes data to StarRocks . ##### Steps[​](#steps-1 "Direct link to Steps") 1. Download [Flink](https://flink.apache.org/downloads/). The version of Flink is supported to be 1.11 or later. 2. Download [Flink CDC connector](https://github.com/ververica/flink-cdc-connectors/releases). Make sure that you download the flink-sql-connector-postgres-cdc-xxx.jar corresponding to the version of Flink. 3. Download [Flink StarRocks connector](https://github.com/StarRocks/flink-connector-starrocks). 4. Copy **flink-sql-connector-postgres-cdc-xxx.jar** and **flink-connector-starrocks-xxx.jar** to **flink-xxx/lib/**. 5. Download [smt.tar.gz](https://cdn-thirdparty.starrocks.com/smt.tar.gz?r=2). 6. Extract and modify the configuration file of SMT. ```bash [db] host = 192.168.1.1 port = 5432 user = xxx password = xxx type = pgsql [other] # number of backends in StarRocks be_num = 3 # `decimal_v3` is supported since StarRocks-1.18.1 use_decimal_v3 = false # directory to save the converted DDL SQL output_dir = ./result [table-rule.1] # pattern to match databases for setting properties database = ^db$ # pattern to match tables for setting properties table = ^table$ # pattern to match schemas for setting properties schema = ^.*$ ############################################ ### flink sink configurations ### DO NOT set `connector`, `table-name`, `database-name`, they are auto-generated ############################################ flink.starrocks.jdbc-url=jdbc:mysql://192.168.1.1:9030 flink.starrocks.load-url= 192.168.1.1:8030 flink.starrocks.username=root flink.starrocks.password= flink.starrocks.sink.max-retries=10 flink.starrocks.sink.buffer-flush.interval-ms=15000 flink.starrocks.sink.properties.format=json flink.starrocks.sink.properties.strip_outer_array=true ``` 7. Execute **starrocks-migrate-tool**. All SQL scripts are generated in the result directory. ```bash $./starrocks-migrate-tool $ls result flink-create.1.sql smt.tar.gz starrocks-create.all.sql flink-create.all.sql starrocks-create.1.sql ``` 8. Use a SQL script whose prefix is **starrocks-create** to generate the table in StarRocks. ```bash mysql -hxx.xx.xx.x -P9030 -uroot -p < starrocks-create.all.sql ``` 9. Use a SQL script whose prefix is **flink-create** to generate the Flink source and sink tables, and start a Flink job to synchronize data. ```bash bin/sql-client.sh embedded < flink-create.all.sql ``` After the above command is successfully executed, the Flink job to synchronize data keeps running. 10. Observe the status of the Flink job. ```bash bin/flink list ``` If the job execution encounters an error, you can view detailed error information in the Flink logs. Also, you can modify the Flink configurations in the file **conf/flink-conf.yaml**, such as memory and slot. ##### Notes[​](#notes-1 "Direct link to Notes") * For PostgreSQL `v9.*`, a special flink-cdc configuration as shown below is required (It is recommended to use PostgreSQL `v10.*` or later. Otherwise, you need to install WAL decoding plugins): ```bash ############################################ ############################################ ### flink-cdc plugin configuration for `postgresql` ############################################ ### for `9.*` decoderbufs, wal2json, wal2json_rds, wal2json_streaming, wal2json_rds_streaming ### refer to https://ververica.github.io/flink-cdc-connectors/master/content/connectors/postgres-cdc.html ### and https://debezium.io/documentation/reference/postgres-plugins.html ### flink.cdc.decoding.plugin.name = decoderbufs ``` * How to enable PostgreSQL WAL? ```bash # Open connection permissions echo "host all all 0.0.0.0/32 trust" >> pg_hba.conf echo "host replication all 0.0.0.0/32 trust" >> pg_hba.conf # Enable wal logical replication echo "wal_level = logical" >> postgresql.conf echo "max_wal_senders = 2" >> postgresql.conf echo "max_replication_slots = 8" >> postgresql.conf ``` Specify replica identity FULL for the tables that need to be synchronized. ```sql ALTER TABLE schema_name.table_name REPLICA IDENTITY FULL ``` After making these changes, restart PostgreSQL . #### Synchronize Oracle to StarRocks[​](#synchronize-oracle-to-starrocks "Direct link to Synchronize Oracle to StarRocks") ##### Introduction[​](#introduction-2 "Direct link to Introduction") Flink CDC connector and SMT can synchronize data from Oracle within subsecond. SMT can automatically generate CREATE TABLE statements of the Flink's source and sink tables based on the cluster information and table structure of Oracle and StarRocks. Flink CDC connector reads the logminer of Oracle and Flink-connector-starrocks writes data to StarRocks. ##### Steps[​](#steps-2 "Direct link to Steps") 1. Download [Flink](https://flink.apache.org/downloads/). The version of Flink is supported to be 1.11 or later. 2. Download [Flink CDC connector](https://github.com/ververica/flink-cdc-connectors/releases). Make sure that you download the flink-sql-connector-oracle-cdc-xxx.jar corresponding to the Flink version. 3. Download [Flink StarRocks connector](https://github.com/StarRocks/flink-connector-starrocks). 4. Copy `flink-sql-connector-oracle-cdc-xxx.jar` and `flink-connector-starrocks-xxx.jar` to `flink-xxx/lib/`. 5. Download [smt.tar.gz](https://cdn-thirdparty.starrocks.com/smt.tar.gz?r=2). 6. Extract and modify the configuration file of SMT. ```bash [db] host = 192.168.1.1 port = 1521 user = xxx password = xxx type = oracle [other] # number of backends in StarRocks be_num = 3 # `decimal_v3` is supported since StarRocks-1.18.1 use_decimal_v3 = false # directory to save the converted DDL SQL output_dir = ./result [table-rule.1] # pattern to match databases for setting properties database = ^db$ # pattern to match tables for setting properties table = ^table$ # pattern to match schemas for setting properties schema = ^.*$ ############################################ ### flink sink configurations ### DO NOT set `connector`, `table-name`, `database-name`, they are auto-generated ############################################ flink.starrocks.jdbc-url=jdbc:mysql://192.168.1.1:9030 flink.starrocks.load-url= 192.168.1.1:8030 flink.starrocks.username=root flink.starrocks.password= flink.starrocks.sink.max-retries=10 flink.starrocks.sink.buffer-flush.interval-ms=15000 flink.starrocks.sink.properties.format=json flink.starrocks.sink.properties.strip_outer_array=true ``` 7. Execute **starrocks-migrate-tool**. All SQL scripts are generated in the result directory. ```bash $./starrocks-migrate-tool $ls result flink-create.1.sql smt.tar.gz starrocks-create.all.sql flink-create.all.sql starrocks-create.1.sql ``` 8. Use a SQL script whose prefix is starrocks-create to generate the table in StarRocks. ```bash mysql -hxx.xx.xx.x -P9030 -uroot -p < starrocks-create.all.sql ``` 9. Use a SQL script whose prefix is flink-create to generate the Flink source and sink tables, and start a Flink job to synchronize data. ```bash bin/sql-client.sh embedded < flink-create.all.sql ``` After the above command is successfully executed, the Flink job to synchronize data keeps running. 10. Observe the status of the Flink job. ```bash bin/flink list ``` If the job execution encounters an error, you can view detailed error information in the Flink logs. Also, you can modify the Flink configurations in the file **conf/flink-conf.yaml**, such as memory and slot. ##### Notes[​](#notes-2 "Direct link to Notes") * Synchronize Oracle using logminer: ```sql # Enable logging alter system set db_recovery_file_dest = '/home/oracle/data' scope=spfile; alter system set db_recovery_file_dest_size = 10G; shutdown immediate; startup mount; alter database archivelog; alter database open; ALTER TABLE schema_name.table_name ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS; ALTER DATABASE ADD SUPPLEMENTAL LOG DATA; # Authorize user creation and grant permissions GRANT CREATE SESSION TO flinkuser; GRANT SET CONTAINER TO flinkuser; GRANT SELECT ON V_$DATABASE TO flinkuser; GRANT FLASHBACK ANY TABLE TO flinkuser; GRANT SELECT ANY TABLE TO flinkuser; GRANT SELECT_CATALOG_ROLE TO flinkuser; GRANT EXECUTE_CATALOG_ROLE TO flinkuser; GRANT SELECT ANY TRANSACTION TO flinkuser; GRANT LOGMINING TO flinkuser; GRANT CREATE TABLE TO flinkuser; GRANT LOCK ANY TABLE TO flinkuser; GRANT ALTER ANY TABLE TO flinkuser; GRANT CREATE SEQUENCE TO flinkuser; GRANT EXECUTE ON DBMS_LOGMNR TO flinkuser; GRANT EXECUTE ON DBMS_LOGMNR_D TO flinkuser; GRANT SELECT ON V_$LOG TO flinkuser; GRANT SELECT ON V_$LOG_HISTORY TO flinkuser; GRANT SELECT ON V_$LOGMNR_LOGS TO flinkuser; GRANT SELECT ON V_$LOGMNR_CONTENTS TO flinkuser; GRANT SELECT ON V_$LOGMNR_PARAMETERS TO flinkuser; GRANT SELECT ON V_$LOGFILE TO flinkuser; GRANT SELECT ON V_$ARCHIVED_LOG TO flinkuser; GRANT SELECT ON V_$ARCHIVE_DEST_STATUS TO flinkuser; ``` * The database configurations in the \[table-rule.1] does not support regular expression, so the complete database names need to be specified. * Because Oracle12c supports CDB mode, SMT internally automatically determines whether CDB is enabled and modifies the flink-cdc configuration correspondingly. However, users need to pay attention to whether the c## prefix needs to be added to the configuration of `[db].user` to avoid issues of insufficient permission. #### Synchronize Hive to StarRocks[​](#synchronize-hive-to-starrocks "Direct link to Synchronize Hive to StarRocks") ##### Introduction[​](#introduction-3 "Direct link to Introduction") This guide explains how to use SMT to synchronize Hive data to StarRocks. During synchronization, a Duplicate table in StarRocks is created and a Flink job keeps running to synchronize data. ##### Steps[​](#steps-3 "Direct link to Steps") ###### Preparations[​](#preparations "Direct link to Preparations") ```sql [db] # hiveserver2 service ip host = 127.0.0.1 # hiveserver2 service port port = 10000 user = hive/emr-header-1.cluster-49148 password = type = hive # only takes effect with `type = hive`. # Available values: kerberos, none, nosasl, kerberos_http, none_http, zk, ldap authentication = kerberos ``` The supported authentication methods are as follows: * nosasl, zk: do not need to specify the `user` and `password`. * none, none\_http, ldap: specify the `user` and `password`. * kerberos, kerberos\_http: perform the following steps: * Execute `kadmin.local` on the Hive cluster and check `list_principals` to find the corresponding principal name. For example when the principal name is `hive/emr-header-1.cluster-49148@EMR.49148.COM`, the user needs to be set as `hive/emr-header-1.cluster-49148`, and the password is left empty. * Execute `kinit -kt /path/to/keytab principal` on the machine where SMT is executed and execute `klist` to see if the correct token is generated. ###### Data synchronization[​](#data-synchronization "Direct link to Data synchronization") 1. Execute **starrocks-migrate-tool**. 2. Use a SQL script whose prefix is **starrocks-create** to generate the table in StarRocks. ```bash mysql -hxx.xx.xx.x -P9030 -uroot -p < starrocks-create.all.sql ``` 3. In **flink/conf/**, create and edit the file **sql-client-defaults.yaml**: ```yaml execution: planner: blink type: batch current-catalog: hive-starrocks catalogs: - name: hive-starrocks type: hive hive-conf-dir: /path/to/apache-hive-xxxx-bin/conf ``` 4. Download the [dependency package](https://nightlies.apache.org/flink/flink-docs-release-1.13/zh/docs/connectors/table/hive/overview/) (flink-sql-connector-hive-xxxx) from the Hive page of the corresponding version of Flink and place it in the `flink/lib` directory. 5. Start the Flink cluster and execute `flink/bin/sql-client.sh embedded < result/flink-create.all.sql` to start data synchronization. #### Synchronize SQL Server to StarRocks[​](#synchronize-sql-server-to-starrocks "Direct link to Synchronize SQL Server to StarRocks") ##### Introduction[​](#introduction-4 "Direct link to Introduction") Flink CDC connector and SMT can synchronize data from SQL Server within subsecond. SMT can automatically generate CREATE TABLE statements of the Flink's source and sink tables based on the cluster information and table structure of SQL Server and StarRocks. Flink CDC connector captures and records row-level changes that occur in the SQL Server database server. The principle is to use the CDC feature provided by SQL Server itself. The CDC capability of SQL Server itself can archive specified changes in the database to specified change tables. The SQL Server CDC connector first reads historical data from the table using JDBC, and then fetches incremental changes from the change tables, thereby achieving full incremental synchronization. Then, Flink-connector-starrocks writes data to StarRocks. ##### Steps[​](#steps-4 "Direct link to Steps") 1. Download [Flink](https://flink.apache.org/downloads/). The version of Flink is supported to be 1.11 or later. 2. Download [Flink CDC connector](https://github.com/ververica/flink-cdc-connectors/releases). Make sure that you download the **flink-sql-connector-sqlserver-cdc-xxx.jar** corresponding to the Flink version. 3. Download [Flink StarRocks connector](https://github.com/StarRocks/flink-connector-starrocks). 4. Copy **flink-sql-connector-sqlserver-cdc-xxx.jar**, **flink-connector-starrocks-xxx.jar** to **flink-xxx/lib/**. 5. Download [smt.tar.gz](https://cdn-thirdparty.starrocks.com/smt.tar.gz?r=2). 6. Extract and modify the configuration file of SMT. ```bash [db] host = 127.0.0.1 port = 1433 user = xxx password = xxx # currently available types: `mysql`, `pgsql`, `oracle`, `hive`, `clickhouse` type = sqlserver [other] # number of backends in StarRocks be_num = 3 # `decimal_v3` is supported since StarRocks-1.18.1 use_decimal_v3 = false # directory to save the converted DDL SQL output_dir = ./result [table-rule.1] # pattern to match databases for setting properties database = ^db$ # pattern to match tables for setting properties table = ^table$ schema = ^.*$ ############################################ ### flink sink configurations ### DO NOT set `connector`, `table-name`, `database-name`, they are auto-generated ############################################ flink.starrocks.jdbc-url=jdbc:mysql://192.168.1.1:9030 flink.starrocks.load-url= 192.168.1.1:8030 flink.starrocks.username=root flink.starrocks.password= flink.starrocks.sink.max-retries=10 flink.starrocks.sink.buffer-flush.interval-ms=15000 flink.starrocks.sink.properties.format=json flink.starrocks.sink.properties.strip_outer_array=true ``` 7. Execute **starrocks-migrate-tool**. All SQL scripts are generated in the `result` directory. ​ \`\`\`Bash $./starrocks-migrate-tool $ls result flink-create.1.sql smt.tar.gz starrocks-create.all.sql flink-create.all.sql starrocks-create.1.sql starrocks-external-create.all.sql ````text 8. Use a SQL script whose prefix is `starrocks-create` to generate the table in StarRocks. ```Bash mysql -hxx.xx.xx.x -P9030 -uroot -p < starrocks-create.all.sql ```` 9. Use a SQL script whose prefix is `flink-create` to generate the Flink source and sink tables, and start a Flink job to synchronize data. ```bash bin/sql-client.sh embedded < flink-create.all.sql ``` After the above command is successfully executed, the Flink job to synchronize data keeps running. 10. Observe the status of the Flink job. ```bash bin/flink list ``` If the job execution encounters an error, you can view detailed error information in the Flink logs. Also, you can modify the Flink configurations in the file **conf/flink-conf.yaml**, such as memory and slot. ##### Notes[​](#notes-3 "Direct link to Notes") 1. Make sure that the Server Agent Service is enabled. Check if the Server Agent Service is running normally. ```sql EXEC master.dbo.xp_servicecontrol N'QUERYSTATE', N'SQLSERVERAGENT' GO ``` Enable the Server Agent Service. ```bash /opt/mssql/bin/mssql-conf set sqlagent.enabled true ``` 2. Make sure that CDC for the corresponding database is enabled. ​ Check if CDC for the corresponding database is enabled. ```sql select is_cdc_enabled, name from sys.databases where name = 'XXX_databases' GO ``` ​ Enable CDC. ​ :::note ​ When executing this command, make sure that the user `serverRole` is `sysadmin`. ​ ::: ```sql USE XXX_databases GO EXEC sys.sp_cdc_enable_db GO ``` 3. Make sure that CDC for the corresponding table is enabled. ```sql EXEC sys.sp_cdc_enable_table @source_schema = 'XXX_schema', @source_name = 'XXX_table', @role_name = NULL, @supports_net_changes = 0; GO ``` #### Synchronize TiDB to StarRocks[​](#synchronize-tidb-to-starrocks "Direct link to Synchronize TiDB to StarRocks") ##### Introduction[​](#introduction-5 "Direct link to Introduction") Flink CDC connector and SMT can synchronize data from TiDB within subsecond. SMT can automatically generate DDL statements for the Flink's source tables and sink tables based on the cluster information and table structure of TiDB and StarRocks. The Flink CDC connector captures data by directly reading full and incremental data from the underlying TiKV storage. The full data is obtained from ranges partitioned based on keys, and the incremental data is obtained by using the CDC Client provided by TiDB. Subsequently, the data is written to StarRocks through Flink-connector-starrocks. ##### Steps[​](#steps-5 "Direct link to Steps") 1. Download [Flink](https://flink.apache.org/downloads/). The version of Flink is supported to be 1.11 or later. 2. Download [Flink CDC connector](https://github.com/ververica/flink-cdc-connectors/releases). Make sure that you download the **flink-sql-connector-tidb-cdc-xxx.jar** corresponding to the Flink version. 3. Download [Flink StarRocks connector](https://github.com/StarRocks/flink-connector-starrocks). 4. Copy **flink-sql-connector-tidb-cdc-xxx.jar**, **flink-connector-starrocks-xxx.jar** to **flink-xxx/lib/**. 5. Download [smt.tar.gz](https://cdn-thirdparty.starrocks.com/smt.tar.gz?r=2). 6. Extract and modify the configuration file of SMT. ```bash [db] host = 127.0.0.1 port = 4000 user = root password = # currently available types: `mysql`, `pgsql`, `oracle`, `hive`, `clickhouse`, `sqlserver`, `tidb` type = tidb # # only takes effect on `type == hive`. # # Available values: kerberos, none, nosasl, kerberos_http, none_http, zk, ldap # authentication = kerberos [other] # number of backends in StarRocks be_num = 3 # `decimal_v3` is supported since StarRocks-1.18.1 use_decimal_v3 = false # directory to save the converted DDL SQL output_dir = ./result [table-rule.1] # pattern to match databases for setting properties database = ^db$ # pattern to match tables for setting properties table = ^table$ schema = ^.*$ ############################################ ### flink sink configurations ### DO NOT set `connector`, `table-name`, `database-name`, they are auto-generated ############################################ flink.starrocks.jdbc-url=jdbc:mysql://192.168.1.1:9030 flink.starrocks.load-url= 192.168.1.1:8030 flink.starrocks.username=root flink.starrocks.password= flink.starrocks.sink.max-retries=10 flink.starrocks.sink.buffer-flush.interval-ms=15000 flink.starrocks.sink.properties.format=json flink.starrocks.sink.properties.strip_outer_array=true ############################################ ### flink-cdc configuration for `tidb` ############################################ # # Only takes effect on TiDB before v4.0.0. # # TiKV cluster's PD address. # flink.cdc.pd-addresses = 127.0.0.1:2379 ``` 7. Execute *starrocks-migrate-tool*\*. All SQL scripts are generated in the `result` directory. ```bash $./starrocks-migrate-tool $ls result flink-create.1.sql smt.tar.gz starrocks-create.all.sql flink-create.all.sql starrocks-create.1.sql starrocks-external-create.all.sql ``` 8. Use a SQL script whose prefix is `starrocks-create` to generate the table in StarRocks. ```bash mysql -hxx.xx.xx.x -P9030 -uroot -p < starrocks-create.all.sql ``` 9. Use a SQL script whose prefix is `flink-create` to generate the Flink source and sink tables, and start a Flink job to synchronize data. ```bash bin/sql-client.sh embedded < flink-create.all.sql ``` ​ After the above command is successfully executed, the Flink job to synchronize data keeps running. 10. Observe the status of the Flink job. ```bash bin/flink list ``` If the job execution encounters an error, you can view detailed error information in the Flink logs. Also, you can modify the Flink configurations in the file **conf/flink-conf.yaml**, such as memory and slot. ##### Notes[​](#notes-4 "Direct link to Notes") For TiDB whose version is before v4.0.0, additional configuration of `flink.cdc.pd-addresses` is required. ```plain ############################################ ### flink-cdc configuration for `tidb` ############################################ # # Only takes effect on TiDB before v4.0.0. # # TiKV cluster's PD address. # flink.cdc.pd-addresses = 127.0.0.1:2379 ``` --- ### Monitor with Datadog This topic describes how to integrate your StarRocks cluster with [Datadog](https://www.datadoghq.com/), a monitoring and security platform. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") Before getting started, you must have the following software installed on your instances: * [Datadog Agent](https://docs.datadoghq.com/getting_started/agent/) * Python > **NOTE** > > When you install Datadog Agent for the first time, Python is also installed as a dependency. We recommend you use this Python in the following steps. #### Prepare StarRocks source code[​](#prepare-starrocks-source-code "Direct link to Prepare StarRocks source code") Since Datadog does not provide the integration kit for StarRocks yet, you need to integrate them using the source code. 1. Launch a terminal, navigate to a local directory to which you have both read and write access, and run the following command to create a dedicated directory for StarRocks source code. ```shell mkdir -p starrocks ``` 2. Download the StarRocks source code package using the following command or on [GitHub](https://github.com/StarRocks/starrocks/tags) to the directory you created. ```shell cd starrocks # Replace with the actual version of StarRocks, for example, "2.5.2". wget https://github.com/StarRocks/starrocks/archive/refs/tags/.tar.gz ``` 3. Extract the files in the package. ```shell # Replace with the actual version of StarRocks, for example, "2.5.2". tar -xzvf .tar.gz --strip-components 1 ``` #### Install and configure FE integration kit[​](#install-and-configure-fe-integration-kit "Direct link to Install and configure FE integration kit") 1. Install Datadog integration kit for FE using source code. ```shell /opt/datadog-agent/embedded/bin/pip install contrib/datadog-connector/starrocks_fe ``` 2. Create the FE integration configuration file **/etc/datadog-agent/conf.d/starrocks\_fe.d/conf.yaml**. ```shell sudo mkdir -p /etc/datadog-agent/conf.d/starrocks_fe.d sudo cp contrib/datadog-connector/starrocks_fe/datadog_checks/starrocks_fe/data/conf.yaml.example /etc/datadog-agent/conf.d/starrocks_fe.d/conf.yaml ``` 3. Modify the FE integration configuration file **/etc/datadog-agent/conf.d/starrocks\_fe.d/conf.yaml**. Examples of some important configuration items: | **Config** | **Example** | **Description** | | --------------- | ------------------------------- | ------------------------------------------------------------------------------------------ | | fe\_metric\_url | `http://localhost:8030/metrics` | The URL used to access the StarRocks FE metrics. | | metrics | `- starrocks_fe_*` | Metrics to be monitored on FE. You can use wildcards `*` to match the configuration items. | #### Install and configure BE integration kit[​](#install-and-configure-be-integration-kit "Direct link to Install and configure BE integration kit") 1. Install Datadog integration kit for BE using source code. ```shell /opt/datadog-agent/embedded/bin/pip install contrib/datadog-connector/starrocks_be ``` 2. Create the BE integration configuration file **/etc/datadog-agent/conf.d/starrocks\_be.d/conf.yaml**. ```shell sudo mkdir -p /etc/datadog-agent/conf.d/starrocks_be.d sudo cp contrib/datadog-connector/starrocks_be/datadog_checks/starrocks_be/data/conf.yaml.example /etc/datadog-agent/conf.d/starrocks_be.d/conf.yaml ``` 3. Modify the BE integration configuration file **/etc/datadog-agent/conf.d/starrocks\_be.d/conf.yaml**. Examples of some important configuration items: | **Config** | **Example** | **Description** | | --------------- | ------------------------------- | ------------------------------------------------------------------------------------------ | | be\_metric\_url | `http://localhost:8040/metrics` | The URL used to access the StarRocks BE metrics. | | metrics | `- starrocks_be_*` | Metrics to be monitored on BE. You can use wildcards `*` to match the configuration items. | #### Restart Datadog Agent[​](#restart-datadog-agent "Direct link to Restart Datadog Agent") Restart Datadog Agent to allow the configuration to take effect. ```shell sudo systemctl stop datadog-agent sudo systemctl start datadog-agent ``` #### Verify integration[​](#verify-integration "Direct link to Verify integration") For instructions to verify the integration, see [Datadog Application](https://docs.datadoghq.com/getting_started/application/). #### Uninstall integration kits[​](#uninstall-integration-kits "Direct link to Uninstall integration kits") You can uninstall the integration kits when you no longer need them. * To uninstall FE integration kit, run the following command: ```shell /opt/datadog-agent/embedded/bin/pip uninstall datadog-starrocks-fe ``` * To uninstall BE integration kit, run the following command: ```shell /opt/datadog-agent/embedded/bin/pip uninstall datadog-starrocks-be ``` --- ### Schema Management and Migration with SQLAlchemy and Alembic This guide introduces how to manage StarRocks schemas using the Python ecosystem — including SQLAlchemy, Alembic, and sqlacodegen — through the **`starrocks` SQLAlchemy** dialect. It is designed to help you understand **why schema migration is useful** and **how to use it effectively with StarRocks**. #### Overview[​](#overview "Direct link to Overview") Many users manage StarRocks tables, views, and materialized views using SQL DDL directly. However, as projects grow, manually maintaining `ALTER TABLE` statements becomes error-prone and hard to track. The **StarRocks SQLAlchemy dialect (`starrocks`)** provides: * A full SQLAlchemy model layer for StarRocks **tables**, **views**, and **materialized views** * **Declarative** definitions for table schema and table properties (including views and materialized views) * Integration with **Alembic** to allow schema changes to be **detected** and **generated** automatically * Compatibility with tools like **sqlacodegen** for reverse-generating models This allows Python users to maintain StarRocks schemas in a **declarative**, **version-controlled**, and **automated** way. #### Key benefits[​](#key-benefits "Direct link to Key benefits") Although schema migration is traditionally associated with OLTP databases, it is also valuable in data warehousing systems such as StarRocks. Teams use [Alembic](https://alembic.sqlalchemy.org/) together with the StarRocks dialect because of the benefits listed below. ##### Declarative schema definition[​](#declarative-schema-definition "Direct link to Declarative schema definition") Once you define schema in Python [ORM](https://docs.sqlalchemy.org/en/20/orm/quickstart.html#orm-quickstart) models or [SQLAlchemy](https://docs.sqlalchemy.org) core style, writing `ALTER TABLE` statements manually is no longer required. ##### Automatic diffing and autogeneration[​](#automatic-diffing-and-autogeneration "Direct link to Automatic diffing and autogeneration") Alembic compares **current StarRocks schema** with **your SQLAlchemy models**, and generates migration scripts automatically (`CREATE`/`DROP`/`ALTER`). ##### Reviewable, version-controlled migrations[​](#reviewable-version-controlled-migrations "Direct link to Reviewable, version-controlled migrations") Each schema change becomes a migration file (Python), so users can track changes and roll back if needed. ##### Consistent workflow across environments[​](#consistent-workflow-across-environments "Direct link to Consistent workflow across environments") Schema changes can be applied to development, staging, and production with the same process. #### Installation and Connection[​](#installation-and-connection "Direct link to Installation and Connection") ##### Prerequisites\*\*[​](#prerequisites "Direct link to Prerequisites**") * StarRocks Python client: 1.3.2 or later * `SQLAlchemy`: 1.4 or later (SQLAlchemy 2.0 is recommended and is required to use `sqlacodegen`) * `Alembic`: 1.16 or later ##### Installing StarRocks Python client[​](#installing-starrocks-python-client "Direct link to Installing StarRocks Python client") Run the following command to install the StarRocks Python client. ```bash pip install starrocks ``` ##### Connecting to StarRocks[​](#connecting-to-starrocks "Direct link to Connecting to StarRocks") Connect to your StarRocks cluster using the following URL. ```bash starrocks://:@:/[.] ``` * `user`: Username used to connect to the cluster. * `password`: User password. * `FE_host`: FE IP address. * `query_port`: FE `query_port` (Default: 9030). * `catalog`: The name of the catalog where your database locates. * `database`: The name of the database you want to connect. After installation, you can quickly validate connectivity using the following code example: ```python from sqlalchemy import create_engine, text # you need to create `mydatabase` first engine = create_engine("starrocks://root@localhost:9030/mydatabase") with engine.connect() as conn: conn.execute(text("SELECT 1")).fetchall() print("Connection successful!") ``` #### Defining StarRocks Models (Declarative ORM)[​](#defining-starrocks-models-declarative-orm "Direct link to Defining StarRocks Models (Declarative ORM)") The StarRocks dialect supports: * Tables * Views * Materialized Views It also supports StarRocks-specific table attributes such as: * `ENGINE` (OLAP) * Key models (`DUPLICATE KEY`, `PRIMARY KEY`, `UNIQUE KEY`, `AGGREGATE KEY`) * `PARTITION BY` variants (RANGE / LIST / Expression partitioning) * `DISTRIBUTED BY` variants (HASH / RANDOM) * `ORDER BY` * Table properties (e.g., `replication_num`, `storage_medium`) important * StarRocks dialect options are passed as keyword arguments prefixed with `starrocks_`. * The `starrocks_` **prefix must be lowercase**. The suffix is accepted in either case (for example, `PRIMARY_KEY` and `primary_key`). * If you specify a table key (e.g. `starrocks_primary_key="id"`), the involved columns **must** also be marked with `primary_key=True` in `Column(...)`, so that SQLAlchemy metadata and Alembic autogenerate can behave correctly. Examples below reflect the real public API and parameter names. ##### Table Example[​](#table-example "Direct link to Table Example") StarRocks table options can be specified in both ORM (via `__table_args__`) and Core (via `Table(..., starrocks_...=...)`) styles. ###### ORM (Declarative) style[​](#orm-declarative-style "Direct link to ORM (Declarative) style") ```python from sqlalchemy import create_engine from sqlalchemy.orm import Mapped, declarative_base, mapped_column from starrocks import INTEGER, STRING # with the same engine as the quick test engine = create_engine("starrocks://root@localhost:9030/mydatabase") Base = declarative_base() class MyTable(Base): __tablename__ = 'my_orm_table' id: Mapped[int] = mapped_column(INTEGER, primary_key=True) name: Mapped[str] = mapped_column(STRING) __table_args__ = { 'comment': 'table comment', 'starrocks_primary_key': 'id', 'starrocks_distributed_by': 'HASH(id) BUCKETS 10', 'starrocks_properties': {'replication_num': '1'} } # Create the table in the database Base.metadata.create_all(engine) ``` ###### Core style[​](#core-style "Direct link to Core style") ```python from sqlalchemy import Column, MetaData, Table, create_engine from starrocks import INTEGER, VARCHAR # with the same engine as the quick test engine = create_engine("starrocks://root@localhost:9030/mydatabase") metadata = MetaData() my_core_table = Table( 'my_core_table', metadata, Column('id', INTEGER, primary_key=True), Column('name', VARCHAR(50)), # StarRocks-specific arguments starrocks_primary_key='id', starrocks_distributed_by='HASH(id) BUCKETS 10', starrocks_properties={"replication_num": "1"} ) # Create the table in the database metadata.create_all(engine) ``` note For a comprehensive reference of table attributes and data types, see [Reference \[4\]](#references). ##### View Example[​](#view-example "Direct link to View Example") Below is the recommended view definition style, using `columns` as a list of dicts (`name`/`comment`). This example is based on an existing table `my_core_table`. ```python from starrocks.schema import View # Reuse the metadata from the Core table example above metadata = my_core_table.metadata user_view = View( "user_view", metadata, definition="SELECT id, name FROM my_core_table WHERE name IS NOT NULL", columns=[ {"name": "id", "comment": "ID"}, {"name": "name", "comment": "Name"}, ], comment="Active users", ) ``` note For more View options and limitations, see [Reference \[5\]](#references). ##### Materialized View Example[​](#materialized-view-example "Direct link to Materialized View Example") Materialized views are defined similarly. The `starrocks_refresh` property is a syntax string that indicates the refresh strategy. ```python from starrocks.schema import MaterializedView # Reuse the metadata from the Core table example above metadata = my_core_table.metadata # Create a simple Materialized View (asynchronous refresh) user_stats_ = MaterializedView( 'user_stats_', metadata, definition='SELECT id, COUNT(*) AS cnt FROM my_core_table GROUP BY id', starrocks_refresh='ASYNC' ) ``` note For more options and ALTER limitations, see [Reference \[6\]](#references). #### Alembic Integration[​](#alembic-integration "Direct link to Alembic Integration") The StarRocks SQLAlchemy dialect provides full support for: * Create / Drop table * Create / Drop view * Create / Drop materialized view * Detecting supported changes on StarRocks-specific attributes (for example, table properties and distribution) This enables Alembic’s **autogenerate** to work properly. ##### Initializing Alembic[​](#initializing-alembic "Direct link to Initializing Alembic") 1. Initialize Alembic: ```bash alembic init migrations ``` 2. Configure your database URL in `alembic.ini`: ```ini # alembic.ini sqlalchemy.url = starrocks://:@:/[.] ``` 3. Enable StarRocks dialect logging (optional): You can enable the `starrocks` logger in `alembic.ini` to observe the detected changes of a table via logs. For details, see [Reference \[2\]](#references). Edit `env.py` (configure both offline and online paths): ```python from alembic import context from starrocks.alembic import render_column_type, include_object_for_view_ from starrocks.alembic.starrocks import StarRocksImpl # noqa: F401 (ensure impl registered) from myapp.models import Base # adjust to your project target_metadata = Base.metadata def run_migrations_offline() -> None: url = context.config.get_main_option("sqlalchemy.url") context.configure( url=url, target_metadata=target_metadata, render_item=render_column_type, include_object=include_object_for_view_ ) with context.begin_transaction(): context.run_migrations() def run_migrations_online() -> None: # ... create engine and connect as in alembic default env.py ... with connectable.connect() as connection: context.configure( connection=connection, target_metadata=target_metadata, render_item=render_column_type, include_object=include_object_for_view_ ) with context.begin_transaction(): context.run_migrations() ``` ##### Generating migrations automatically[​](#generating-migrations-automatically "Direct link to Generating migrations automatically") ```bash alembic revision --autogenerate -m "initial schema" ``` Alembic will compare SQLAlchemy models with the actual StarRocks schema, and output the correct DDL. ##### Applying migrations[​](#applying-migrations "Direct link to Applying migrations") ```bash alembic upgrade head ``` Downgrade is also supported (where reversible). important StarRocks DDL is not transactional across multiple statements. If an upgrade fails midway, you may need to inspect what has already been applied and **perform manual remediation** (for example, write a compensating migration or run manual DDL) before re-running. #### Supported Schema Change Operations[​](#supported-schema-change-operations "Direct link to Supported Schema Change Operations") The dialect supports Alembic autogenerate for: * **Tables**: create / drop, and diffing of StarRocks-specific attributes declared via `starrocks_*` (within StarRocks ALTER support) * **Views**: create / drop / alter (mainly definition-related changes; some attributes are immutable) * **Materialized Views**: create / drop / alter (limited to mutable clauses such as refresh strategies and properties) Some StarRocks DDL changes are not reversible or not alterable. You can only make these changes by dropping and recreate the table/view/materialized view. If you specify these changes in the dialect, autogenerate will **warn or raise**. #### End-to-End Example (Recommended Reading for Beginners)[​](#end-to-end-example-recommended-reading-for-beginners "Direct link to End-to-End Example (Recommended Reading for Beginners)") This section shows a runnable end-to-end workflow, including where to pause and review generated files. ##### Step 1. Create a project directory and initialize Alembic[​](#step-1-create-a-project-directory-and-initialize-alembic "Direct link to Step 1. Create a project directory and initialize Alembic") ```bash mkdir my_sr_alembic_project cd my_sr_alembic_project alembic init alembic ``` ##### Step 2. Configure `alembic.ini`[​](#step-2-configure-alembicini "Direct link to step-2-configure-alembicini") Edit the URL in `alembic.ini`: ```ini sqlalchemy.url = starrocks://root@localhost:9030/mydatabase ``` ##### Step 3. Define your models[​](#step-3-define-your-models "Direct link to Step 3. Define your models") Create a package for your models: ```bash mkdir -p myapp touch myapp/__init__.py ``` Create `myapp/models.py` and put your table/view/materialized view definitions in the package: note When using Alembic migrations, do not call `metadata.create_all(engine)` in your models module. ```python from sqlalchemy import Column, Table from sqlalchemy.orm import Mapped, declarative_base, mapped_column from starrocks import INTEGER, STRING, VARCHAR from starrocks.schema import MaterializedView, View Base = declarative_base() # --- ORM table --- class MyOrmTable(Base): __tablename__ = "my_orm_table" id: Mapped[int] = mapped_column(INTEGER, primary_key=True) name: Mapped[str] = mapped_column(STRING) __table_args__ = { "comment": "table comment", "starrocks_primary_key": "id", "starrocks_distributed_by": "HASH(id) BUCKETS 10", "starrocks_properties": {"replication_num": "1"}, } # --- Core table on the same metadata (important for Alembic target_metadata) --- my_core_table = Table( "my_core_table", Base.metadata, Column("id", INTEGER, primary_key=True), Column("name", VARCHAR(50)), comment="core table comment", starrocks_primary_key="id", starrocks_distributed_by="HASH(id) BUCKETS 10", starrocks_properties={"replication_num": "1"}, ) # --- View --- user_view = View( "user_view", Base.metadata, definition="SELECT id, name FROM my_core_table WHERE name IS NOT NULL", columns=[ {"name": "id", "comment": "ID"}, {"name": "name", "comment": "Name"}, ], comment="Active users", ) # --- Materialized View --- user_stats_mv = MaterializedView( "user_stats_mv", Base.metadata, definition="SELECT id, COUNT(*) AS cnt FROM my_core_table GROUP BY id", starrocks_refresh="ASYNC", ) ``` ##### Step 4. Configure `env.py` for autogenerate[​](#step-4-configure-envpy-for-autogenerate "Direct link to step-4-configure-envpy-for-autogenerate") Edit `alembic/env.py`: 1. Import `myapp.models` to set the `target_metadata`. 2. Import `render_column_type`, and `include_object_for_view_mv` to set them in both `run_migrations_offline()` and `run_migrations_online()` to properly handle views and MVs, and to properly render StarRocks column types. note You need to add or modify these lines in `env.py`, rather than replace the generated `env.py` file. ```python from alembic import context from starrocks.alembic import render_column_type, include_object_for_view_mv from starrocks.alembic.starrocks import StarRocksImpl # noqa: F401 from myapp.models import Base target_metadata = Base.metadata # Optional: set version table replication for single-BE dev clusters version_table_kwargs = {"starrocks_properties": {"replication_num": "1"}} # In both run_migrations_offline() and run_migrations_online(), ensure: def run_migrations_offline() -> None: url = context.config.get_main_option("sqlalchemy.url") context.configure( url=url, target_metadata=target_metadata, literal_binds=True, render_item=render_column_type, include_object=include_object_for_view_mv, version_table_kwargs=version_table_kwargs, ) def run_migrations_online() -> None: # ... create engine and connect as in alembic default env.py ... with connectable.connect() as connection: context.configure( connection=connection, target_metadata=target_metadata, render_item=render_column_type, include_object=include_object_for_view_mv, version_table_kwargs=version_table_kwargs, ) ``` ##### Step 5. Autogenerate the first revision[​](#step-5-autogenerate-the-first-revision "Direct link to Step 5. Autogenerate the first revision") ```bash alembic revision --autogenerate -m "create initial schema" ``` Pause and review: 1. Check the generated migration file under `alembic/versions/`. 2. Ensure it contains the expected operations (for example, `create_table`, `create_view`, `create_materialized_view`). 3. Make sure it does not contain unexpected drops or alters. ##### Step 6. Preview SQL and apply[​](#step-6-preview-sql-and-apply "Direct link to Step 6. Preview SQL and apply") Preview SQL: ```bash alembic upgrade head --sql ``` Pause and review: 1. Confirm the DDL is in the order you expect. 2. Identify any potentially heavy operations and consider splitting migrations if needed. Apply: ```bash alembic upgrade head ``` important StarRocks DDL is not transactional across multiple statements. If an upgrade fails midway, you may need to inspect what has already been applied and perform manual remediation before re-running. ##### Step 7. Make a change and autogenerate again[​](#step-7-make-a-change-and-autogenerate-again "Direct link to Step 7. Make a change and autogenerate again") Update `myapp/models.py` to: * **Modify an existing table** (`my_core_table`): add a column, or update the table comment, and change one table property. * **Add a new table** (`my_new_table`). note Adding a column can be a time-consuming schema change. StarRocks allows only **one running schema change job per table** at a time. In practice, it is recommended to keep “add/drop/modify columns” changes separate from other heavy changes (for example, additional add/drop columns or mass property changes), and split them into multiple Alembic revisions if needed. ```python from sqlalchemy import Column, Table from starrocks import INTEGER, VARCHAR # Modify an existing table (add a column) # (Update the existing my_core_table definition in-place.) my_core_table = Table( "my_core_table", Base.metadata, Column("id", INTEGER, primary_key=True), Column("name", VARCHAR(50)), Column("age", INTEGER), # added column only starrocks_primary_key='id', starrocks_distributed_by='HASH(id) BUCKETS 10', starrocks_properties={"replication_num": "1"}, ) my_new_table = Table( "my_new_table", Base.metadata, Column("id", INTEGER, primary_key=True), Column("name", VARCHAR(50)), starrocks_primary_key="id", starrocks_distributed_by="HASH(id) BUCKETS 10", starrocks_properties={"replication_num": "1"}, ) ``` ```bash alembic revision --autogenerate -m "add a new table, change a old table" ``` Pause and review: Make sure the new migration contains: * a `create_table(...)` for `my_new_table`, and * expected operations for the `my_core_table` changes (for example, add column / set comment / set properties). Preview SQL and apply: ```bash alembic upgrade head --sql alembic upgrade head ``` #### Using sqlacodegen[​](#using-sqlacodegen "Direct link to Using sqlacodegen") [`sqlacodegen`](https://github.com/agronholm/sqlacodegen) can reverse-generate SQLAlchemy models directly from StarRocks: ```bash sqlacodegen --options include_dialect_options,keep_dialect_types \ --generator tables \ starrocks://:@:/[catalog.] > models.py ``` Supported objects: * Tables * Views * Materialized views * Partitioning, distribution, and order-by clauses, and properties This is useful when onboarding an existing StarRocks schema into Alembic. You can directly use above command to generate the Python script for tables/views/materialized views defined in the **End-to-End Example** section. note * It is recommended to add `--generator tables` when generating Core-style models (ORM generators may reorder columns according to `NOT NULL` / `NULL` attribute). * Key columns may be generated as `NOT NULL`. If you want them nullable, adjust the generated model manually. #### Limitations and Best Practices[​](#limitations-and-best-practices "Direct link to Limitations and Best Practices") * Some StarRocks DDL operations require dropping and recreating the table; autogenerate will warn or raise rather than silently producing unavailable SQL. * Keys model changes (for example, changing DUPLICATE KEY to PRIMARY KEY) are not supported via `ALTER TABLE`; use an explicit plan (usually dropping and recreating with backfill). * StarRocks does not provide transactional DDL across multiple statements; review generated migrations and apply them operationally. If a migration fails midway, you may need to handle rollback **manually**. * For distribution, if you omit the `BUCKETS` clause, StarRocks may auto-assign bucket count; the dialect is designed to avoid noisy diffs in that case. #### Summary[​](#summary "Direct link to Summary") With the StarRocks SQLAlchemy dialect and Alembic integration, you can: * ✔ Use declarative models to define StarRocks schemas * ✔ Automatically detect and generate schema migration scripts * ✔ Use version control for schema evolution * ✔ Manage views and materialized views declaratively * ✔ Reverse-engineer existing schemas using sqlacodegen This brings StarRocks schema management into the modern Python data engineering ecosystem and significantly simplifies cross-environment schema consistency. #### References[​](#references "Direct link to References") \[1]: [starrocks-python-client README](https://github.com/StarRocks/starrocks/blob/main/contrib/starrocks-python-client/README.md) \[2]: [Alembic Integration](https://github.com/StarRocks/starrocks/blob/main/contrib/starrocks-python-client/docs/usage_guide/alembic.md) \[3]: [SQLAlchemy details](https://github.com/StarRocks/starrocks/blob/main/contrib/starrocks-python-client/docs/usage_guide/sqlalchemy.md) \[4]: [Table Support](https://github.com/StarRocks/starrocks/blob/main/contrib/starrocks-python-client/docs/usage_guide/tables.md) \[5]: [View Support](https://github.com/StarRocks/starrocks/blob/main/contrib/starrocks-python-client/docs/usage_guide/views.md) \[6]: [Materialized View Support](https://github.com/StarRocks/starrocks/blob/main/contrib/starrocks-python-client/docs/usage_guide/materialized_views.md) --- ### Apache Flink ### Continuously load data from Apache Flink® StarRocks provides a self-developed connector named StarRocks Connector for Apache Flink® (Flink connector for short) to help you load data into a StarRocks table by using Flink. The basic principle is to accumulate the data and then load it all at a time into StarRocks through [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). The Flink connector supports DataStream API, Table API & SQL, and Python API. It has a higher and more stable performance than [flink-connector-jdbc](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/jdbc/) provided by Apache Flink®. > **NOTICE** > > Loading data into StarRocks tables with Flink connector needs SELECT and INSERT privileges on the target StarRocks table. If you do not have these privileges, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant these privileges to the user that you use to connect to your StarRocks cluster. #### Version requirements[​](#version-requirements "Direct link to Version requirements") | Connector | Flink | StarRocks | Java | Scala | | --------- | ----------------------------- | ------------- | ---- | --------- | | 1.2.15 | 1.16,1.17,1.18,1.19,1.20 | 2.1 and later | 8 | 2.11,2.12 | | 1.2.14 | 1.16,1.17,1.18,1.19,1.20 | 2.1 and later | 8 | 2.11,2.12 | | 1.2.12 | 1.16,1.17,1.18,1.19,1.20 | 2.1 and later | 8 | 2.11,2.12 | | 1.2.11 | 1.15,1.16,1.17,1.18,1.19,1.20 | 2.1 and later | 8 | 2.11,2.12 | #### Obtain Flink connector[​](#obtain-flink-connector "Direct link to Obtain Flink connector") You can obtain the Flink connector JAR file in the following ways: * Directly download the compiled Flink connector JAR file. * Add the Flink connector as a dependency in your Maven project and then download the JAR file. * Compile the source code of the Flink connector into a JAR file by yourself. The naming format of the Flink connector JAR file is as follows: * Since Flink 1.15, it's `flink-connector-starrocks-${connector_version}_flink-${flink_version}.jar`. For example, if you install Flink 1.15 and you want to use Flink connector 1.2.7, you can use `flink-connector-starrocks-1.2.7_flink-1.15.jar`. * Prior to Flink 1.15, it's `flink-connector-starrocks-${connector_version}_flink-${flink_version}_${scala_version}.jar`. For example, if you install Flink 1.14 and Scala 2.12 in your environment, and you want to use Flink connector 1.2.7, you can use `flink-connector-starrocks-1.2.7_flink-1.14_2.12.jar`. > **NOTICE** > > In general, the latest version of the Flink connector only maintains compatibility with the three most recent versions of Flink. ##### Download the compiled Jar file[​](#download-the-compiled-jar-file "Direct link to Download the compiled Jar file") Directly download the corresponding version of the Flink connector Jar file from the [Maven Central Repository](https://repo1.maven.org/maven2/com/starrocks). ##### Maven Dependency[​](#maven-dependency "Direct link to Maven Dependency") In your Maven project's `pom.xml` file, add the Flink connector as a dependency according to the following format. Replace `flink_version`, `scala_version`, and `connector_version` with the respective versions. * In Flink 1.15 and later ```xml com.starrocks flink-connector-starrocks ${connector_version}_flink-${flink_version} ``` * In versions earlier than Flink 1.15 ```xml com.starrocks flink-connector-starrocks ${connector_version}_flink-${flink_version}_${scala_version} ``` ##### Compile by yourself[​](#compile-by-yourself "Direct link to Compile by yourself") 1. Download the [Flink connector source code](https://github.com/StarRocks/starrocks-connector-for-apache-flink). 2. Execute the following command to compile the source code of Flink connector into a JAR file. Note that `flink_version` is replaced with the corresponding Flink version. ```bash sh build.sh ``` For example, if the Flink version in your environment is 1.16, you need to execute the following command: ```bash sh build.sh 1.16 ``` 3. Go to the `target/` directory to find the Flink connector JAR file, such as `flink-connector-starrocks-1.2.7_flink-1.16-SNAPSHOT.jar`, generated upon compilation. > **NOTE** > > The name of Flink connector which is not formally released contains the `SNAPSHOT` suffix. #### Options[​](#options "Direct link to Options") ##### General Options[​](#general-options "Direct link to General Options") ###### connector[​](#connector "Direct link to connector") * **Required**: Yes * **Default value**: NONE * **Description**: The connector that you want to use. The value must be "starrocks". ###### jdbc-url[​](#jdbc-url "Direct link to jdbc-url") * **Required**: Yes * **Default value**: NONE * **Description**: The address that is used to connect to the MySQL server of the FE. You can specify multiple addresses, which must be separated by a comma (,). Format: `jdbc:mysql://:,:,:`. ###### load-url[​](#load-url "Direct link to load-url") * **Required**: Yes * **Default value**: NONE * **Description**: The address that is used to connect to the HTTP server of the FE. You can specify multiple addresses, which must be separated by a semicolon (;). Format: `:;:`. ###### database-name[​](#database-name "Direct link to database-name") * **Required**: Yes * **Default value**: NONE * **Description**: The name of the StarRocks database into which you want to load data. ###### table-name[​](#table-name "Direct link to table-name") * **Required**: Yes * **Default value**: NONE * **Description**: The name of the table that you want to use to load data into StarRocks. ###### username[​](#username "Direct link to username") * **Required**: Yes * **Default value**: NONE * **Description**: The username of the account that you want to use to load data into StarRocks. The account needs [SELECT and INSERT privileges](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) on the target StarRocks table. ###### password[​](#password "Direct link to password") * **Required**: Yes * **Default value**: NONE * **Description**: The password of the preceding account. ###### sink.version[​](#sinkversion "Direct link to sink.version") * **Required**: No * **Default value**: AUTO * **Description**: The interface used to load data. This parameter is supported from Flink connector version 1.2.4 onwards. Valid Values: * `V1`: Use [Stream Load](https://docs.starrocks.io/docs/loading/StreamLoad.md) interface to load data. Connectors before 1.2.4 only support this mode. * `V2`: Use [Stream Load transaction](https://docs.starrocks.io/docs/loading/Stream_Load_transaction_interface.md) interface to load data. It requires StarRocks to be at least version 2.4. Recommends `V2` because it optimizes the memory usage and provides a more stable exactly-once implementation. * `AUTO`: If the version of StarRocks supports transaction Stream Load, will choose `V2` automatically, otherwise choose `V1` ###### sink.label-prefix[​](#sinklabel-prefix "Direct link to sink.label-prefix") * **Required**: No * **Default value**: NONE * **Description**: The label prefix used by Stream Load. Recommend to configure it if you are using exactly-once with connector 1.2.8 and later. See [exactly-once usage notes](#exactly-once). ###### sink.semantic[​](#sinksemantic "Direct link to sink.semantic") * **Required**: No * **Default value**: at-least-once * **Description**: The semantic guaranteed by sink. Valid values: **at-least-once** and **exactly-once**. ###### sink.buffer-flush.max-bytes[​](#sinkbuffer-flushmax-bytes "Direct link to sink.buffer-flush.max-bytes") * **Required**: No * **Default value**: 94371840(90M) * **Description**: The maximum size of data that can be accumulated in memory before being sent to StarRocks at a time. The maximum value ranges from 64 MB to 10 GB. Setting this parameter to a larger value can improve loading performance but may increase loading latency. This parameter only takes effect when `sink.semantic` is set to `at-least-once`. If `sink.semantic` is set to `exactly-once`, the data in memory is flushed when a Flink checkpoint is triggered. In this circumstance, this parameter does not take effect. ###### sink.buffer-flush.max-rows[​](#sinkbuffer-flushmax-rows "Direct link to sink.buffer-flush.max-rows") * **Required**: No * **Default value**: 500000 * **Description**: The maximum number of rows that can be accumulated in memory before being sent to StarRocks at a time. This parameter is available only when `sink.version` is `V1` and `sink.semantic` is `at-least-once`. Valid values: 64000 to 5000000. ###### sink.buffer-flush.interval-ms[​](#sinkbuffer-flushinterval-ms "Direct link to sink.buffer-flush.interval-ms") * **Required**: No * **Default value**: 300000 * **Description**: The interval at which data is flushed. This parameter is available only when `sink.semantic` is `at-least-once`. Unit: ms. Valid value range: * For versions earlier than v1.2.14: \[1000, 3600000] * For v1.2.14 and later: (0, 3600000]. ###### sink.max-retries[​](#sinkmax-retries "Direct link to sink.max-retries") * **Required**: No * **Default value**: 3 * **Description**: The number of times that the system retries to perform the Stream Load job. This parameter is available only when you set `sink.version` to `V1`. Valid values: 0 to 10. ###### sink.connect.timeout-ms[​](#sinkconnecttimeout-ms "Direct link to sink.connect.timeout-ms") * **Required**: No * **Default value**: 30000 * **Description**: The timeout for establishing HTTP connection. Valid values: 100 to 60000. Unit: ms. Before Flink connector v1.2.9, the default value is `1000`. ###### sink.socket.timeout-ms[​](#sinksockettimeout-ms "Direct link to sink.socket.timeout-ms") * **Required**: No * **Default value**: -1 * **Description**: Supported since 1.2.10. The time duration for which the HTTP client waits for data. Unit: ms. The default value `-1` means there is no timeout. ###### sink.sanitize-error-log[​](#sinksanitize-error-log "Direct link to sink.sanitize-error-log") * **Required**: No * **Default value**: false * **Description**: Supported since 1.2.12. Whether to sanitize sensitive data in the error log for production security. When this item is set to `true`, sensitive row data and column values in Stream Load error logs are redacted in both the connector and SDK logs. The value defaults to `false` for backward compatibility. ###### sink.wait-for-continue.timeout-ms[​](#sinkwait-for-continuetimeout-ms "Direct link to sink.wait-for-continue.timeout-ms") * **Required**: No * **Default value**: 10000 * **Description**: Supported since 1.2.7. The timeout for waiting response of HTTP 100-continue from the FE. Valid values: `3000` to `60000`. Unit: ms ###### sink.ignore.update-before[​](#sinkignoreupdate-before "Direct link to sink.ignore.update-before") * **Required**: No * **Default value**: true * **Description**: Supported since version 1.2.8. Whether to ignore `UPDATE_BEFORE` records from Flink when loading data to Primary Key tables. If this parameter is set to false, the record is treated as a delete operation to StarRocks table. ###### sink.parallelism[​](#sinkparallelism "Direct link to sink.parallelism") * **Required**: No * **Default value**: NONE * **Description**: The parallelism of loading. Only available for Flink SQL. If this parameter is not specified, Flink planner decides the parallelism. **In the scenario of multi-parallelism, users need to guarantee data is written in the correct order.** ###### sink.properties.\*[​](#sinkproperties "Direct link to sink.properties.*") * **Required**: No * **Default value**: NONE * **Description**: The parameters that are used to control Stream Load behavior. For example, the parameter `sink.properties.format` specifies the format used for Stream Load, such as CSV or JSON. For a list of supported parameters and their descriptions, see [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). ###### sink.properties.format[​](#sinkpropertiesformat "Direct link to sink.properties.format") * **Required**: No * **Default value**: csv * **Description**: The format used for Stream Load. The Flink connector will transform each batch of data to the format before sending them to StarRocks. Valid values: `csv` and `json`. ###### sink.properties.column\_separator[​](#sinkpropertiescolumn_separator "Direct link to sink.properties.column_separator") * **Required**: No * **Default value**: \t * **Description**: The column separator for CSV-formatted data. ###### sink.properties.row\_delimiter[​](#sinkpropertiesrow_delimiter "Direct link to sink.properties.row_delimiter") * **Required**: No * **Default value**: \n * **Description**: The row delimiter for CSV-formatted data. ###### sink.properties.max\_filter\_ratio[​](#sinkpropertiesmax_filter_ratio "Direct link to sink.properties.max_filter_ratio") * **Required**: No * **Default value**: 0 * **Description**: The maximum error tolerance of the Stream Load. It's the maximum percentage of data records that can be filtered out due to inadequate data quality. Valid values: `0` to `1`. Default value: `0`. See [Stream Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md) for details. ###### sink.properties.partial\_update[​](#sinkpropertiespartial_update "Direct link to sink.properties.partial_update") * **Required**: NO * **Default value**: `FALSE` * **Description**: Whether to use partial updates. Valid values: `TRUE` and `FALSE`. Default value: `FALSE`, indicating to disable this feature. ###### sink.properties.partial\_update\_mode[​](#sinkpropertiespartial_update_mode "Direct link to sink.properties.partial_update_mode") * **Required**: NO * **Default value**: `row` * **Description**: Specifies the mode for partial updates. Valid values: `row` and `column`. * The value `row` (default) means partial updates in row mode, which is more suitable for real-time updates with many columns and small batches. * The value `column` means partial updates in column mode, which is more suitable for batch updates with few columns and many rows. In such scenarios, enabling the column mode offers faster update speeds. For example, in a table with 100 columns, if only 10 columns (10% of the total) are updated for all rows, the update speed of the column mode is 10 times faster. ###### sink.properties.strict\_mode[​](#sinkpropertiesstrict_mode "Direct link to sink.properties.strict_mode") * **Required**: No * **Default value**: false * **Description**: Specifies whether to enable the strict mode for Stream Load. It affects the loading behavior when there are unqualified rows, such as inconsistent column values. Valid values: `true` and `false`. Default value: `false`. See [Stream Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md) for details. ###### sink.properties.compression[​](#sinkpropertiescompression "Direct link to sink.properties.compression") * **Required**: No * **Default value**: NONE * **Description**: The compression algorithm used for Stream Load. Valid values: `lz4_frame`. Compression for the JSON format requires Flink connector 1.2.10+ and StarRocks v3.2.7+. Compression for the CSV format only requires Flink connector 1.2.11+. ###### sink.properties.prepared\_timeout[​](#sinkpropertiesprepared_timeout "Direct link to sink.properties.prepared_timeout") * **Required**: No * **Default value**: NONE * **Description**: Supported since 1.2.12 and only effective when `sink.version` is set to `V2`. Requires StarRocks 3.5.4 or later. Sets the timeout in seconds for the Transaction Stream Load phase from `PREPARED` to `COMMITTED`. Typically, only needed for exactly-once; at-least-once usually does not require setting this (the connector defaults to 300s). If not set in exactly-once, StarRocks FE configuration `prepared_transaction_default_timeout_second` (default 86400s) applies. See [StarRocks Transaction timeout management](https://docs.starrocks.io/docs/loading/Stream_Load_transaction_interface.md#transaction-timeout-management). ###### sink.publish-timeout.ms[​](#sinkpublish-timeoutms "Direct link to sink.publish-timeout.ms") * **Required**: No * **Default value**: -1 * **Description**: Supported since 1.2.14 and only effective when `sink.version` is set to `V2`. Timeout in milliseconds for the Publish phase. If a transaction stays in COMMITTED status longer than this timeout, the system will consider it as successful. The default value `-1` means using StarRocks server-side default behavior. When Merge Commit is enabled, the default timeout is 10000 ms. ##### Merge Commit options[​](#merge-commit-options "Direct link to Merge Commit options") Supported from v1.2.14 onwards. Merge Commit allows the system to merge data from multiple subtasks into a single Stream Load transaction for better performance. You can enable this feature by setting `sink.properties.enable_merge_commit` to `true`. For more details about the merge commit feature in StarRocks, see [Merge Commit parameters](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md#merge-commit-parameters). The following Stream Load properties are used to control the Merge Commit behavior: ###### sink.properties.enable\_merge\_commit[​](#sinkpropertiesenable_merge_commit "Direct link to sink.properties.enable_merge_commit") * **Required**: No * **Default value**: false * **Description**: Whether to enable Merge Commit. ###### sink.properties.merge\_commit\_interval\_ms[​](#sinkpropertiesmerge_commit_interval_ms "Direct link to sink.properties.merge_commit_interval_ms") * **Required**: Yes (when Merge Commit is enabled) * **Default value**: NONE * **Description**: The Merge Commit time window in milliseconds. The system merges loading requests received within this window into a single transaction. A larger value improves merging efficiency but increases latency. This property must be set when `enable_merge_commit` is set to `true`. ###### sink.properties.merge\_commit\_parallel[​](#sinkpropertiesmerge_commit_parallel "Direct link to sink.properties.merge_commit_parallel") * **Required**: No * **Default value**: 3 * **Description**: The degree of parallelism for the loading plan created for each Merge Commit-enabled transaction. It is different from `sink.parallelism` which controls the parallelism of the Flink sink operator. ###### sink.properties.merge\_commit\_async[​](#sinkpropertiesmerge_commit_async "Direct link to sink.properties.merge_commit_async") * **Required**: No * **Default value**: true * **Description**: The server's return mode for Merge Commit. The default value is `true` (asynchronous), overriding the system default behavior (synchronous) for better throughput. In the asynchronous mode, the server returns immediately after receiving the data. The connector leverages Flink's checkpoint mechanism to ensure no data loss under the asynchronous mode, providing at-least-once guarantee. In most cases, you do not need to change this value. ###### sink.merge-commit.max-concurrent-requests[​](#sinkmerge-commitmax-concurrent-requests "Direct link to sink.merge-commit.max-concurrent-requests") * **Required**: No * **Default value**: Integer.MAX\_VALUE * **Description**: The maximum number of concurrent Stream Load requests. Set this property to `0` to ensure in-order (serial) loading, which is useful for Primary Key tables. A negative value is treated as `Integer.MAX_VALUE` (unlimited concurrency). ###### sink.merge-commit.chunk.size[​](#sinkmerge-commitchunksize "Direct link to sink.merge-commit.chunk.size") * **Required**: No * **Default value**: 20971520 * **Description**: The maximum size of data (in bytes) accumulated in a chunk before it is flushed and sent to StarRocks via a Stream Load request. A larger value improves throughput but increases memory usage and latency; a smaller value reduces memory usage and latency but may lower throughput. When `max-concurrent-requests` is set to `0` (in-order mode), the default value of this property is changed to 500 MB because only one request runs at a time so a larger batch maximizes throughput. #### Data type mapping between Flink and StarRocks[​](#data-type-mapping-between-flink-and-starrocks "Direct link to Data type mapping between Flink and StarRocks") | Flink data type | StarRocks data type | | ------------------------------------- | ------------------- | | BOOLEAN | BOOLEAN | | TINYINT | TINYINT | | SMALLINT | SMALLINT | | INTEGER | INTEGER | | BIGINT | BIGINT | | FLOAT | FLOAT | | DOUBLE | DOUBLE | | DECIMAL | DECIMAL | | BINARY | INT | | CHAR | STRING | | VARCHAR | STRING | | STRING | STRING | | DATE | DATE | | TIMESTAMP\_WITHOUT\_TIME\_ZONE(N) | DATETIME | | TIMESTAMP\_WITH\_LOCAL\_TIME\_ZONE(N) | DATETIME | | ARRAY\ | ARRAY\ | | MAP\ | JSON STRING | | ROW\ | JSON STRING | #### Usage notes[​](#usage-notes "Direct link to Usage notes") ##### Exactly Once[​](#exactly-once "Direct link to Exactly Once") * If you want sink to guarantee exactly-once semantics, we recommend you to upgrade StarRocks to 2.5 or later, and Flink connector to 1.2.4 or later * Since Flink connector 1.2.4, the exactly-once is redesigned based on [Stream Load transaction interface](https://docs.starrocks.io/docs/loading/Stream_Load_transaction_interface.md) provided by StarRocks since 2.4. Compared to the previous implementation based on non-transactional Stream Load non-transactional interface, the new implementation reduces memory usage and checkpoint overhead, thereby enhancing real-time performance and stability of loading. * If the version of StarRocks is earlier than 2.4 or the version of Flink connector is earlier than 1.2.4, the sink will automatically choose the implementation based on Stream Load non-transactional interface. * Configurations to guarantee exactly-once * The value of `sink.semantic` needs to be `exactly-once`. * If the version of Flink connector is 1.2.8 and later, it is recommended to specify the value of `sink.label-prefix`. Note that the label prefix must be unique among all types of loading in StarRocks, such as Flink jobs, Routine Load, and Broker Load. * If the label prefix is specified, the Flink connector will use the label prefix to clean up lingering transactions that may be generated in some Flink failure scenarios, such as the Flink job fails when a checkpoint is still in progress. These lingering transactions are generally in `PREPARED` status if you use `SHOW PROC '/transactions//running';` to view them in StarRocks. When the Flink job restores from checkpoint, the Flink connector will find these lingering transactions according to the label prefix and some information in checkpoint, and abort them. The Flink connector can not abort them when the Flink job exits because of the two-phase-commit mechanism to implement the exactly-once. When the Flink job exits, the Flink connector has not received the notification from Flink checkpoint coordinator whether the transactions should be included in a successful checkpoint, and it may lead to data loss if these transactions are aborted anyway. You can have an overview about how to achieve end-to-end exactly-once in Flink in this [blogpost](https://flink.apache.org/2018/02/28/an-overview-of-end-to-end-exactly-once-processing-in-apache-flink-with-apache-kafka-too/). * If the label prefix is not specified, lingering transactions will be cleaned up by StarRocks only after they time out. However the number of running transactions can reach the limitation of StarRocks `max_running_txn_num_per_db` if Flink jobs fail frequently before transactions time out. You can set a smaller timeout for `PREPARED` transactions to make them expired faster when the label prefix is not specified. See the following about how to set the prepared timeout. * If you are certain that the Flink job will eventually recover from checkpoint or savepoint after a long downtime because of stop or continuous failover, please adjust the following StarRocks configurations accordingly, to avoid data loss. * Adjust `PREPARED` transaction timeout. See the following about how to set the timeout. The timeout needs to be larger than the downtime of the Flink job. Otherwise, the lingering transactions that are included in a successful checkpoint may be aborted because of timeout before you restart the Flink job, which leads to data loss. Note that when you set a larger value to this configuration, it is better to specify the value of `sink.label-prefix` so that the lingering transactions can be cleaned according to the label prefix and some information in checkpoint, instead of due to timeout (which may cause data loss). * `label_keep_max_second` and `label_keep_max_num`: StarRocks FE configurations, default values are `259200` and `1000` respectively. For details, see [FE configurations](https://docs.starrocks.io/docs/loading/loading_introduction/loading_considerations.md#fe-configurations). The value of `label_keep_max_second` needs to be larger than the downtime of the Flink job. Otherwise, the Flink connector can not check the state of transactions in StarRocks by using the transaction labels saved in the Flink's savepoint or checkpoint and figure out whether these transactions are committed or not, which may eventually lead to data loss. * How to set the timeout for PREPARED transactions * For Connector 1.2.12+ and StarRocks 3.5.4+, you can set the timeout by configuring the connector parameter `sink.properties.prepared_timeout`. By default, the value is not set, and it falls back to the StarRocks FE's global configuration `prepared_transaction_default_timeout_second` (default value is `86400`). * For other versions of Connector or StarRocks, you can set the timeout by configuring the StarRocks FE's global configuration `prepared_transaction_default_timeout_second` (default value is `86400`). ##### Flush Policy[​](#flush-policy "Direct link to Flush Policy") The Flink connector will buffer the data in memory, and flush them in batch to StarRocks via Stream Load. How the flush is triggered is different between at-least-once and exactly-once. For at-least-once, the flush will be triggered when any of the following conditions are met: * the bytes of buffered rows reaches the limit `sink.buffer-flush.max-bytes` * the number of buffered rows reaches the limit `sink.buffer-flush.max-rows`. (Only valid for sink version V1) * the elapsed time since the last flush reaches the limit `sink.buffer-flush.interval-ms` * a checkpoint is triggered For exactly-once, the flush only happens when a checkpoint is triggered. ##### Merge Commit[​](#merge-commit "Direct link to Merge Commit") Merge Commit helps scale throughput without proportionally increasing StarRocks transaction overhead. Without Merge Commit, each Flink sink subtask maintains its own Stream Load transaction, so increasing `sink.parallelism` leads to more concurrent transactions and higher I/O and Compaction costs on StarRocks. Conversely, keeping parallelism low limits the pipeline's overall capacity. With Merge Commit is enabled, data from multiple sink subtasks is merged into a single transaction within each Merge window. This allows you to increase `sink.parallelism` for higher throughput without increasing the number of transactions. For configuration examples, see [Load data with merge commit](#load-data-with-merge-commit). Below are some important notes when using Merge Commit: * **Single parallelism has no benefit** If the Flink sink parallelism is 1, enabling Merge Commit provides no benefit since there is only one subtask sending data. It may even introduce additional latency due to the Merge Commit time window on the server side. * **Only at-least-once semantic** Merge Commit only guarantees at-least-once semantic. It does not support exactly-once semantic. Do not set `sink.semantic` to `exactly-once` when Merge Commit is enabled. * **Ordering for Primary Key tables** By default, `sink.merge-commit.max-concurrent-requests` is `Integer.MAX_VALUE`, which means a single sink subtask may send multiple Stream Load requests concurrently. This can cause out-of-order loading, which may be problematic for Primary Key tables. To ensure in-order loading, set `sink.merge-commit.max-concurrent-requests` to `0`, which, however, will reduce throughput. Alternatively, you can use Conditional Update to prevent newer data from being overwritten by older data. For configuration examples, see [In-order loading for Primary Key tables](#in-order-loading-for-primary-key-tables). * **End-to-end loading latency** The total loading latency consists of two parts: * **Connector batching latency**: Controlled by `sink.buffer-flush.interval-ms` and `sink.merge-commit.chunk.size`. Data is flushed from the connector when either the chunk size limit is reached or the flush interval elapses, whichever comes first. The maximum connector-side latency is `sink.buffer-flush.interval-ms`. A smaller `sink.buffer-flush.interval-ms` reduces connector-side latency but sends data in smaller batches. * **StarRocks merge window**: Controlled by `sink.properties.merge_commit_interval_ms`. The system waits for this duration to merge requests from multiple subtasks into a single transaction. A larger value improves merging efficiency (more requests will be merged into one transaction) but increases server-side latency. * As a general guideline, set `sink.buffer-flush.interval-ms` to be smaller than or equal to `sink.properties.merge_commit_interval_ms`, so that each subtask can flush at least once within each Merge window. For example, if `merge_commit_interval_ms` is `10000` (10s), you could set `sink.buffer-flush.interval-ms` to `5000` (5 seconds) or less. * **Tuning `sink.parallelism` and `sink.properties.merge_commit_parallel`** These two parameters control parallelism at different layers and should be tuned independently: * `sink.parallelism` controls the number of Flink sink subtasks. Each subtask buffers and sends data to StarRocks. Increase this value when Flink sink operators are CPU- or memory-bound — you can monitor Flink's per-operator CPU and memory usage to determine whether more subtasks are needed. * `sink.properties.merge_commit_parallel` controls the degree of parallelism for the loading plan that StarRocks creates for each Merge Commit transaction. Increase this value when StarRocks becomes the bottleneck. You can monitor the StarRocks metrics [merge\_commit\_pending\_total](https://docs.starrocks.io/docs/administration/management/monitoring/metrics.md#merge_commit_pending_total) (number of pending Merge Commit tasks) and [merge\_commit\_pending\_bytes](https://docs.starrocks.io/docs/administration/management/monitoring/metrics.md#merge_commit_pending_bytes) (bytes held by pending tasks) to determine whether more parallelism is needed on the StarRocks side — sustained high values indicate that the loading plan cannot keep up with incoming data. * **Relationship between `sink.merge-commit.chunk.size` and `sink.buffer-flush.max-bytes`**: * `sink.merge-commit.chunk.size` controls the maximum data size per individual Stream Load request (per chunk). When data in a chunk reaches this size, it is flushed immediately. * `sink.buffer-flush.max-bytes` controls the total memory limit for all cached data across all tables. When the total cached data exceeds this limit, the connector will evict chunks early to free memory. * Therefore, `sink.buffer-flush.max-bytes` should be set larger than `sink.merge-commit.chunk.size` to allow at least one full chunk to be accumulated. In general, `sink.buffer-flush.max-bytes` should be several times larger than `sink.merge-commit.chunk.size`, especially when there are multiple tables or high concurrency. ##### Monitoring load metrics[​](#monitoring-load-metrics "Direct link to Monitoring load metrics") The Flink connector provides the following metrics to monitor loading. | Metric | Type | Description | | ------------------------ | ------- | ------------------------------------------------------------------ | | totalFlushBytes | counter | successfully flushed bytes. | | totalFlushRows | counter | number of rows successfully flushed. | | totalFlushSucceededTimes | counter | number of times that the data is successfully flushed. | | totalFlushFailedTimes | counter | number of times that the data fails to be flushed. | | totalFilteredRows | counter | number of rows filtered, which is also included in totalFlushRows. | #### Examples[​](#examples "Direct link to Examples") The following examples show how to use the Flink connector to load data into a StarRocks table with Flink SQL or Flink DataStream. ##### Preparations[​](#preparations "Direct link to Preparations") ###### Create a StarRocks table[​](#create-a-starrocks-table "Direct link to Create a StarRocks table") Create a database `test` and create a Primary Key table `score_board`. ```sql CREATE DATABASE `test`; CREATE TABLE `test`.`score_board` ( `id` int(11) NOT NULL COMMENT "", `name` varchar(65533) NULL DEFAULT "" COMMENT "", `score` int(11) NOT NULL DEFAULT "0" COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`id`) COMMENT "OLAP" DISTRIBUTED BY HASH(`id`); ``` ###### Set up Flink environment[​](#set-up-flink-environment "Direct link to Set up Flink environment") * Download Flink binary [Flink 1.15.2](https://archive.apache.org/dist/flink/flink-1.15.2/flink-1.15.2-bin-scala_2.12.tgz), and unzip it to directory `flink-1.15.2`. * Download [Flink connector 1.2.7](https://repo1.maven.org/maven2/com/starrocks/flink-connector-starrocks/1.2.7_flink-1.15/flink-connector-starrocks-1.2.7_flink-1.15.jar), and put it into the directory `flink-1.15.2/lib`. * Run the following commands to start a Flink cluster: ```shell cd flink-1.15.2 ./bin/start-cluster.sh ``` ###### Network configuration[​](#network-configuration "Direct link to Network configuration") Ensure that the machine where Flink is located can access the FE nodes of the StarRocks cluster via the [`http_port`](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#http_port) (default: `8030`) and [`query_port`](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#query_port) (default: `9030`), and the BE nodes via the [`be_http_port`](https://docs.starrocks.io/docs/administration/management/BE_configuration.md#be_http_port) (default: `8040`). ##### Run with Flink SQL[​](#run-with-flink-sql "Direct link to Run with Flink SQL") * Run the following command to start a Flink SQL client. ```shell ./bin/sql-client.sh ``` * Create a Flink table `score_board`, and insert values into the table via Flink SQL Client. Note you must define the primary key in the Flink DDL if you want to load data into a Primary Key table of StarRocks. It's optional for other types of StarRocks tables. ```sql CREATE TABLE `score_board` ( `id` INT, `name` STRING, `score` INT, PRIMARY KEY (id) NOT ENFORCED ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030', 'load-url' = '127.0.0.1:8030', 'database-name' = 'test', 'table-name' = 'score_board', 'username' = 'root', 'password' = '' ); INSERT INTO `score_board` VALUES (1, 'starrocks', 100), (2, 'flink', 100); ``` ##### Run with Flink DataStream[​](#run-with-flink-datastream "Direct link to Run with Flink DataStream") There are several ways to implement a Flink DataStream job according to the type of the input records, such as a CSV Java `String`, a JSON Java `String` or a custom Java object. * The input records are CSV-format `String`. See [LoadCsvRecords](https://github.com/StarRocks/starrocks-connector-for-apache-flink/tree/cd8086cfedc64d5181785bdf5e89a847dc294c1d/examples/src/main/java/com/starrocks/connector/flink/examples/datastream) for a complete example. ```java /** * Generate CSV-format records. Each record has three values separated by "\t". * These values will be loaded to the columns `id`, `name`, and `score` in the StarRocks table. */ String[] records = new String[]{ "1\tstarrocks-csv\t100", "2\tflink-csv\t100" }; DataStream source = env.fromElements(records); /** * Configure the connector with the required properties. * You also need to add properties "sink.properties.format" and "sink.properties.column_separator" * to tell the connector the input records are CSV-format, and the column separator is "\t". * You can also use other column separators in the CSV-format records, * but remember to modify the "sink.properties.column_separator" correspondingly. */ StarRocksSinkOptions options = StarRocksSinkOptions.builder() .withProperty("jdbc-url", jdbcUrl) .withProperty("load-url", loadUrl) .withProperty("database-name", "test") .withProperty("table-name", "score_board") .withProperty("username", "root") .withProperty("password", "") .withProperty("sink.properties.format", "csv") .withProperty("sink.properties.column_separator", "\t") .build(); // Create the sink with the options. SinkFunction starRockSink = StarRocksSink.sink(options); source.addSink(starRockSink); ``` * The input records are JSON-format `String`. See [LoadJsonRecords](https://github.com/StarRocks/starrocks-connector-for-apache-flink/tree/cd8086cfedc64d5181785bdf5e89a847dc294c1d/examples/src/main/java/com/starrocks/connector/flink/examples/datastream) for a complete example. ```java /** * Generate JSON-format records. * Each record has three key-value pairs corresponding to the columns `id`, `name`, and `score` in the StarRocks table. */ String[] records = new String[]{ "{\"id\":1, \"name\":\"starrocks-json\", \"score\":100}", "{\"id\":2, \"name\":\"flink-json\", \"score\":100}", }; DataStream source = env.fromElements(records); /** * Configure the connector with the required properties. * You also need to add properties "sink.properties.format" and "sink.properties.strip_outer_array" * to tell the connector the input records are JSON-format and to strip the outermost array structure. */ StarRocksSinkOptions options = StarRocksSinkOptions.builder() .withProperty("jdbc-url", jdbcUrl) .withProperty("load-url", loadUrl) .withProperty("database-name", "test") .withProperty("table-name", "score_board") .withProperty("username", "root") .withProperty("password", "") .withProperty("sink.properties.format", "json") .withProperty("sink.properties.strip_outer_array", "true") .build(); // Create the sink with the options. SinkFunction starRockSink = StarRocksSink.sink(options); source.addSink(starRockSink); ``` * The input records are custom Java objects. See [LoadCustomJavaRecords](https://github.com/StarRocks/starrocks-connector-for-apache-flink/tree/cd8086cfedc64d5181785bdf5e89a847dc294c1d/examples/src/main/java/com/starrocks/connector/flink/examples/datastream) for a complete example. * In this example, the input record is a simple POJO `RowData`. ```java public static class RowData { public int id; public String name; public int score; public RowData() {} public RowData(int id, String name, int score) { this.id = id; this.name = name; this.score = score; } } ``` * The main program is as follows: ```java // Generate records which use RowData as the container. RowData[] records = new RowData[]{ new RowData(1, "starrocks-rowdata", 100), new RowData(2, "flink-rowdata", 100), }; DataStream source = env.fromElements(records); // Configure the connector with the required properties. StarRocksSinkOptions options = StarRocksSinkOptions.builder() .withProperty("jdbc-url", jdbcUrl) .withProperty("load-url", loadUrl) .withProperty("database-name", "test") .withProperty("table-name", "score_board") .withProperty("username", "root") .withProperty("password", "") .build(); /** * The Flink connector will use a Java object array (Object[]) to represent a row to be loaded into the StarRocks table, * and each element is the value for a column. * You need to define the schema of the Object[] which matches that of the StarRocks table. */ TableSchema schema = TableSchema.builder() .field("id", DataTypes.INT().notNull()) .field("name", DataTypes.STRING()) .field("score", DataTypes.INT()) // When the StarRocks table is a Primary Key table, you must specify notNull(), for example, DataTypes.INT().notNull(), for the primary key `id`. .primaryKey("id") .build(); // Transform the RowData to the Object[] according to the schema. RowDataTransformer transformer = new RowDataTransformer(); // Create the sink with the schema, options, and transformer. SinkFunction starRockSink = StarRocksSink.sink(schema, options, transformer); source.addSink(starRockSink); ``` * The `RowDataTransformer` in the main program is defined as follows: ```java private static class RowDataTransformer implements StarRocksSinkRowBuilder { /** * Set each element of the object array according to the input RowData. * The schema of the array matches that of the StarRocks table. */ @Override public void accept(Object[] internalRow, RowData rowData) { internalRow[0] = rowData.id; internalRow[1] = rowData.name; internalRow[2] = rowData.score; // When the StarRocks table is a Primary Key table, you need to set the last element to indicate whether the data loading is an UPSERT or DELETE operation. internalRow[internalRow.length - 1] = StarRocksSinkOP.UPSERT.ordinal(); } } ``` ##### Synchronize data with Flink CDC 3.0 (with schema change supported)[​](#synchronize-data-with-flink-cdc-30-with-schema-change-supported "Direct link to Synchronize data with Flink CDC 3.0 (with schema change supported)") [Flink CDC 3.0](https://nightlies.apache.org/flink/flink-cdc-docs-stable) framework can be used to easily build a streaming ELT pipeline from CDC sources (such as MySQL and Kafka) to StarRocks. The pipeline can synchronize whole database, merged sharding tables, and schema changes from sources to StarRocks. Since v1.2.9, the Flink connector for StarRocks is integrated into this framework as [StarRocks Pipeline Connector](https://nightlies.apache.org/flink/flink-cdc-docs-release-3.1/docs/connectors/pipeline-connectors/starrocks/). The StarRocks Pipeline Connector supports: * Automatic creation of databases and tables * Schema change synchronization * Full and incremental data synchronization For quick start, see [Streaming ELT from MySQL to StarRocks using Flink CDC 3.0 with StarRocks Pipeline Connector](https://nightlies.apache.org/flink/flink-cdc-docs-release-3.4/docs/get-started/quickstart/mysql-to-starrocks/). It is advised to use StarRocks v3.2.1 and later versions to enable [fast\_schema\_evolution](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md#set-fast-schema-evolution). It will improve the speed of adding or dropping columns and reduce resource usage. #### Best practices[​](#best-practices "Direct link to Best practices") ##### Load data to a Primary Key table[​](#load-data-to-a-primary-key-table "Direct link to Load data to a Primary Key table") This section will show how to load data to a StarRocks Primary Key table to achieve partial updates and conditional updates. You can see [Change data through loading](https://docs.starrocks.io/docs/loading/Load_to_Primary_Key_tables.md) for the introduction of those features. These examples use Flink SQL. ###### Preparations[​](#preparations-1 "Direct link to Preparations") Create a database `test` and create a Primary Key table `score_board` in StarRocks. ```sql CREATE DATABASE `test`; CREATE TABLE `test`.`score_board` ( `id` int(11) NOT NULL COMMENT "", `name` varchar(65533) NULL DEFAULT "" COMMENT "", `score` int(11) NOT NULL DEFAULT "0" COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`id`) COMMENT "OLAP" DISTRIBUTED BY HASH(`id`); ``` ###### Partial update[​](#partial-update "Direct link to Partial update") This example will show how to load data only to columns `id` and `name`. 1. Insert two data rows into the StarRocks table `score_board` in MySQL client. ```sql mysql> INSERT INTO `score_board` VALUES (1, 'starrocks', 100), (2, 'flink', 100); mysql> select * from score_board; +------+-----------+-------+ | id | name | score | +------+-----------+-------+ | 1 | starrocks | 100 | | 2 | flink | 100 | +------+-----------+-------+ 2 rows in set (0.02 sec) ``` 2. Create a Flink table `score_board` in Flink SQL client. * Define the DDL which only includes the columns `id` and `name`. * Set the option `sink.properties.partial_update` to `true` which tells the Flink connector to perform partial updates. * If the Flink connector version `<=` 1.2.7, you also need to set the option `sink.properties.columns` to `id,name,__op` to tells the Flink connector which columns need to be updated. Note that you need to append the field `__op` at the end. The field `__op` indicates that the data loading is an UPSERT or DELETE operation, and its values are set by the connector automatically. ```sql CREATE TABLE `score_board` ( `id` INT, `name` STRING, PRIMARY KEY (id) NOT ENFORCED ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030', 'load-url' = '127.0.0.1:8030', 'database-name' = 'test', 'table-name' = 'score_board', 'username' = 'root', 'password' = '', 'sink.properties.partial_update' = 'true', -- only for Flink connector version <= 1.2.7 'sink.properties.columns' = 'id,name,__op' ); ``` 3. Insert two data rows into the Flink table. The primary keys of the data rows are as same as these of rows in the StarRocks table. but the values in the column `name` are modified. ```sql INSERT INTO `score_board` VALUES (1, 'starrocks-update'), (2, 'flink-update'); ``` 4. Query the StarRocks table in MySQL client. ```sql mysql> select * from score_board; +------+------------------+-------+ | id | name | score | +------+------------------+-------+ | 1 | starrocks-update | 100 | | 2 | flink-update | 100 | +------+------------------+-------+ 2 rows in set (0.02 sec) ``` You can see that only values for `name` change, and the values for `score` do not change. ###### Conditional update[​](#conditional-update "Direct link to Conditional update") This example will show how to do conditional update according to the value of column `score`. The update for an `id` takes effect only when the new value for `score` is has a greater or equal to the old value. 1. Insert two data rows into the StarRocks table in MySQL client. ```sql mysql> INSERT INTO `score_board` VALUES (1, 'starrocks', 100), (2, 'flink', 100); mysql> select * from score_board; +------+-----------+-------+ | id | name | score | +------+-----------+-------+ | 1 | starrocks | 100 | | 2 | flink | 100 | +------+-----------+-------+ 2 rows in set (0.02 sec) ``` 2. Create a Flink table `score_board` in the following ways: * Define the DDL including all of columns. * Set the option `sink.properties.merge_condition` to `score` to tell the connector to use the column `score` as the condition. * Set the option `sink.version` to `V1` or `V2`. Both support conditional update. ```sql CREATE TABLE `score_board` ( `id` INT, `name` STRING, `score` INT, PRIMARY KEY (id) NOT ENFORCED ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030', 'load-url' = '127.0.0.1:8030', 'database-name' = 'test', 'table-name' = 'score_board', 'username' = 'root', 'password' = '', 'sink.properties.merge_condition' = 'score', 'sink.version' = 'V1' ); ``` 3. Insert two data rows into the Flink table. The primary keys of the data rows are as same as these of rows in the StarRocks table. The first data row has a smaller value in the column `score`, and the second data row has a larger value in the column `score`. ```sql INSERT INTO `score_board` VALUES (1, 'starrocks-update', 99), (2, 'flink-update', 101); ``` 4. Query the StarRocks table in MySQL client. ```sql mysql> select * from score_board; +------+--------------+-------+ | id | name | score | +------+--------------+-------+ | 1 | starrocks | 100 | | 2 | flink-update | 101 | +------+--------------+-------+ 2 rows in set (0.03 sec) ``` You can see that only the values of the second data row change, and the values of the first data row do not change. ##### Load data with Merge Commit[​](#load-data-with-merge-commit "Direct link to Load data with Merge Commit") This section shows how to use Merge Commit to improve loading throughput when you have multiple Flink sink subtasks writing to the same StarRocks table. These examples use Flink SQL and StarRocks v3.4.0 or later. ###### Preparations[​](#preparations-2 "Direct link to Preparations") Create a database `test` and create a Primary Key table `score_board` in StarRocks. ```sql CREATE DATABASE `test`; CREATE TABLE `test`.`score_board` ( `id` int(11) NOT NULL COMMENT "", `name` varchar(65533) NULL DEFAULT "" COMMENT "", `score` int(11) NOT NULL DEFAULT "0" COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`id`) COMMENT "OLAP" DISTRIBUTED BY HASH(`id`); ``` ###### Basic configuration[​](#basic-configuration "Direct link to Basic configuration") This Flink SQL enables merge commit with a 10-second merge window. Data from all sink subtasks is merged into a single transaction within each window. ```sql CREATE TABLE `score_board` ( `id` INT, `name` STRING, `score` INT, PRIMARY KEY (id) NOT ENFORCED ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030', 'load-url' = '127.0.0.1:8030', 'database-name' = 'test', 'table-name' = 'score_board', 'username' = 'root', 'password' = '', 'sink.properties.enable_merge_commit' = 'true', 'sink.properties.merge_commit_interval_ms' = '10000', 'sink.buffer-flush.interval-ms' = '5000' ); ``` Insert data into the Flink table. The data will be loaded into StarRocks via merge commit. ```sql INSERT INTO `score_board` VALUES (1, 'starrocks', 100), (2, 'flink', 95), (3, 'spark', 90); ``` ###### In-order loading for Primary Key tables[​](#in-order-loading-for-primary-key-tables "Direct link to In-order loading for Primary Key tables") By default, a single sink subtask may send multiple Stream Load requests concurrently, which can cause out-of-order loading. For Primary Key tables where data ordering matters, there are two approaches to handle this issue. **Method 1: Use `sink.merge-commit.max-concurrent-requests`** Set `sink.merge-commit.max-concurrent-requests` to `0` to ensure each subtask sends requests one at a time. This guarantees in-order loading but may reduce throughput. ```sql CREATE TABLE `score_board` ( `id` INT, `name` STRING, `score` INT, PRIMARY KEY (id) NOT ENFORCED ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030', 'load-url' = '127.0.0.1:8030', 'database-name' = 'test', 'table-name' = 'score_board', 'username' = 'root', 'password' = '', 'sink.properties.enable_merge_commit' = 'true', 'sink.properties.merge_commit_interval_ms' = '10000', 'sink.buffer-flush.interval-ms' = '5000', 'sink.merge-commit.max-concurrent-requests' = '0' ); INSERT INTO `score_board` VALUES (1, 'starrocks', 100), (2, 'flink', 95), (3, 'spark', 90); ``` **Method 2: Use Conditional Update** If you want to keep concurrent requests for higher throughput but still prevent older data from overwriting newer data, you can use [Conditional Update](#conditional-update). Set `sink.properties.merge_condition` to a column (for example, a version or timestamp column) so that an update only takes effect when the incoming value is greater than or equal to the existing value. ```sql CREATE TABLE `score_board` ( `id` INT, `name` STRING, `score` INT, PRIMARY KEY (id) NOT ENFORCED ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030', 'load-url' = '127.0.0.1:8030', 'database-name' = 'test', 'table-name' = 'score_board', 'username' = 'root', 'password' = '', 'sink.properties.enable_merge_commit' = 'true', 'sink.properties.merge_commit_interval_ms' = '10000', 'sink.buffer-flush.interval-ms' = '5000', 'sink.properties.merge_condition' = 'score' ); INSERT INTO `score_board` VALUES (1, 'starrocks', 100), (2, 'flink', 95), (3, 'spark', 90); ``` With this configuration, concurrent requests are allowed (default `sink.merge-commit.max-concurrent-requests` is `Integer.MAX_VALUE`), but an update to a row only takes effect when the new `score` is greater than or equal to the existing `score`. This prevents newer data from being overwritten by older data even under out-of-order loading. ##### Load data into columns of BITMAP type[​](#load-data-into-columns-of-bitmap-type "Direct link to Load data into columns of BITMAP type") [`BITMAP`](https://docs.starrocks.io/docs/sql-reference/data-types/other-data-types/BITMAP.md) is often used to accelerate count distinct, such as counting UV, see [Use Bitmap for exact Count Distinct](https://docs.starrocks.io/docs/using_starrocks/distinct_values/Using_bitmap.md). Here we take the counting of UV as an example to show how to load data into columns of the `BITMAP` type. 1. Create a StarRocks Aggregate table in MySQL client. In the database `test`, create an Aggregate table `page_uv` where the column `visit_users` is defined as the `BITMAP` type and configured with the aggregate function `BITMAP_UNION`. ```sql CREATE TABLE `test`.`page_uv` ( `page_id` INT NOT NULL COMMENT 'page ID', `visit_date` datetime NOT NULL COMMENT 'access time', `visit_users` BITMAP BITMAP_UNION NOT NULL COMMENT 'user ID' ) ENGINE=OLAP AGGREGATE KEY(`page_id`, `visit_date`) DISTRIBUTED BY HASH(`page_id`); ``` 2. Create a Flink table in Flink SQL client. The column `visit_user_id` in the Flink table is of `BIGINT` type, and we want to load this column to the column `visit_users` of `BITMAP` type in the StarRocks table. So when defining the DDL of the Flink table, note that: * Because Flink does not support `BITMAP`, you need to define a column `visit_user_id` as `BIGINT` type to represent the column `visit_users` of `BITMAP` type in the StarRocks table. * You need to set the option `sink.properties.columns` to `page_id,visit_date,user_id,visit_users=to_bitmap(visit_user_id)`, which tells the connector the column mapping between the Flink table and StarRocks table. Also you need to use [`to_bitmap`](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/to_bitmap.md) function to tell the connector to convert the data of `BIGINT` type into `BITMAP` type. ```sql CREATE TABLE `page_uv` ( `page_id` INT, `visit_date` TIMESTAMP, `visit_user_id` BIGINT ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030', 'load-url' = '127.0.0.1:8030', 'database-name' = 'test', 'table-name' = 'page_uv', 'username' = 'root', 'password' = '', 'sink.properties.columns' = 'page_id,visit_date,visit_user_id,visit_users=to_bitmap(visit_user_id)' ); ``` 3. Load data into Flink table in Flink SQL client. ```sql INSERT INTO `page_uv` VALUES (1, CAST('2020-06-23 01:30:30' AS TIMESTAMP), 13), (1, CAST('2020-06-23 01:30:30' AS TIMESTAMP), 23), (1, CAST('2020-06-23 01:30:30' AS TIMESTAMP), 33), (1, CAST('2020-06-23 02:30:30' AS TIMESTAMP), 13), (2, CAST('2020-06-23 01:30:30' AS TIMESTAMP), 23); ``` 4. Calculate page UVs from the StarRocks table in MySQL client. ```sql MySQL [test]> SELECT `page_id`, COUNT(DISTINCT `visit_users`) FROM `page_uv` GROUP BY `page_id`; +---------+-----------------------------+ | page_id | count(DISTINCT visit_users) | +---------+-----------------------------+ | 2 | 1 | | 1 | 3 | +---------+-----------------------------+ 2 rows in set (0.05 sec) ``` ##### Load data into columns of HLL type[​](#load-data-into-columns-of-hll-type "Direct link to Load data into columns of HLL type") [`HLL`](https://docs.starrocks.io/docs/sql-reference/data-types/other-data-types/HLL.md) can be used for approximate count distinct, see [Use HLL for approximate count distinct](https://docs.starrocks.io/docs/using_starrocks/distinct_values/Using_HLL.md). Here we take the counting of UV as an example to show how to load data into columns of the `HLL` type. 1. Create a StarRocks Aggregate table In the database `test`, create an Aggregate table `hll_uv` where the column `visit_users` is defined as the `HLL` type and configured with the aggregate function `HLL_UNION`. ```sql CREATE TABLE `hll_uv` ( `page_id` INT NOT NULL COMMENT 'page ID', `visit_date` datetime NOT NULL COMMENT 'access time', `visit_users` HLL HLL_UNION NOT NULL COMMENT 'user ID' ) ENGINE=OLAP AGGREGATE KEY(`page_id`, `visit_date`) DISTRIBUTED BY HASH(`page_id`); ``` 2. Create a Flink table in Flink SQL client. The column `visit_user_id` in the Flink table is of `BIGINT` type, and we want to load this column to the column `visit_users` of `HLL` type in the StarRocks table. So when defining the DDL of the Flink table, note that: * Because Flink does not support `BITMAP`, you need to define a column `visit_user_id` as `BIGINT` type to represent the column `visit_users` of `HLL` type in the StarRocks table. * You need to set the option `sink.properties.columns` to `page_id,visit_date,user_id,visit_users=hll_hash(visit_user_id)` which tells the connector the column mapping between Flink table and StarRocks table. Also you need to use [`hll_hash`](https://docs.starrocks.io/docs/sql-reference/sql-functions/scalar-functions/hll_hash.md) function to tell the connector to convert the data of `BIGINT` type into `HLL` type. ```sql CREATE TABLE `hll_uv` ( `page_id` INT, `visit_date` TIMESTAMP, `visit_user_id` BIGINT ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030', 'load-url' = '127.0.0.1:8030', 'database-name' = 'test', 'table-name' = 'hll_uv', 'username' = 'root', 'password' = '', 'sink.properties.columns' = 'page_id,visit_date,visit_user_id,visit_users=hll_hash(visit_user_id)' ); ``` 3. Load data into Flink table in Flink SQL client. ```sql INSERT INTO `hll_uv` VALUES (3, CAST('2023-07-24 12:00:00' AS TIMESTAMP), 78), (4, CAST('2023-07-24 13:20:10' AS TIMESTAMP), 2), (3, CAST('2023-07-24 12:30:00' AS TIMESTAMP), 674); ``` 4. Calculate page UVs from the StarRocks table in MySQL client. ```sql mysql> SELECT `page_id`, COUNT(DISTINCT `visit_users`) FROM `hll_uv` GROUP BY `page_id`; **+---------+-----------------------------+ | page_id | count(DISTINCT visit_users) | +---------+-----------------------------+ | 3 | 2 | | 4 | 1 | +---------+-----------------------------+ 2 rows in set (0.04 sec) ``` --- ### Kafka connector for StarRocks ### Load data using Kafka connector StarRocks provides a self-developed connector named Apache Kafka® connector (StarRocks Connector for Apache Kafka®, Kafka connector for short), as a sink connector, that continuously consumes messages from Kafka and loads them into StarRocks. The Kafka connector guarantees at-least-once semantics. The Kafka connector can seamlessly integrate with Kafka Connect, which allows StarRocks better integrated with the Kafka ecosystem. It is a wise choice if you want to load real-time data into StarRocks. Compared with Routine Load, it is recommended to use the Kafka connector in the following scenarios: * Compared with Routine Load which only supports loading data in CSV, JSON, and Avro formats, Kafka connector can load data in more formats, such as Protobuf. As long as data can be converted into JSON and CSV formats using Kafka Connect's converters, data can be loaded into StarRocks via the Kafka connector. * Customize data transformation, such as Debezium-formatted CDC data. * Load data from multiple Kafka topics. * Load data from Confluent Cloud. * Need finer control over load batch sizes, parallelism, and other parameters to achieve a balance between load speed and resource utilization. #### Preparations[​](#preparations "Direct link to Preparations") ##### Version requirements[​](#version-requirements "Direct link to Version requirements") | Connector | Kafka | StarRocks | Java | | --------- | --------- | ------------- | ---- | | 1.0.6 | 3.4+/4.0+ | 2.5 and later | 8 | | 1.0.5 | 3.4 | 2.5 and later | 8 | | 1.0.4 | 3.4 | 2.5 and later | 8 | | 1.0.3 | 3.4 | 2.5 and later | 8 | ##### Set up Kafka environment[​](#set-up-kafka-environment "Direct link to Set up Kafka environment") Both self-managed Apache Kafka clusters and Confluent Cloud are supported. * For a self-managed Apache Kafka cluster, you can refer to [Apache Kafka quickstart](https://kafka.apache.org/quickstart) to quickly deploy a Kafka cluster. Kafka Connect is already integrated into Kafka. * For Confluent Cloud, make sure that you have a Confluent account and have created a cluster. ##### Download Kafka connector[​](#download-kafka-connector "Direct link to Download Kafka connector") Submit the Kafka connector into Kafka Connect: * Self-managed Kafka cluster: Download [starrocks-connector-for-kafka-x.y.z-with-dependencies.jar](https://github.com/StarRocks/starrocks-connector-for-kafka/releases). * Confluent Cloud: Currently, the Kafka connector is not uploaded to Confluent Hub. You need to download [starrocks-connector-for-kafka-x.y.z-with-dependencies.jar](https://github.com/StarRocks/starrocks-connector-for-kafka/releases), package it into a ZIP file and upload the ZIP file to Confluent Cloud. ##### Network configuration[​](#network-configuration "Direct link to Network configuration") Ensure that the machine where Kafka is located can access the FE nodes of the StarRocks cluster via the [`http_port`](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#http_port) (default: `8030`) and [`query_port`](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#query_port) (default: `9030`), and the BE nodes via the [`be_http_port`](https://docs.starrocks.io/docs/administration/management/BE_configuration.md#be_http_port) (default: `8040`). #### Usage[​](#usage "Direct link to Usage") This section uses a self-managed Kafka cluster as an example to explain how to configure the Kafka connector and the Kafka Connect, and then run the Kafka Connect to load data into StarRocks. ##### Prepare a dataset[​](#prepare-a-dataset "Direct link to Prepare a dataset") Suppose that JSON-format data exists in the topic `test` in a Kafka cluster. ```json {"id":1,"city":"New York"} {"id":2,"city":"Los Angeles"} {"id":3,"city":"Chicago"} ``` ##### Create a table[​](#create-a-table "Direct link to Create a table") Create the table `test_tbl` in the database `example_db` in the StarRocks cluster according to the keys of the JSON-format data. ```sql CREATE DATABASE example_db; USE example_db; CREATE TABLE test_tbl (id INT, city STRING); ``` ##### Configure Kafka connector and Kafka Connect, and then run Kafka Connect to load data[​](#configure-kafka-connector-and-kafka-connect-and-then-run-kafka-connect-to-load-data "Direct link to Configure Kafka connector and Kafka Connect, and then run Kafka Connect to load data") ###### Run Kafka Connect in standalone mode[​](#run-kafka-connect-in-standalone-mode "Direct link to Run Kafka Connect in standalone mode") 1. Configure the Kafka connector. In the **config** directory under the Kafka installation directory, create the configuration file **connect-StarRocks-sink.properties** for the Kafka connector, and configure the following parameters. For more parameters and descriptions, see [Parameters](#Parameters). info * In this example, the Kafka connector provided by StarRocks is a sink connector that can continuously consume data from Kafka and load data into StarRocks. * If the source data is CDC data, such as data in Debezium format, and the StarRocks table is a Primary Key table, you also need to [configure `transform`](#load-debezium-formatted-cdc-data) in the configuration file **connect-StarRocks-sink.properties** for the Kafka connector provided by StarRocks, to synchronize the source data changes to the Primary Key table. ```yaml name=starrocks-kafka-connector connector.class=com.starrocks.connector.kafka.StarRocksSinkConnector topics=test key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter key.converter.schemas.enable=true value.converter.schemas.enable=false # The HTTP URL of the FE in your StarRocks cluster. The default port is 8030. starrocks.http.url=192.168.xxx.xxx:8030 # If the Kafka topic name is different from the StarRocks table name, you need to configure the mapping relationship between them. starrocks.topic2table.map=test:test_tbl # Enter the StarRocks username. starrocks.username=user1 # Enter the StarRocks password. starrocks.password=123456 starrocks.database.name=example_db sink.properties.strip_outer_array=true ``` 2. Configure and run the Kafka Connect. 1. Configure the Kafka Connect. In the configuration file **config/connect-standalone.properties** in the **config** directory, configure the following parameters. For more parameters and descriptions, see [Running Kafka Connect](https://kafka.apache.org/documentation.html#connect_running). ```yaml # The addresses of Kafka brokers. Multiple addresses of Kafka brokers need to be separated by commas (,). # Note that this example uses PLAINTEXT as the security protocol to access the Kafka cluster. If you are using other security protocol to access the Kafka cluster, you need to configure the relevant information in this file. bootstrap.servers=:9092 offset.storage.file.filename=/tmp/connect.offsets offset.flush.interval.ms=10000 key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter key.converter.schemas.enable=true value.converter.schemas.enable=false # The absolute path of starrocks-connector-for-kafka-x.y.z-with-dependencies.jar. plugin.path=/home/kafka-connect/starrocks-kafka-connector ``` 2. Run the Kafka Connect. ```bash CLASSPATH=/home/kafka-connect/starrocks-kafka-connector/* bin/connect-standalone.sh config/connect-standalone.properties config/connect-starrocks-sink.properties ``` ###### Run Kafka Connect in distributed mode[​](#run-kafka-connect-in-distributed-mode "Direct link to Run Kafka Connect in distributed mode") 1. Configure and run the Kafka Connect. 1. Configure the Kafka Connect. In the configuration file `config/connect-distributed.properties` in the **config** directory, configure the following parameters. For more parameters and descriptions, refer to [Running Kafka Connect](https://kafka.apache.org/documentation.html#connect_running). ```yaml # The addresses of Kafka brokers. Multiple addresses of Kafka brokers need to be separated by commas (,). # Note that this example uses PLAINTEXT as the security protocol to access the Kafka cluster. If you are using other security protocol to access the Kafka cluster, you need to configure the relevant information in this file. bootstrap.servers=:9092 offset.storage.file.filename=/tmp/connect.offsets offset.flush.interval.ms=10000 key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter key.converter.schemas.enable=true value.converter.schemas.enable=false # The absolute path of starrocks-connector-for-kafka-x.y.z-with-dependencies.jar. plugin.path=/home/kafka-connect/starrocks-kafka-connector ``` 2. Run the Kafka Connect. ```bash CLASSPATH=/home/kafka-connect/starrocks-kafka-connector/* bin/connect-distributed.sh config/connect-distributed.properties ``` 2. Configure and create the Kafka connector. Note that in distributed mode, you need to configure and create the Kafka connector through the REST API. For parameters and descriptions, see [Parameters](#Parameters). info * In this example, the Kafka connector provided by StarRocks is a sink connector that can continuously consume data from Kafka and load data into StarRocks. * If the source data is CDC data, such as data in Debezium format, and the StarRocks table is a Primary Key table, you also need to [configure `transform`](#load-debezium-formatted-cdc-data) in the configuration file **connect-StarRocks-sink.properties** for the Kafka connector provided by StarRocks, to synchronize the source data changes to the Primary Key table. ```shell curl -i http://127.0.0.1:8083/connectors -H "Content-Type: application/json" -X POST -d '{ "name":"starrocks-kafka-connector", "config":{ "connector.class":"com.starrocks.connector.kafka.StarRocksSinkConnector", "topics":"test", "key.converter":"org.apache.kafka.connect.json.JsonConverter", "value.converter":"org.apache.kafka.connect.json.JsonConverter", "key.converter.schemas.enable":"true", "value.converter.schemas.enable":"false", "starrocks.http.url":"192.168.xxx.xxx:8030", "starrocks.topic2table.map":"test:test_tbl", "starrocks.username":"user1", "starrocks.password":"123456", "starrocks.database.name":"example_db", "sink.properties.strip_outer_array":"true" } }' ``` ###### Query StarRocks table[​](#query-starrocks-table "Direct link to Query StarRocks table") Query the target StarRocks table `test_tbl`. ```mysql MySQL [example_db]> select * from test_tbl; +------+-------------+ | id | city | +------+-------------+ | 1 | New York | | 2 | Los Angeles | | 3 | Chicago | +------+-------------+ 3 rows in set (0.01 sec) ``` The data is successfully loaded when the above result is returned. #### Parameters[​](#parameters "Direct link to Parameters") ##### name[​](#name "Direct link to name") **Required**: YES
**Default value**:
**Description**: Name for this Kafka connector. It must be globally unique among all Kafka connectors within this Kafka Connect cluster. For example, starrocks-kafka-connector. ##### connector.class[​](#connectorclass "Direct link to connector.class") **Required**: YES
**Default value**:
**Description**: Class used by this Kafka connector's sink. Set the value to `com.starrocks.connector.kafka.StarRocksSinkConnector`. ##### topics[​](#topics "Direct link to topics") **Required**:
**Default value**:
**Description**: One or more topics to subscribe to, where each topic corresponds to a StarRocks table. By default, StarRocks assumes that the topic name matches the name of the StarRocks table. So StarRocks determines the target StarRocks table by using the topic name. Please choose either to fill in `topics` or `topics.regex` (below), but not both. However, if the StarRocks table name is not the same as the topic name, then use the optional `starrocks.topic2table.map` parameter (below) to specify the mapping from topic name to table name. ##### topics.regex[​](#topicsregex "Direct link to topics.regex") **Required**:
**Default value**: **Description**: Regular expression to match the one or more topics to subscribe to. For more description, see `topics`. Please choose either to fill in `topics.regex` or `topics` (above), but not both.
##### starrocks.topic2table.map[​](#starrockstopic2tablemap "Direct link to starrocks.topic2table.map") **Required**: NO
**Default value**:
**Description**: The mapping of the StarRocks table name and the topic name when the topic name is different from the StarRocks table name. The format is `:,:,...`. ##### starrocks.http.url[​](#starrockshttpurl "Direct link to starrocks.http.url") **Required**: YES
**Default value**:
**Description**: The HTTP URL of the FE in your StarRocks cluster. The format is `:,:,...`. Multiple addresses are separated by commas (,). For example, `192.168.xxx.xxx:8030,192.168.xxx.xxx:8030`. ##### starrocks.database.name[​](#starrocksdatabasename "Direct link to starrocks.database.name") **Required**: YES
**Default value**:
**Description**: The name of StarRocks database. ##### starrocks.username[​](#starrocksusername "Direct link to starrocks.username") **Required**: YES
**Default value**:
**Description**: The username of your StarRocks cluster account. The user needs the [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) privilege on the StarRocks table. ##### starrocks.password[​](#starrockspassword "Direct link to starrocks.password") **Required**: YES
**Default value**:
**Description**: The password of your StarRocks cluster account. ##### key.converter[​](#keyconverter "Direct link to key.converter") **Required**: NO
**Default value**: Key converter used by Kafka Connect cluster
**Description**: This parameter specifies the key converter for the sink connector (Kafka-connector-starrocks), which is used to deserialize the keys of Kafka data. The default key converter is the one used by Kafka Connect cluster. ##### value.converter[​](#valueconverter "Direct link to value.converter") **Required**: NO
**Default value**: Value converter used by Kafka Connect cluster
**Description**: This parameter specifies the value converter for the sink connector (Kafka-connector-starrocks), which is used to deserialize the values of Kafka data. The default value converter is the one used by Kafka Connect cluster. ##### key.converter.schema.registry.url[​](#keyconverterschemaregistryurl "Direct link to key.converter.schema.registry.url") **Required**: NO
**Default value**:
**Description**: Schema registry URL for the key converter. ##### value.converter.schema.registry.url[​](#valueconverterschemaregistryurl "Direct link to value.converter.schema.registry.url") **Required**: NO
**Default value**:
**Description**: Schema registry URL for the value converter. ##### tasks.max[​](#tasksmax "Direct link to tasks.max") **Required**: NO
**Default value**: 1
**Description**: The upper limit for the number of task threads that the Kafka connector can create, which is usually the same as the number of CPU cores on the worker nodes in the Kafka Connect cluster. You can tune this parameter to control load performance. ##### bufferflush.maxbytes[​](#bufferflushmaxbytes "Direct link to bufferflush.maxbytes") **Required**: NO
**Default value**: 94371840(90M)
**Description**: The maximum size of data that can be accumulated in memory before being sent to StarRocks at a time. The maximum value ranges from 64 MB to 10 GB. Keep in mind that the Stream Load SDK buffer may create multiple Stream Load jobs to buffer data. Therefore, the threshold mentioned here refers to the total data size. ##### bufferflush.intervalms[​](#bufferflushintervalms "Direct link to bufferflush.intervalms") **Required**: NO
**Default value**: 1000
**Description**: Interval for sending a batch of data which controls the load latency. Range: \[1000, 3600000]. ##### connect.timeoutms[​](#connecttimeoutms "Direct link to connect.timeoutms") **Required**: NO
**Default value**: 1000
**Description**: Timeout for connecting to the HTTP URL. Range: \[100, 60000]. ##### sink.properties.\*[​](#sinkproperties "Direct link to sink.properties.*") **Required**:
**Default value**:
**Description**: Stream Load parameters o control load behavior. For example, the parameter `sink.properties.format` specifies the format used for Stream Load, such as CSV or JSON. For a list of supported parameters and their descriptions, see [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). ##### sink.properties.format[​](#sinkpropertiesformat "Direct link to sink.properties.format") **Required**: NO
**Default value**: json
**Description**: The format used for Stream Load. The Kafka connector will transform each batch of data to the format before sending them to StarRocks. Valid values: `csv` and `json`. For more information, see [CSV parameters](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md#csv-parameters) and [JSON parameters](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md#json-parameters). ##### sink.properties.partial\_update[​](#sinkpropertiespartial_update "Direct link to sink.properties.partial_update") **Required**: NO
**Default value**: `FALSE`
**Description**: Whether to use partial updates. Valid values: `TRUE` and `FALSE`. Default value: `FALSE`, indicating to disable this feature. ##### sink.properties.partial\_update\_mode[​](#sinkpropertiespartial_update_mode "Direct link to sink.properties.partial_update_mode") **Required**: NO
**Default value**: `row`
**Description**: Specifies the mode for partial updates. Valid values: `row` and `column`. * The value `row` (default) means partial updates in row mode, which is more suitable for real-time updates with many columns and small batches. * The value `column` means partial updates in column mode, which is more suitable for batch updates with few columns and many rows. In such scenarios, enabling the column mode offers faster update speeds. For example, in a table with 100 columns, if only 10 columns (10% of the total) are updated for all rows, the update speed of the column mode is 10 times faster. #### Usage Notes[​](#usage-notes "Direct link to Usage Notes") ##### Flush Policy[​](#flush-policy "Direct link to Flush Policy") The Kafka connector will buffer the data in memory, and flush them in batch to StarRocks via Stream Load. The flush will be triggered when any of the following conditions are met: * The bytes of buffered rows reaches the limit `bufferflush.maxbytes`. * The elapsed time since the last flush reaches the limit `bufferflush.intervalms`. * The interval at which the connector tries committing offsets for tasks is reached. The interval is controlled by the Kafka Connect configuration [`offset.flush.interval.ms`](https://docs.confluent.io/platform/current/connect/references/allconfigs.html), and the default values is `60000`. For lower data latency, adjust these configurations in the Kafka connector settings. However, more frequent flushes will increase CPU and I/O usage. ##### Limits[​](#limits "Direct link to Limits") * It is not supported to flatten a single message from a Kafka topic into multiple data rows and load into StarRocks. * The sink of the Kafka connector provided by StarRocks guarantees at-least-once semantics. #### Best practices[​](#best-practices "Direct link to Best practices") ##### Load Debezium-formatted CDC data[​](#load-debezium-formatted-cdc-data "Direct link to Load Debezium-formatted CDC data") Debezium is a popular Change Data Capture (CDC) tool that supports monitoring data changes in various database systems and streaming these changes to Kafka. The following example demonstrates how to configure and use the Kafka connector to write PostgreSQL changes to a **Primary Key table** in StarRocks. ###### Step 1: Install and start Kafka[​](#step-1-install-and-start-kafka "Direct link to Step 1: Install and start Kafka") > **NOTE** > > You can skip this step if you have your own Kafka environment. 1. [Download](https://dlcdn.apache.org/kafka/) the latest Kafka release from the official site and extract the package. ```bash tar -xzf kafka_2.13-3.7.0.tgz cd kafka_2.13-3.7.0 ``` 2. Start the Kafka environment. Generate a Kafka cluster UUID. ```bash KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)" ``` Format the log directories. ```bash bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/server.properties ``` Start the Kafka server. ```bash bin/kafka-server-start.sh config/kraft/server.properties ``` ###### Step 2: Configure PostgreSQL[​](#step-2-configure-postgresql "Direct link to Step 2: Configure PostgreSQL") 1. Make sure the PostgreSQL user is granted `REPLICATION` privileges. 2. Adjust PostgreSQL configuration. Set `wal_level` to `logical` in **postgresql.conf**. ```properties wal_level = logical ``` Restart the PostgreSQL server to apply changes. ```bash pg_ctl restart ``` 3. Prepare the dataset. Create a table and insert test data. ```sql CREATE TABLE customers ( id int primary key , first_name varchar(65533) NULL, last_name varchar(65533) NULL , email varchar(65533) NULL ); INSERT INTO customers VALUES (1,'a','a','a@a.com'); ``` 4. Verify the CDC log messages in Kafka. ```json { "schema": { "type": "struct", "fields": [ { "type": "struct", "fields": [ { "type": "int32", "optional": false, "field": "id" }, { "type": "string", "optional": true, "field": "first_name" }, { "type": "string", "optional": true, "field": "last_name" }, { "type": "string", "optional": true, "field": "email" } ], "optional": true, "name": "test.public.customers.Value", "field": "before" }, { "type": "struct", "fields": [ { "type": "int32", "optional": false, "field": "id" }, { "type": "string", "optional": true, "field": "first_name" }, { "type": "string", "optional": true, "field": "last_name" }, { "type": "string", "optional": true, "field": "email" } ], "optional": true, "name": "test.public.customers.Value", "field": "after" }, { "type": "struct", "fields": [ { "type": "string", "optional": false, "field": "version" }, { "type": "string", "optional": false, "field": "connector" }, { "type": "string", "optional": false, "field": "name" }, { "type": "int64", "optional": false, "field": "ts_ms" }, { "type": "string", "optional": true, "name": "io.debezium.data.Enum", "version": 1, "parameters": { "allowed": "true,last,false,incremental" }, "default": "false", "field": "snapshot" }, { "type": "string", "optional": false, "field": "db" }, { "type": "string", "optional": true, "field": "sequence" }, { "type": "string", "optional": false, "field": "schema" }, { "type": "string", "optional": false, "field": "table" }, { "type": "int64", "optional": true, "field": "txId" }, { "type": "int64", "optional": true, "field": "lsn" }, { "type": "int64", "optional": true, "field": "xmin" } ], "optional": false, "name": "io.debezium.connector.postgresql.Source", "field": "source" }, { "type": "string", "optional": false, "field": "op" }, { "type": "int64", "optional": true, "field": "ts_ms" }, { "type": "struct", "fields": [ { "type": "string", "optional": false, "field": "id" }, { "type": "int64", "optional": false, "field": "total_order" }, { "type": "int64", "optional": false, "field": "data_collection_order" } ], "optional": true, "name": "event.block", "version": 1, "field": "transaction" } ], "optional": false, "name": "test.public.customers.Envelope", "version": 1 }, "payload": { "before": null, "after": { "id": 1, "first_name": "a", "last_name": "a", "email": "a@a.com" }, "source": { "version": "2.5.3.Final", "connector": "postgresql", "name": "test", "ts_ms": 1714283798721, "snapshot": "false", "db": "postgres", "sequence": "[\"22910216\",\"22910504\"]", "schema": "public", "table": "customers", "txId": 756, "lsn": 22910504, "xmin": null }, "op": "c", "ts_ms": 1714283798790, "transaction": null } } ``` ###### Step 3: Configure StarRocks[​](#step-3-configure-starrocks "Direct link to Step 3: Configure StarRocks") Create a Primary Key table in StarRocks with the same schema as the source table in PostgreSQL. ```sql CREATE TABLE `customers` ( `id` int(11) COMMENT "", `first_name` varchar(65533) NULL COMMENT "", `last_name` varchar(65533) NULL COMMENT "", `email` varchar(65533) NULL COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`id`) DISTRIBUTED BY hash(id) buckets 1 PROPERTIES ( "bucket_size" = "4294967296", "in_memory" = "false", "enable_persistent_index" = "true", "replicated_storage" = "true", "fast_schema_evolution" = "true" ); ``` ###### Step 4: Install connector[​](#step-4-install-connector "Direct link to Step 4: Install connector") 1. Download the connectors and extract the packages in the **plugins** directory. ```bash mkdir plugins tar -zxvf debezium-debezium-connector-postgresql-2.5.3.zip -C plugins mv starrocks-connector-for-kafka-x.y.z-with-dependencies.jar plugins ``` This directory is the value of the configuration item `plugin.path` in **config/connect-standalone.properties**. ```properties plugin.path=/path/to/kafka_2.13-3.7.0/plugins ``` 2. Configure the PostgreSQL source connector in **pg-source.properties**. ```json { "name": "inventory-connector", "config": { "connector.class": "io.debezium.connector.postgresql.PostgresConnector", "plugin.name": "pgoutput", "database.hostname": "localhost", "database.port": "5432", "database.user": "postgres", "database.password": "", "database.dbname" : "postgres", "topic.prefix": "test" } } ``` 3. Configure the StarRocks sink connector in **sr-sink.properties**. ```json { "name": "starrocks-kafka-connector", "config": { "connector.class": "com.starrocks.connector.kafka.StarRocksSinkConnector", "tasks.max": "1", "topics": "test.public.customers", "starrocks.http.url": "172.26.195.69:28030", "starrocks.database.name": "test", "starrocks.username": "root", "starrocks.password": "StarRocks@123", "sink.properties.strip_outer_array": "true", "connect.timeoutms": "3000", "starrocks.topic2table.map": "test.public.customers:customers", "transforms": "addfield,unwrap", "transforms.addfield.type": "com.starrocks.connector.kafka.transforms.AddOpFieldForDebeziumRecord", "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState", "transforms.unwrap.drop.tombstones": "true", "transforms.unwrap.delete.handling.mode": "rewrite" } } ``` > **NOTE** > > * If the StarRocks table is not a Primary Key table, you do not need to specify the `addfield` transform. > * The unwrap transform is provided by Debezium and is used to unwrap Debezium's complex data structure based on the operation type. For more information, see [New Record State Extraction](https://debezium.io/documentation/reference/stable/transformations/event-flattening.html). 4. Configure Kafka Connect. Configure the following configuration items in the Kafka Connect configuration file **config/connect-standalone.properties**. ```properties # The addresses of Kafka brokers. Multiple addresses of Kafka brokers need to be separated by commas (,). # Note that this example uses PLAINTEXT as the security protocol to access the Kafka cluster. # If you use other security protocol to access the Kafka cluster, configure the relevant information in this part. bootstrap.servers=:9092 offset.storage.file.filename=/tmp/connect.offsets key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter key.converter.schemas.enable=true value.converter.schemas.enable=false # The absolute path of starrocks-connector-for-kafka-x.y.z-with-dependencies.jar. plugin.path=/home/kafka-connect/starrocks-kafka-connector # Parameters that control the flush policy. For more information, see the Usage Note section. offset.flush.interval.ms=10000 bufferflush.maxbytes = xxx bufferflush.intervalms = xxx ``` For descriptions of more parameters, see [Running Kafka Connect](https://kafka.apache.org/documentation.html#connect_running). ###### Step 5: Start Kafka Connect in Standalone Mode[​](#step-5-start-kafka-connect-in-standalone-mode "Direct link to Step 5: Start Kafka Connect in Standalone Mode") Run Kafka Connect in standalone mode to initiate the connectors. ```bash bin/connect-standalone.sh config/connect-standalone.properties config/pg-source.properties config/sr-sink.properties ``` ###### Step 6: Verify data ingestion[​](#step-6-verify-data-ingestion "Direct link to Step 6: Verify data ingestion") Test the following operations and ensure the data is correctly ingested into StarRocks. ###### INSERT[​](#insert "Direct link to INSERT") * In PostgreSQL: ```plain postgres=# insert into customers values (2,'b','b','b@b.com'); INSERT 0 1 postgres=# select * from customers; id | first_name | last_name | email ----+------------+-----------+--------- 1 | a | a | a@a.com 2 | b | b | b@b.com (2 rows) ``` * In StarRocks: ```plain MySQL [test]> select * from customers; +------+------------+-----------+---------+ | id | first_name | last_name | email | +------+------------+-----------+---------+ | 1 | a | a | a@a.com | | 2 | b | b | b@b.com | +------+------------+-----------+---------+ 2 rows in set (0.01 sec) ``` ###### UPDATE[​](#update "Direct link to UPDATE") * In PostgreSQL: ```plain postgres=# update customers set email='c@c.com'; UPDATE 2 postgres=# select * from customers; id | first_name | last_name | email ----+------------+-----------+--------- 1 | a | a | c@c.com 2 | b | b | c@c.com (2 rows) ``` * In StarRocks: ```plain MySQL [test]> select * from customers; +------+------------+-----------+---------+ | id | first_name | last_name | email | +------+------------+-----------+---------+ | 1 | a | a | c@c.com | | 2 | b | b | c@c.com | +------+------------+-----------+---------+ 2 rows in set (0.00 sec) ``` ###### DELETE[​](#delete "Direct link to DELETE") * In PostgreSQL: ```plain postgres=# delete from customers where id=1; DELETE 1 postgres=# select * from customers; id | first_name | last_name | email ----+------------+-----------+--------- 2 | b | b | c@c.com (1 row) ``` * In StarRocks: ```plain MySQL [test]> select * from customers; +------+------------+-----------+---------+ | id | first_name | last_name | email | +------+------------+-----------+---------+ | 2 | b | b | c@c.com | +------+------------+-----------+---------+ 1 row in set (0.00 sec) ``` --- ### Routine Load ### Load data using Routine Load tip Try Routine Load out in this [Quick Start](https://docs.starrocks.io/docs/quick_start/routine-load.md) This topic introduces how to create a Routine Load job to stream Kafka messages (events) into StarRocks, and familiarizes you with some basic concepts about Routine Load. To continuously load messages of a stream into StarRocks, you can store the message stream in a Kafka topic, and create a Routine Load job to consume the messages. The Routine Load job persists in StarRocks, generates a series of load tasks to consume the messages in all or part of the partitions in the topic, and loads the messages into StarRocks. A Routine Load job supports exactly-once delivery semantics to guarantee the data loaded into StarRocks is neither lost nor duplicated. Routine Load supports data transformation at data loading and supports data changes made by UPSERT and DELETE operations during data loading. For more information, see [Transform data at loading](https://docs.starrocks.io/docs/loading/Etl_in_loading.md) and [Change data through loading](https://docs.starrocks.io/docs/loading/Load_to_Primary_Key_tables.md). You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. #### Supported data formats[​](#supported-data-formats "Direct link to Supported data formats") Routine Load now supports consuming CSV, JSON, and Avro (supported since v3.0.1) formatted data from a Kafka cluster. > **NOTE** > > For CSV data, take note of the following points: > > * You can use a UTF-8 string, such as a comma (,), tab, or pipe (|), whose length does not exceed 50 bytes as a text delimiter. > * Null values are denoted by using `\N`. For example, a data file consists of three columns, and a record from that data file holds data in the first and third columns but no data in the second column. In this situation, you need to use `\N` in the second column to denote a null value. This means the record must be compiled as `a,\N,b` instead of `a,,b`. `a,,b` denotes that the second column of the record holds an empty string. #### Basic concepts[​](#basic-concepts "Direct link to Basic concepts") ![routine load](/assets/images/4.5.2-1-a6d6d07699a1f32c703d4b80921e2e44.png) ##### Terminology[​](#terminology "Direct link to Terminology") * **Load job** A Routine Load job is a long-running job. As long as its status is RUNNING, a load job continuously generates one or multiple concurrent load tasks which consume the messages in a topic of a Kafka cluster and load the data into StarRocks. * **Load task** A load job is split into multiple load tasks by certain rules. A load task is the basic unit of data loading. As an individual event, a load task implements the load mechanism based on [Stream Load](https://docs.starrocks.io/docs/loading/StreamLoad.md). Multiple load tasks concurrently consume the messages from different partitions of a topic, and load the data into StarRocks. ##### Workflow[​](#workflow "Direct link to Workflow") 1. **Create a Routine Load job.** To load data from Kafka, you need to create a Routine Load job by running the [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md) statement. The FE parses the statement, and creates the job according to the properties you have specified. 2. **The FE splits the job into multiple load tasks.** The FE split the job into multiple load tasks based on certain rules. Each load task is an individual transaction. The splitting rules are as follows: * The FE calculates the actual concurrent number of the load tasks according to the desired concurrent number `desired_concurrent_number`, the partition number in the Kafka topic, and the number of the BE nodes that are alive. * The FE splits the job into load tasks based on the actual concurrent number calculated, and arranges the tasks in the task queue. Each Kafka topic consists of multiple partitions. The relation between the topic partition and the load task is as follows: * A partition is uniquely assigned to a load task, and all messages from the partition are consumed by the load task. * A load task can consume messages from one or more partitions. * All partitions are distributed evenly among load tasks. 3. **Multiple load tasks run concurrently to consume the messages from multiple Kafka topic partitions, and load the data into StarRocks** 1. **The FE schedules and submits load tasks**: the FE schedules the load tasks in the queue on a timely basis, and assigns them to selected Coordinator BE nodes. The interval between load tasks is defined by the configuration item `max_batch_interval`. The FE distributes the load tasks evenly to all BE nodes. See [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md#examples) for more information about `max_batch_interval`. 2. The Coordinator BE starts the load task, consumes messages in partitions, parses and filters the data. A load task lasts until the pre-defined amount of messages are consumed or the pre-defined time limit is reached. The message batch size and time limit are defined in the FE configurations `max_routine_load_batch_size` and `routine_load_task_consume_second`. For detailed information, see [FE Configuration](https://docs.starrocks.io/docs/administration/management/FE_configuration.md). The Coordinator BE then distributes the messages to the Executor BEs. The Executor BEs write the messages to disks. > **NOTE** > > StarRocks supports access to Kafka via security protocols including SASL\_SSL, SAS\_PLAINTEXT, SSL, and PLAINTEXT. This topic uses connecting to Kafka via PLAINTEXT as an example. If you need to connect to Kafka via other security protocols, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). 4. **The FE generates new load tasks to load data continuously.** After the Executor BEs has written the data to disks, the Coordinator BE reports the result of the load task to the FE. Based on the result, the FE then generates new load tasks to load the data continuously. Or the FE retries the failed tasks to make sure the data loaded into StarRocks is neither lost nor duplicated. #### Create a Routine Load job[​](#create-a-routine-load-job "Direct link to Create a Routine Load job") The following three examples describe how to consume CSV-format, JSON-format and Avro-format data in Kafka, and load the data into StarRocks by creating a Routine Load job. For detailed syntax and parameter descriptions, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). ##### Load CSV-format data[​](#load-csv-format-data "Direct link to Load CSV-format data") This section describes how to create a Routine Load job to consume CSV-format data in a Kafka cluster, and load the data into StarRocks. ###### Prepare a dataset[​](#prepare-a-dataset "Direct link to Prepare a dataset") Suppose there is a CSV-format dataset in the topic `ordertest1` in a Kafka cluster. Every message in the dataset includes six fields: order ID, payment date, customer name, nationality, gender, and price. ```plain 2020050802,2020-05-08,Johann Georg Faust,Deutschland,male,895 2020050802,2020-05-08,Julien Sorel,France,male,893 2020050803,2020-05-08,Dorian Grey,UK,male,1262 2020050901,2020-05-09,Anna Karenina",Russia,female,175 2020051001,2020-05-10,Tess Durbeyfield,US,female,986 2020051101,2020-05-11,Edogawa Conan,japan,male,8924 ``` ###### Create a table[​](#create-a-table "Direct link to Create a table") According to the fields of CSV-format data, create the table `example_tbl1` in the database `example_db`. The following example creates a table with 5 fields excluding the field of customer gender in the CSV-format data. ```sql CREATE TABLE example_db.example_tbl1 ( `order_id` bigint NOT NULL COMMENT "Order ID", `pay_dt` date NOT NULL COMMENT "Payment date", `customer_name` varchar(26) NULL COMMENT "Customer name", `nationality` varchar(26) NULL COMMENT "Nationality", `price`double NULL COMMENT "Price" ) ENGINE=OLAP DUPLICATE KEY (order_id,pay_dt) DISTRIBUTED BY HASH(`order_id`); ``` > **NOTICE** > > Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). ###### Submit a Routine Load job[​](#submit-a-routine-load-job "Direct link to Submit a Routine Load job") Execute the following statement to submit a Routine Load job named `example_tbl1_ordertest1` to consume the messages in the topic `ordertest1` and load the data into the table `example_tbl1`. The load task consumes the messages from the initial offset in the specified partitions of the topic. ```sql CREATE ROUTINE LOAD example_db.example_tbl1_ordertest1 ON example_tbl1 COLUMNS TERMINATED BY ",", COLUMNS (order_id, pay_dt, customer_name, nationality, temp_gender, price) PROPERTIES ( "desired_concurrent_number" = "5" ) FROM KAFKA ( "kafka_broker_list" = ":,:", "kafka_topic" = "ordertest1", "kafka_partitions" = "0,1,2,3,4", "property.kafka_default_offsets" = "OFFSET_BEGINNING" ); ``` After submitting the load job, you can execute the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement to check the status of the load job. * **load job name** There could be multiple load job on a table. Therefore, we recommend you name a load job with the corresponding Kafka topic and the time when the load job is submitted. It helps you distinguish the load job on each table. * **Column separator** The property `COLUMN TERMINATED BY` defines the column separator of the CSV-format data. The default is `\t`. * **Kafka topic partition and offset** You can specify the properties `kafka_partitions` and `kafka_offsets` to specify the partitions and offsets to consume the messages. For example, if you want the load job to consume messages from the Kafka partitions `"0,1,2,3,4"` of the topic `ordertest1` all with the initial offsets, you can specify the properties as follows: If you want the load job to consume messages from the Kafka partitions `"0,1,2,3,4"`and you need to specify a separate starting offset for each partition, you can configure as follows: ```sql "kafka_partitions" ="0,1,2,3,4", "kafka_offsets" = "OFFSET_BEGINNING, OFFSET_END, 1000, 2000, 3000" ``` You can also set the default offsets of all partitions with the property `property.kafka_default_offsets`. ```sql "kafka_partitions" ="0,1,2,3,4", "property.kafka_default_offsets" = "OFFSET_BEGINNING" ``` For detailed information, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). * **Data mapping and transformation** To specify the mapping and transformation relationship between the CSV-format data, and the StarRocks table, you need to use the `COLUMNS` parameter. **Data mapping:** * StarRocks extracts the columns in the CSV-format data and maps them **in sequence** onto the fields declared in the `COLUMNS` parameter. * StarRocks extracts the fields declared in the `COLUMNS` parameter and maps them **by name** onto the columns of StarRocks table. **Data transformation:** And because the example excludes the column of customer gender from the CSV-format data, the field `temp_gender` in `COLUMNS` parameter is used as a placeholder for this field. The other fields are mapped to columns of the StarRocks table `example_tbl1` directly. For more information about data transformation, see [Transform data at loading](https://docs.starrocks.io/docs/loading/Etl_in_loading.md). > **NOTE** > > You do not need to specify the `COLUMNS` parameter if the names, number, and order of the columns in the CSV-format data completely correspond to those of the StarRocks table. * **Task concurrency** When there are many Kafka topic partitions and enough BE nodes, you can accelerate the loading by increasing the task concurrency. To increase the actual load task concurrency, you can increase the desired load task concurrency `desired_concurrent_number` when you create a routine load job. You can also set the dynamic configuration item of FE `max_routine_load_task_concurrent_num` ( default maximum load task currency ) to a larger value. For more information about `max_routine_load_task_concurrent_num`, please see [FE configuration items](https://docs.starrocks.io/docs/administration/management/FE_configuration.md). The actual task concurrency is defined by the minimum value among the number of BE nodes that are alive, the number of the pre-specified Kafka topic partitions, and the values of `desired_concurrent_number` and `max_routine_load_task_concurrent_num`. In the example, the number of BE nodes that are alive is `5`, the number of the pre-specified Kafka topic partitions is `5`, and the value of `max_routine_load_task_concurrent_num` is `5`. To increase the actual load task concurrency, you can increase the `desired_concurrent_number` from the default value `3` to `5`. For more about the properties, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). ##### Load JSON-format data[​](#load-json-format-data "Direct link to Load JSON-format data") This section describes how to create a Routine Load job to consume JSON-format data in a Kafka cluster, and load the data into StarRocks. ###### Prepare a dataset[​](#prepare-a-dataset-1 "Direct link to Prepare a dataset") Suppose there is a JSON-format dataset in the topic `ordertest2` in a Kafka cluster. The dataset includes six keys: commodity ID, customer name, nationality, payment time, and price. Besides, you want to transform the payment time column into the DATE type, and load it into the `pay_dt` column in the StarRocks table. ```json {"commodity_id": "1", "customer_name": "Mark Twain", "country": "US","pay_time": 1589191487,"price": 875} {"commodity_id": "2", "customer_name": "Oscar Wilde", "country": "UK","pay_time": 1589191487,"price": 895} {"commodity_id": "3", "customer_name": "Antoine de Saint-Exupéry","country": "France","pay_time": 1589191487,"price": 895} ``` > **CAUTION** Each JSON object in a row must be in one Kafka message, otherwise a JSON parsing error is returned. ###### Create a table[​](#create-a-table-1 "Direct link to Create a table") According to the keys of the JSON-format data, create the table `example_tbl2` in the database `example_db`. ```sql CREATE TABLE `example_tbl2` ( `commodity_id` varchar(26) NULL COMMENT "Commodity ID", `customer_name` varchar(26) NULL COMMENT "Customer name", `country` varchar(26) NULL COMMENT "Country", `pay_time` bigint(20) NULL COMMENT "Payment time", `pay_dt` date NULL COMMENT "Payment date", `price`double SUM NULL COMMENT "Price" ) ENGINE=OLAP AGGREGATE KEY(`commodity_id`,`customer_name`,`country`,`pay_time`,`pay_dt`) DISTRIBUTED BY HASH(`commodity_id`); ``` > **NOTICE** > > Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). ###### Submit a Routine Load job[​](#submit-a-routine-load-job-1 "Direct link to Submit a Routine Load job") Execute the following statement to submit a Routine Load job named `example_tbl2_ordertest2` to consume the messages in the topic `ordertest2` and load the data into the table `example_tbl2`. The load task consumes the messages from the initial offset in the specified partitions of the topic. ```sql CREATE ROUTINE LOAD example_db.example_tbl2_ordertest2 ON example_tbl2 COLUMNS(commodity_id, customer_name, country, pay_time, price, pay_dt=from_unixtime(pay_time, '%Y%m%d')) PROPERTIES ( "desired_concurrent_number" = "5", "format" = "json", "jsonpaths" = "[\"$.commodity_id\",\"$.customer_name\",\"$.country\",\"$.pay_time\",\"$.price\"]" ) FROM KAFKA ( "kafka_broker_list" =":,:", "kafka_topic" = "ordertest2", "kafka_partitions" ="0,1,2,3,4", "property.kafka_default_offsets" = "OFFSET_BEGINNING" ); ``` After submitting the load job, you can execute the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement to check the status of the load job. * **Data format** You need to specify `"format" = "json"` in the clause `PROPERTIES` to define that the data format is JSON. * **Data mapping and transformation** To specify the mapping and transformation relationship between the JSON-format data, and the StarRocks table, you need to specify the parameter `COLUMNS` and property`jsonpaths`. The order of fields specified in the `COLUMNS` parameter must match that of the JSON-format data, and the name of fields must match that of the StarRocks table. The property `jsonpaths` is used to extract the required fields from the JSON data. These fields are then named by the property `COLUMNS`. Because the example needs to transform the payment time field to the DATE data type, and load the data into the `pay_dt` column in the StarRocks table, you need to use the from\_unixtime function. The other fields are mapped to fields of the table `example_tbl2` directly. **Data mapping:** * StarRocks extracts the `name` and `code` keys of JSON-format data and maps them onto the keys declared in the `jsonpaths` property. * StarRocks extracts the keys declared in the `jsonpaths` property and maps them **in sequence** onto the fields declared in the `COLUMNS` parameter. * StarRocks extracts the fields declared in the `COLUMNS` parameter and maps them **by name** onto the columns of StarRocks table. **Data transformation**: * Because the example needs to transform the key `pay_time` to the DATE data type, and load the data into the `pay_dt` column in the StarRocks table, you need to use the from\_unixtime function in `COLUMNS` parameter. The other fields are mapped to fields of the table `example_tbl2` directly. * And because the example excludes the column of customer gender from the JSON-format data, the field `temp_gender` in `COLUMNS` parameter is used as a placeholder for this field. The other fields are mapped to columns of the StarRocks table `example_tbl1` directly. For more information about data transformation, see [Transform data at loading](https://docs.starrocks.io/docs/loading/Etl_in_loading.md). > **NOTE** > > You do not need to specify the `COLUMNS` parameter if the names and number of the keys in the JSON object completely match those of fields in the StarRocks table. ##### Load Avro-format data[​](#load-avro-format-data "Direct link to Load Avro-format data") Since v3.0.1, StarRocks supports loading Avro data by using Routine Load. ###### Prepare a dataset[​](#prepare-a-dataset-2 "Direct link to Prepare a dataset") ###### Avro schema[​](#avro-schema "Direct link to Avro schema") 1. Create the following Avro schema file `avro_schema.avsc`: ```json { "type": "record", "name": "sensor_log", "fields" : [ {"name": "id", "type": "long"}, {"name": "name", "type": "string"}, {"name": "checked", "type" : "boolean"}, {"name": "data", "type": "double"}, {"name": "sensor_type", "type": {"type": "enum", "name": "sensor_type_enum", "symbols" : ["TEMPERATURE", "HUMIDITY", "AIR-PRESSURE"]}} ] } ``` 2. Register the Avro schema in the [Schema Registry](https://docs.confluent.io/cloud/current/get-started/schema-registry.html#create-a-schema). ###### Avro data[​](#avro-data "Direct link to Avro data") Prepare the Avro data and send it to the Kafka topic `topic_0`. ###### Create a table[​](#create-a-table-2 "Direct link to Create a table") According to the fields of Avro data, create a table `sensor_log` in the target database `example_db` in the StarRocks cluster. The column names of the table must match the field names in the Avro data. For the data type mapping between the table columns and the Avro data fields, see \[Data types mapping]\(#Data types mapping). ```sql CREATE TABLE example_db.sensor_log ( `id` bigint NOT NULL COMMENT "sensor id", `name` varchar(26) NOT NULL COMMENT "sensor name", `checked` boolean NOT NULL COMMENT "checked", `data` double NULL COMMENT "sensor data", `sensor_type` varchar(26) NOT NULL COMMENT "sensor type" ) ENGINE=OLAP DUPLICATE KEY (id) DISTRIBUTED BY HASH(`id`); ``` > **NOTICE** > > Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). ###### Submit a Routine Load job[​](#submit-a-routine-load-job-2 "Direct link to Submit a Routine Load job") Execute the following statement to submit a Routine Load job named `sensor_log_load_job` to consume the Avro messages in the Kafka topic `topic_0` and load the data into the table `sensor_log` in the database `sensor`. The load job consumes the messages from the initial offset in the specified partitions of the topic. ```sql CREATE ROUTINE LOAD example_db.sensor_log_load_job ON sensor_log PROPERTIES ( "format" = "avro" ) FROM KAFKA ( "kafka_broker_list" = ":,:,...", "confluent.schema.registry.url" = "http://172.xx.xxx.xxx:8081", "kafka_topic" = "topic_0", "kafka_partitions" = "0,1,2,3,4,5", "property.kafka_default_offsets" = "OFFSET_BEGINNING" ); ``` * Data Format You need to specify `"format = "avro"` in the clause `PROPERTIES` to define that the data format is Avro. * Schema Registry You need to configure `confluent.schema.registry.url` to specify the URL of the Schema Registry where the Avro schema is registered. StarRocks retrieves the Avro schema by using this URL. The format is as follows: ```plaintext confluent.schema.registry.url = http[s]://[:@][:] ``` * Data mapping and transformation To specify the mapping and transformation relationship between the Avro-format data and the StarRocks table, you need to specify the parameter `COLUMNS` and property `jsonpaths`. The order of fields specified in the `COLUMNS` parameter must match that of the fields in the property `jsonpaths`, and the names of fields must match these of the StarRocks table. The property `jsonpaths` is used to extract the required fields from the Avro data. These fields are then named by the property `COLUMNS`. For more information about data transformation, see [Transform data at loading](https://docs.starrocks.io/docs/loading/Etl_in_loading.md). > NOTE > > You do not need to specify the `COLUMNS` parameter if the names and number of the fields in the Avro record completely match those of columns in the StarRocks table. After submitting the load job, you can execute the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement to check the status of the load job. ###### Data types mapping[​](#data-types-mapping "Direct link to Data types mapping") The data type mapping between the Avro data fields you want to load and the StarRocks table columns is as follows: ###### Primitive types[​](#primitive-types "Direct link to Primitive types") | Avro | StarRocks | | ------- | --------- | | nul | NULL | | boolean | BOOLEAN | | int | INT | | long | BIGINT | | float | FLOAT | | double | DOUBLE | | bytes | STRING | | string | STRING | ###### Complex types[​](#complex-types "Direct link to Complex types") | Avro | StarRocks | | -------------- | -------------------------------------------------------------------------- | | record | STRUCT, or load the entire RECORD or its subfields into StarRocks as JSON. | | enums | STRING | | arrays | ARRAY | | maps | MAP or JSON | | union(T, null) | NULLABLE(T) | | fixed | STRING | ###### Limits[​](#limits "Direct link to Limits") * Currently, StarRocks does not support schema evolution. * Each Kafka message must only contain a single Avro data record. ##### Access source message metadata[​](#access-source-message-metadata "Direct link to Access source message metadata") When loading JSON- or Avro-format data, you can populate destination columns from a message's Kafka/Pulsar metadata — topic, partition, offset, timestamp, key, and headers — instead of from the message payload. You declare the metadata in an `INCLUDE METADATA (...)` clause that binds each metadata key to an alias; the alias is an ordinary source column you reference from `COLUMNS`. This is useful for auditing (which topic/partition/offset a row came from), event-time processing (using the message timestamp), and routing on a header value. ###### Syntax[​](#syntax "Direct link to Syntax") ```sql INCLUDE METADATA ( [AS ] [, [AS ] ...] ) ``` `INCLUDE METADATA` is a load property; place it among the other load properties (such as `COLUMNS` and `WHERE`), in any order, before the `PROPERTIES` and `FROM` clauses. `AS ` is optional. If it is omitted, the alias defaults to ``. The alias must be unique within the clause, and it must not collide with a payload field, a destination-table column, or a reserved column name. ###### Metadata keys[​](#metadata-keys "Direct link to Metadata keys") The supported keys depend on the data source. | Source | Key | Type | Description | | ------ | ----------------- | ---------------------- | ------------------------------------------------------------------------------------------------- | | KAFKA | `TOPIC` | VARCHAR | Topic name. | | KAFKA | `PARTITION` | INT | Partition number. | | KAFKA | `OFFSET` | BIGINT | Message offset within the partition. | | KAFKA | `TIMESTAMP_MS` | BIGINT | Record timestamp in milliseconds since epoch. `NULL` when the broker reports no timestamp. | | KAFKA | `KEY` | VARCHAR | Message key as raw bytes. `NULL` when the message has no key. | | KAFKA | `HEADERS` | MAP\ | All headers as a map. On duplicate keys, the last value wins. | | PULSAR | `TOPIC` | VARCHAR | The logical topic the job consumes (a partitioned topic's `-partition-N` suffix is not included). | | PULSAR | `PARTITION` | INT | Partition index, parsed from the per-message topic name. `NULL` for a non-partitioned topic. | | PULSAR | `KEY` | VARCHAR | Partition key. `NULL` when the message has no key. | | PULSAR | `MESSAGE_ID` | VARCHAR | Message ID. | | PULSAR | `PUBLISH_TIME_MS` | BIGINT | Publish time in milliseconds since epoch. | | PULSAR | `EVENT_TIME_MS` | BIGINT | Event time in milliseconds since epoch. `NULL` when the producer did not set it. | | PULSAR | `PROPERTIES` | MAP\ | All properties as a map. On duplicate keys, the last value wins. | To read a single header/property value, use `element_at(, '')` over the `HEADERS`/`PROPERTIES` map: the last value wins on a duplicate key, and the result is `NULL` when the key is absent. Header/property values are raw bytes placed into VARCHAR as-is (no UTF-8 validation). ###### Usage notes[​](#usage-notes "Direct link to Usage notes") * `INCLUDE METADATA` is available for `format = json` (Kafka and Pulsar) and `format = avro` (Kafka only; Pulsar Routine Load does not support Avro). It is not supported for CSV, where one message can expand into many rows and per-message metadata would be ambiguous. * A metadata alias is an ordinary source column: reference it anywhere in the `COLUMNS` expressions. If a payload field has the same name as a metadata key, specify a different alias with `AS ` to avoid ambiguity. * `OFFSET` is Kafka-only and `MESSAGE_ID` is Pulsar-only; using a key unsupported by the source raises an error that lists the keys supported for that source. * `HEADERS`/`PROPERTIES` is a `MAP\`. The source headers/properties are an ordered list that may repeat a key; duplicates collapse into the map with the last value winning (an `element_at(map, 'name')` lookup is likewise last-wins, and returns `NULL` when the key is absent). Values are raw bytes stored in VARCHAR as-is — there is no UTF-8 validation or decoding. ###### Example[​](#example "Direct link to Example") Load the order payload field `order_id` together with the source topic, partition, offset, the message timestamp converted to a `DATETIME`, and a `trace-id` header: ```sql CREATE TABLE example_db.orders_with_meta ( order_id BIGINT, src_topic VARCHAR(256), src_partition INT, src_offset BIGINT, msg_time DATETIME, trace_id VARCHAR(128) ) ENGINE = OLAP DUPLICATE KEY(order_id) DISTRIBUTED BY HASH(order_id); CREATE ROUTINE LOAD example_db.orders_with_meta_job ON orders_with_meta INCLUDE METADATA ( TOPIC AS m_topic, PARTITION AS m_partition, OFFSET AS m_offset, TIMESTAMP_MS AS m_timestamp, HEADERS AS m_headers ), COLUMNS ( order_id, src_topic = m_topic, src_partition = m_partition, src_offset = m_offset, msg_time = from_unixtime(m_timestamp / 1000), trace_id = element_at(m_headers, 'trace-id') ) PROPERTIES ( "format" = "json", "jsonpaths" = "[\"$.order_id\"]" ) FROM KAFKA ( "kafka_broker_list" = ":,...", "kafka_topic" = "topic_orders", "property.kafka_default_offsets" = "OFFSET_BEGINNING" ); ``` A metadata alias may be used inside an expression (as with `from_unixtime(m_timestamp / 1000)` above); only payload columns are listed in `jsonpaths`. #### Check a load job and task[​](#check-a-load-job-and-task "Direct link to Check a load job and task") ##### Check a load job[​](#check-a-load-job "Direct link to Check a load job") Execute the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement to check the status of the load job `example_tbl2_ordertest2`. StarRocks returns the execution state `State`, the statistical information (including the total rows consumed and the total rows loaded) `Statistics`, and the progress of the load job `progress`. If the state of the load job is automatically changed to **PAUSED**, it is possibly because the number of error rows has exceeded the threshold. For detailed instructions on setting this threshold, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). You can check the files `ReasonOfStateChanged` and `ErrorLogUrls` to identify and troubleshoot the problem. Having fixed the problem, you can then execute the [RESUME ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/RESUME_ROUTINE_LOAD.md) statement to resume the **PAUSED** load job. If the state of the load job is **CANCELLED**, it is possibly because the load job encounters an exception (such as the table has been dropped). You can check the files `ReasonOfStateChanged` and `ErrorLogUrls` to identify and troubleshoot the problem. However, you cannot resume a **CANCELLED** load job. ```sql MySQL [example_db]> SHOW ROUTINE LOAD FOR example_tbl2_ordertest2 \G *************************** 1. row *************************** Id: 63013 Name: example_tbl2_ordertest2 CreateTime: 2022-08-10 17:09:00 PauseTime: NULL EndTime: NULL DbName: default_cluster:example_db TableName: example_tbl2 State: RUNNING DataSourceType: KAFKA CurrentTaskNum: 3 JobProperties: {"partitions":"*","partial_update":"false","columnToColumnExpr":"commodity_id,customer_name,country,pay_time,pay_dt=from_unixtime(`pay_time`, '%Y%m%d'),price","maxBatchIntervalS":"20","whereExpr":"*","dataFormat":"json","timezone":"Asia/Shanghai","format":"json","json_root":"","strict_mode":"false","jsonpaths":"[\"$.commodity_id\",\"$.customer_name\",\"$.country\",\"$.pay_time\",\"$.price\"]","desireTaskConcurrentNum":"3","maxErrorNum":"0","strip_outer_array":"false","currentTaskConcurrentNum":"3","maxBatchRows":"200000"} DataSourceProperties: {"topic":"ordertest2","currentKafkaPartitions":"0,1,2,3,4","brokerList":":,:"} CustomProperties: {"kafka_default_offsets":"OFFSET_BEGINNING"} Statistic: {"receivedBytes":230,"errorRows":0,"committedTaskNum":1,"loadedRows":2,"loadRowsRate":0,"abortedTaskNum":0,"totalRows":2,"unselectedRows":0,"receivedBytesRate":0,"taskExecuteTimeMs":522} Progress: {"0":"1","1":"OFFSET_ZERO","2":"OFFSET_ZERO","3":"OFFSET_ZERO","4":"OFFSET_ZERO"} ReasonOfStateChanged: ErrorLogUrls: OtherMsg: ``` > **CAUTION** > > You cannot check a load job that has stopped or has not yet started. ##### Check a load task[​](#check-a-load-task "Direct link to Check a load task") Execute the [SHOW ROUTINE LOAD TASK](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD_TASK.md) statement to check the load tasks of the load job `example_tbl2_ordertest2`, such as how many tasks are currently running, the Kafka topic partitions that are consumed and the consumption progress `DataSourceProperties`, and the corresponding Coordinator BE node `BeId`. ```sql MySQL [example_db]> SHOW ROUTINE LOAD TASK WHERE JobName = "example_tbl2_ordertest2" \G *************************** 1. row *************************** TaskId: 18c3a823-d73e-4a64-b9cb-b9eced026753 TxnId: -1 TxnStatus: UNKNOWN JobId: 63013 CreateTime: 2022-08-10 17:09:05 LastScheduledTime: 2022-08-10 17:47:27 ExecuteStartTime: NULL Timeout: 60 BeId: -1 DataSourceProperties: {"1":0,"4":0} Message: there is no new data in kafka, wait for 20 seconds to schedule again *************************** 2. row *************************** TaskId: f76c97ac-26aa-4b41-8194-a8ba2063eb00 TxnId: -1 TxnStatus: UNKNOWN JobId: 63013 CreateTime: 2022-08-10 17:09:05 LastScheduledTime: 2022-08-10 17:47:26 ExecuteStartTime: NULL Timeout: 60 BeId: -1 DataSourceProperties: {"2":0} Message: there is no new data in kafka, wait for 20 seconds to schedule again *************************** 3. row *************************** TaskId: 1a327a34-99f4-4f8d-8014-3cd38db99ec6 TxnId: -1 TxnStatus: UNKNOWN JobId: 63013 CreateTime: 2022-08-10 17:09:26 LastScheduledTime: 2022-08-10 17:47:27 ExecuteStartTime: NULL Timeout: 60 BeId: -1 DataSourceProperties: {"0":2,"3":0} Message: there is no new data in kafka, wait for 20 seconds to schedule again ``` #### Pause a load job[​](#pause-a-load-job "Direct link to Pause a load job") You can execute the [PAUSE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/PAUSE_ROUTINE_LOAD.md) statement to pause a load job. The state of the load job will be **PAUSED** after the statement is executed. However, it has not stopped. You can execute the [RESUME ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/RESUME_ROUTINE_LOAD.md) statement to resume it. You can also check its status with the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement. The following example pauses the load job `example_tbl2_ordertest2`: ```sql PAUSE ROUTINE LOAD FOR example_tbl2_ordertest2; ``` #### Resume a load job[​](#resume-a-load-job "Direct link to Resume a load job") You can execute the [RESUME ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/RESUME_ROUTINE_LOAD.md) statement to resume a paused load job. The state of the load job will be **NEED\_SCHEDULE** temporarily (because the load job is being re-scheduled), and then become **RUNNING**. You can check its status with the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement. The following example resumes the paused load job `example_tbl2_ordertest2`: ```sql RESUME ROUTINE LOAD FOR example_tbl2_ordertest2; ``` #### Alter a load job[​](#alter-a-load-job "Direct link to Alter a load job") Before altering a load job, you must pause it with the [PAUSE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/PAUSE_ROUTINE_LOAD.md) statement. Then you can execute the [ALTER ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/ALTER_ROUTINE_LOAD.md). After altering it, you can execute the [RESUME ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/RESUME_ROUTINE_LOAD.md) statement to resume it, and check its status with the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement. Suppose the number of the BE nodes that are alive increases to `6` and the Kafka topic partitions to be consumed is `"0,1,2,3,4,5,6,7"`. If you want to increase the actual load task concurrency, you can execute the following statement to increase the number of desired task concurrency `desired_concurrent_number` to `6` (greater than or equal to the number of BE nodes that are alive), and specify the Kafka topic partitions and initial offsets. > **NOTE** > > Because the actual task concurrency is determined by the minimum value of multiple parameters, you must make sure that the value of the FE dynamic parameter `max_routine_load_task_concurrent_num` is greater than or equal to `6`. ```sql ALTER ROUTINE LOAD FOR example_tbl2_ordertest2 PROPERTIES ( "desired_concurrent_number" = "6" ) FROM kafka ( "kafka_partitions" = "0,1,2,3,4,5,6,7", "kafka_offsets" = "OFFSET_BEGINNING,OFFSET_BEGINNING,OFFSET_BEGINNING,OFFSET_BEGINNING,OFFSET_END,OFFSET_END,OFFSET_END,OFFSET_END" ); ``` #### Stop a load job[​](#stop-a-load-job "Direct link to Stop a load job") You can execute the [STOP ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/STOP_ROUTINE_LOAD.md) statement to stop a load job. The state of the load job will be **STOPPED** after the statement is executed, and you cannot resume a stopped load job. You cannot check the status of a stopped load job with the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement. The following example stops the load job `example_tbl2_ordertest2`: ```sql STOP ROUTINE LOAD FOR example_tbl2_ordertest2; ``` --- ### Stream with StarRocks Pipe #### Advantages of Pipe[​](#advantages-of-pipe "Direct link to Advantages of Pipe") Pipe is ideal for continuous data loading and large-scale data loading: * **Large-scale data loading in micro-batches helps reduce the cost of retries caused by data errors.** With the help of Pipe, StarRocks enables the efficient loading of a large number of data files with a significant data volume in total. Pipe automatically splits the files based on their number or size, breaking down the load job into smaller, sequential tasks. This approach ensures that errors in one file do not impact the entire load job. The load status of each file is recorded by Pipe, allowing you to easily identify and fix files that contain errors. By minimizing the need for retries due to data errors, this approach helps to reduce costs. * **Continuous data loading helps reduce manpower.** Pipe helps you write new or updated data files to a specific location and continuously load the new data from these files into StarRocks. After you create a Pipe job with `"AUTO_INGEST" = "TRUE"` specified, it will constantly monitor changes to the data files stored in the specified path and automatically load new or updated data from the data files into the destination StarRocks table. Additionally, Pipe performs file uniqueness checks to help prevent duplicate data loading.During the loading process, Pipe checks the uniqueness of each data file based on the file name and digest. If a file with a specific file name and digest has already been processed by a Pipe job, the Pipe job will skip all subsequent files with the same file name and digest. Note that object storage like AWS S3 uses ETag as file digest, and HDFS uses LastModifiedTime as file digest. The load status of each data file is recorded and saved to the `information_schema.pipe_files` view. After a Pipe job associated with the view is deleted, the records about the files loaded in that job will also be deleted. ##### Data flow[​](#data-flow "Direct link to Data flow") ![Pipe data flow](/assets/images/pipe_data_flow-2a4dc0b44a06c987d9afc0ecf632f5d9.png) #### [📄️ HDFS](https://docs.starrocks.io/docs/integrations/streaming/pipe/hdfs.md) [Pipe example using loading from HDFS](https://docs.starrocks.io/docs/integrations/streaming/pipe/hdfs.md) #### [📄️ S3](https://docs.starrocks.io/docs/integrations/streaming/pipe/s3.md) [Pipe example loading from S3](https://docs.starrocks.io/docs/integrations/streaming/pipe/s3.md) --- ### HDFS Load ### Load data from HDFS StarRocks provides the following options for loading data from HDFS: * Synchronous loading using [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md)+[`FILES()`](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md) * Asynchronous loading using [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) * Continuous asynchronous loading using [Pipe](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/CREATE_PIPE.md) Each of these options has its own advantages, which are detailed in the following sections. In most cases, we recommend that you use the INSERT+`FILES()` method, which is much easier to use. However, the INSERT+`FILES()` method currently supports only the Parquet, ORC, and CSV file formats. Therefore, if you need to load data of other file formats such as JSON, or perform data changes such as DELETE during data loading, you can resort to Broker Load. If you need to load a large number of data files with a significant data volume in total (for example, more than 100 GB or even 1 TB), we recommend that you use the Pipe method. Pipe can split the files based on their number or size, breaking down the load job into smaller, sequential tasks. This approach ensures that errors in one file do not impact the entire load job and minimizes the need for retries due to data errors. #### Before you begin[​](#before-you-begin "Direct link to Before you begin") ##### Make source data ready[​](#make-source-data-ready "Direct link to Make source data ready") Make sure the source data you want to load into StarRocks is properly stored in your HDFS cluster. This topic assumes that you want to load `/user/amber/user_behavior_ten_million_rows.parquet` from HDFS into StarRocks. ##### Check privileges[​](#check-privileges "Direct link to Check privileges") You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. ##### Gather authentication details[​](#gather-authentication-details "Direct link to Gather authentication details") You can use the simple authentication method to establish connections with your HDFS cluster. To use simple authentication, you need to gather the username and password of the account that you can use to access the NameNode of the HDFS cluster. #### Use INSERT+FILES()[​](#use-insertfiles "Direct link to Use INSERT+FILES()") This method is available from v3.1 onwards and currently supports only the Parquet, ORC, and CSV (from v3.3.0 onwards) file formats. ##### Advantages of INSERT+FILES()[​](#advantages-of-insertfiles "Direct link to Advantages of INSERT+FILES()") [`FILES()`](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md) can read the file stored in cloud storage based on the path-related properties you specify, infer the table schema of the data in the file, and then return the data from the file as data rows. With `FILES()`, you can: * Query the data directly from HDFS using [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md). * Create and load a table using [CREATE TABLE AS SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE_AS_SELECT.md) (CTAS). * Load the data into an existing table using [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md). ##### Typical examples[​](#typical-examples "Direct link to Typical examples") ###### Querying directly from HDFS using SELECT[​](#querying-directly-from-hdfs-using-select "Direct link to Querying directly from HDFS using SELECT") Querying directly from HDFS using SELECT+`FILES()` can give a good preview of the content of a dataset before you create a table. For example: * Get a preview of the dataset without storing the data. * Query for the min and max values and decide what data types to use. * Check for `NULL` values. The following example queries the data file `/user/amber/user_behavior_ten_million_rows.parquet` stored in the HDFS cluster: ```sql SELECT * FROM FILES ( "path" = "hdfs://:/user/amber/user_behavior_ten_million_rows.parquet", "format" = "parquet", "hadoop.security.authentication" = "simple", "username" = "", "password" = "" ) LIMIT 3; ``` The system returns the following query result: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 543711 | 829192 | 2355072 | pv | 2017-11-27 08:22:37 | | 543711 | 2056618 | 3645362 | pv | 2017-11-27 10:16:46 | | 543711 | 1165492 | 3645362 | pv | 2017-11-27 10:17:00 | +--------+---------+------------+--------------+---------------------+ ``` > **NOTE** > > Notice that the column names as returned above are provided by the Parquet file. ###### Creating and loading a table using CTAS[​](#creating-and-loading-a-table-using-ctas "Direct link to Creating and loading a table using CTAS") This is a continuation of the previous example. The previous query is wrapped in CREATE TABLE AS SELECT (CTAS) to automate the table creation using schema inference. This means StarRocks will infer the table schema, create the table you want, and then load the data into the table. The column names and types are not required to create a table when using the `FILES()` table function with Parquet files as the Parquet format includes the column names. > **NOTE** > > The syntax of CREATE TABLE when using schema inference does not allow setting the number of replicas, so set it before creating the table. The example below is for a system with three replicas: > > ```sql > ADMIN SET FRONTEND CONFIG ('default_replication_num' = "3"); > > ``` Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Use CTAS to create a table and load the data of the data file `/user/amber/user_behavior_ten_million_rows.parquet` into the table: ```sql CREATE TABLE user_behavior_inferred AS SELECT * FROM FILES ( "path" = "hdfs://:/user/amber/user_behavior_ten_million_rows.parquet", "format" = "parquet", "hadoop.security.authentication" = "simple", "username" = "", "password" = "" ); ``` After creating the table, you can view its schema by using [DESCRIBE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DESCRIBE.md): ```sql DESCRIBE user_behavior_inferred; ``` The system returns the following query result: ```plain +--------------+-----------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------+-----------+------+-------+---------+-------+ | UserID | bigint | YES | true | NULL | | | ItemID | bigint | YES | true | NULL | | | CategoryID | bigint | YES | true | NULL | | | BehaviorType | varbinary | YES | false | NULL | | | Timestamp | varbinary | YES | false | NULL | | +--------------+-----------+------+-------+---------+-------+ ``` Query the table to verify that the data has been loaded into it. Example: ```sql SELECT * from user_behavior_inferred LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plaintext +--------+--------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+--------+------------+--------------+---------------------+ | 84 | 56257 | 1879194 | pv | 2017-11-26 05:56:23 | | 84 | 108021 | 2982027 | pv | 2017-12-02 05:43:00 | | 84 | 390657 | 1879194 | pv | 2017-11-28 11:20:30 | +--------+--------+------------+--------------+---------------------+ ``` ###### Loading into an existing table using INSERT[​](#loading-into-an-existing-table-using-insert "Direct link to Loading into an existing table using INSERT") You may want to customize the table that you are inserting into, for example, the: * column data type, nullable setting, or default values * key types and columns * data partitioning and bucketing > **NOTE** > > Creating the most efficient table structure requires knowledge of how the data will be used and the content of the columns. This topic does not cover table design. For information about table design, see [Table types](https://docs.starrocks.io/docs/table_design/StarRocks_table_design.md). In this example, we are creating a table based on knowledge of how the table will be queried and the data in the Parquet file. The knowledge of the data in the Parquet file can be gained by querying the file directly in HDFS. * Since a query of the dataset in HDFS indicates that the `Timestamp` column contains data that matches a VARBINARY data type, the column type is specified in the following DDL. * By querying the data in HDFS, you can find that there are no `NULL` values in the dataset, so the DDL does not set any columns as nullable. * Based on knowledge of the expected query types, the sort key and bucketing column are set to the column `UserID`. Your use case might be different for this data, so you might decide to use `ItemID` in addition to or instead of `UserID` for the sort key. Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand (we recommend that the table have the same schema as the Parquet file you want to load from HDFS): ```sql CREATE TABLE user_behavior_declared ( UserID int(11), ItemID int(11), CategoryID int(11), BehaviorType varchar(65533), Timestamp varbinary ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID); ``` Display the schema so that you can compare it with the inferred schema produced by the `FILES()` table function: ```sql DESCRIBE user_behavior_declared; ``` ```plaintext +--------------+----------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------+----------------+------+-------+---------+-------+ | UserID | int | NO | true | NULL | | | ItemID | int | NO | false | NULL | | | CategoryID | int | NO | false | NULL | | | BehaviorType | varchar(65533) | NO | false | NULL | | | Timestamp | varbinary | NO | false | NULL | | +--------------+----------------+------+-------+---------+-------+ 5 rows in set (0.00 sec) ``` tip Compare the schema you just created with the schema inferred earlier using the `FILES()` table function. Look at: * data types * nullable * key fields To better control the schema of the destination table and for better query performance, we recommend that you specify the table schema by hand in production environments. After creating the table, you can load it with INSERT INTO SELECT FROM FILES(): ```sql INSERT INTO user_behavior_declared SELECT * FROM FILES ( "path" = "hdfs://:/user/amber/user_behavior_ten_million_rows.parquet", "format" = "parquet", "hadoop.security.authentication" = "simple", "username" = "", "password" = "" ); ``` After the load is complete, you can query the table to verify that the data has been loaded into it. Example: ```sql SELECT * from user_behavior_declared LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 107 | 1568743 | 4476428 | pv | 2017-11-25 14:29:53 | | 107 | 470767 | 1020087 | pv | 2017-11-25 14:32:31 | | 107 | 358238 | 1817004 | pv | 2017-11-25 14:43:23 | +--------+---------+------------+--------------+---------------------+ ``` ###### Check load progress[​](#check-load-progress "Direct link to Check load progress") You can query the progress of INSERT jobs from the [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md) view in the StarRocks Information Schema. This feature is supported from v3.1 onwards. Example: ```sql SELECT * FROM information_schema.loads ORDER BY JOB_ID DESC; ``` For information about the fields provided in the `loads` view, see [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md). If you have submitted multiple load jobs, you can filter on the `LABEL` associated with the job. Example: ```sql SELECT * FROM information_schema.loads WHERE LABEL = 'insert_0d86c3f9-851f-11ee-9c3e-00163e044958' \G *************************** 1. row *************************** JOB_ID: 10214 LABEL: insert_0d86c3f9-851f-11ee-9c3e-00163e044958 DATABASE_NAME: mydatabase STATE: FINISHED PROGRESS: ETL:100%; LOAD:100% TYPE: INSERT PRIORITY: NORMAL SCAN_ROWS: 10000000 FILTERED_ROWS: 0 UNSELECTED_ROWS: 0 SINK_ROWS: 10000000 ETL_INFO: TASK_INFO: resource:N/A; timeout(s):300; max_filter_ratio:0.0 CREATE_TIME: 2023-11-17 15:58:14 ETL_START_TIME: 2023-11-17 15:58:14 ETL_FINISH_TIME: 2023-11-17 15:58:14 LOAD_START_TIME: 2023-11-17 15:58:14 LOAD_FINISH_TIME: 2023-11-17 15:58:18 JOB_DETAILS: {"All backends":{"0d86c3f9-851f-11ee-9c3e-00163e044958":[10120]},"FileNumber":0,"FileSize":0,"InternalTableLoadBytes":311710786,"InternalTableLoadRows":10000000,"ScanBytes":581574034,"ScanRows":10000000,"TaskNumber":1,"Unfinished backends":{"0d86c3f9-851f-11ee-9c3e-00163e044958":[]}} ERROR_MSG: NULL TRACKING_URL: NULL TRACKING_SQL: NULL REJECTED_RECORD_PATH: NULL ``` > **NOTE** > > INSERT is a synchronous command. If an INSERT job is still running, you need to open another session to check its execution status. #### Use Broker Load[​](#use-broker-load "Direct link to Use Broker Load") An asynchronous Broker Load process handles making the connection to HDFS, pulling the data, and storing the data in StarRocks. This method supports the following file formats: * Parquet * ORC * CSV * JSON (supported from v3.2.3 onwards) ##### Advantages of Broker Load[​](#advantages-of-broker-load "Direct link to Advantages of Broker Load") * Broker Load runs in the background and clients do not need to stay connected for the job to continue. * Broker Load is preferred for long-running jobs, with the default timeout spanning 4 hours. * In addition to Parquet and ORC file format, Broker Load supports CSV file format and JSON file format (JSON file format is supported from v3.2.3 onwards). ##### Data flow[​](#data-flow "Direct link to Data flow") ![Workflow of Broker Load](/assets/images/broker_load_how-to-work_en-bb36de70866e6366b2b21808f0f77be8.png) 1. The user creates a load job. 2. The frontend (FE) creates a query plan and distributes the plan to the backend nodes (BEs) or compute nodes (CNs). 3. The BEs or CNs pull the data from the source and load the data into StarRocks. ##### Typical example[​](#typical-example "Direct link to Typical example") Create a table, start a load process that pulls the data file `/user/amber/user_behavior_ten_million_rows.parquet` from HDFS, and verify the progress and success of the data loading. ###### Create a database and a table[​](#create-a-database-and-a-table "Direct link to Create a database and a table") Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand (we recommend that the table has the same schema as the Parquet file that you want to load from HDFS): ```sql CREATE TABLE user_behavior ( UserID int(11), ItemID int(11), CategoryID int(11), BehaviorType varchar(65533), Timestamp varbinary ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID); ``` ###### Start a Broker Load[​](#start-a-broker-load "Direct link to Start a Broker Load") Run the following command to start a Broker Load job that loads data from the data file `/user/amber/user_behavior_ten_million_rows.parquet` to the `user_behavior` table: ```sql LOAD LABEL user_behavior ( DATA INFILE("hdfs://:/user/amber/user_behavior_ten_million_rows.parquet") INTO TABLE user_behavior FORMAT AS "parquet" ) WITH BROKER ( "hadoop.security.authentication" = "simple", "username" = "", "password" = "" ) PROPERTIES ( "timeout" = "72000" ); ``` This job has four main sections: * `LABEL`: A string used when querying the state of the load job. * `LOAD` declaration: The source URI, source data format, and destination table name. * `BROKER`: The connection details for the source. * `PROPERTIES`: The timeout value and any other properties to apply to the load job. For detailed syntax and parameter descriptions, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). ###### Check load progress[​](#check-load-progress-1 "Direct link to Check load progress") You can query the progress of Broker Load jobs from the `information_schema.loads` view. This feature is supported from v3.1 onwards. ```sql SELECT * FROM information_schema.loads; ``` For information about the fields provided in the `loads` view, see [Information Schema](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md)). If you have submitted multiple load jobs, you can filter on the `LABEL` associated with the job. Example: ```sql SELECT * FROM information_schema.loads WHERE LABEL = 'user_behavior'; ``` In the output below there are two entries for the load job `user_behavior`: * The first record shows a state of `CANCELLED`. Scroll to `ERROR_MSG`, and you can see that the job has failed due to `listPath failed`. * The second record shows a state of `FINISHED`, which means that the job has succeeded. ```plaintext JOB_ID|LABEL |DATABASE_NAME|STATE |PROGRESS |TYPE |PRIORITY|SCAN_ROWS|FILTERED_ROWS|UNSELECTED_ROWS|SINK_ROWS|ETL_INFO|TASK_INFO |CREATE_TIME |ETL_START_TIME |ETL_FINISH_TIME |LOAD_START_TIME |LOAD_FINISH_TIME |JOB_DETAILS |ERROR_MSG |TRACKING_URL|TRACKING_SQL|REJECTED_RECORD_PATH| ------+-------------------------------------------+-------------+---------+-------------------+------+--------+---------+-------------+---------------+---------+--------+----------------------------------------------------+-------------------+-------------------+-------------------+-------------------+-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+------------+------------+--------------------+ 10121|user_behavior |mydatabase |CANCELLED|ETL:N/A; LOAD:N/A |BROKER|NORMAL | 0| 0| 0| 0| |resource:N/A; timeout(s):72000; max_filter_ratio:0.0|2023-08-10 14:59:30| | | |2023-08-10 14:59:34|{"All backends":{},"FileNumber":0,"FileSize":0,"InternalTableLoadBytes":0,"InternalTableLoadRows":0,"ScanBytes":0,"ScanRows":0,"TaskNumber":0,"Unfinished backends":{}} |type:ETL_RUN_FAIL; msg:listPath failed| | | | 10106|user_behavior |mydatabase |FINISHED |ETL:100%; LOAD:100%|BROKER|NORMAL | 86953525| 0| 0| 86953525| |resource:N/A; timeout(s):72000; max_filter_ratio:0.0|2023-08-10 14:50:15|2023-08-10 14:50:19|2023-08-10 14:50:19|2023-08-10 14:50:19|2023-08-10 14:55:10|{"All backends":{"a5fe5e1d-d7d0-4826-ba99-c7348f9a5f2f":[10004]},"FileNumber":1,"FileSize":1225637388,"InternalTableLoadBytes":2710603082,"InternalTableLoadRows":86953525,"ScanBytes":1225637388,"ScanRows":86953525,"TaskNumber":1,"Unfinished backends":{"a5| | | | | ``` After you confirm that the load job has finished, you can check a subset of the destination table to see if the data has been successfully loaded. Example: ```sql SELECT * from user_behavior LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 142 | 2869980 | 2939262 | pv | 2017-11-25 03:43:22 | | 142 | 2522236 | 1669167 | pv | 2017-11-25 15:14:12 | | 142 | 3031639 | 3607361 | pv | 2017-11-25 15:19:25 | +--------+---------+------------+--------------+---------------------+ ``` #### Use Pipe[​](#use-pipe "Direct link to Use Pipe") Starting from v3.2, StarRocks provides the Pipe loading method, which currently supports only the Parquet and ORC file formats. ##### Advantages of Pipe[​](#advantages-of-pipe "Direct link to Advantages of Pipe") Pipe is ideal for continuous data loading and large-scale data loading: * **Large-scale data loading in micro-batches helps reduce the cost of retries caused by data errors.** With the help of Pipe, StarRocks enables the efficient loading of a large number of data files with a significant data volume in total. Pipe automatically splits the files based on their number or size, breaking down the load job into smaller, sequential tasks. This approach ensures that errors in one file do not impact the entire load job. The load status of each file is recorded by Pipe, allowing you to easily identify and fix files that contain errors. By minimizing the need for retries due to data errors, this approach helps to reduce costs. * **Continuous data loading helps reduce manpower.** Pipe helps you write new or updated data files to a specific location and continuously load the new data from these files into StarRocks. After you create a Pipe job with `"AUTO_INGEST" = "TRUE"` specified, it will constantly monitor changes to the data files stored in the specified path and automatically load new or updated data from the data files into the destination StarRocks table. Additionally, Pipe performs file uniqueness checks to help prevent duplicate data loading.During the loading process, Pipe checks the uniqueness of each data file based on the file name and digest. If a file with a specific file name and digest has already been processed by a Pipe job, the Pipe job will skip all subsequent files with the same file name and digest. Note that HDFS uses LastModifiedTime as file digest. The load status of each data file is recorded and saved to the `information_schema.pipe_files` view. After a Pipe job associated with the view is deleted, the records about the files loaded in that job will also be deleted. ##### Data flow[​](#data-flow "Direct link to Data flow") ![Pipe data flow](/assets/images/pipe_data_flow-2a4dc0b44a06c987d9afc0ecf632f5d9.png) Pipe is ideal for continuous data loading and large-scale data loading: * **Large-scale data loading in micro-batches helps reduce the cost of retries caused by data errors.** With the help of Pipe, StarRocks enables the efficient loading of a large number of data files with a significant data volume in total. Pipe automatically splits the files based on their number or size, breaking down the load job into smaller, sequential tasks. This approach ensures that errors in one file do not impact the entire load job. The load status of each file is recorded by Pipe, allowing you to easily identify and fix files that contain errors. By minimizing the need for retries due to data errors, this approach helps to reduce costs. * **Continuous data loading helps reduce manpower.** Pipe helps you write new or updated data files to a specific location and continuously load the new data from these files into StarRocks. After you create a Pipe job with `"AUTO_INGEST" = "TRUE"` specified, it will constantly monitor changes to the data files stored in the specified path and automatically load new or updated data from the data files into the destination StarRocks table. Additionally, Pipe performs file uniqueness checks to help prevent duplicate data loading. During the loading process, Pipe checks the uniqueness of each data file based on the file name and digest. If a file with a specific file name and digest has already been processed by a Pipe job, the Pipe job will skip all subsequent files with the same file name and digest. Note that HDFS uses `LastModifiedTime` as file digest. The load status of each data file is recorded and saved to the `information_schema.pipe_files` view. After a Pipe job associated with the view is deleted, the records about the files loaded in that job will also be deleted. ##### Data flow[​](#data-flow-1 "Direct link to Data flow") ![Pipe data flow](/assets/images/pipe_data_flow-2a4dc0b44a06c987d9afc0ecf632f5d9.png) ##### Differences between Pipe and INSERT+FILES()[​](#differences-between-pipe-and-insertfiles "Direct link to Differences between Pipe and INSERT+FILES()") A Pipe job is split into one or more transactions based on the size and number of rows in each data file. Users can query the intermediate results during the loading process. In contrast, an INSERT+`FILES()` job is processed as a single transaction, and users are unable to view the data during the loading process. ##### File loading sequence[​](#file-loading-sequence "Direct link to File loading sequence") For each Pipe job, StarRocks maintains a file queue, from which it fetches and loads data files as micro-batches. Pipe does not ensure that the data files are loaded in the same order as they are uploaded. Therefore, newer data may be loaded prior to older data. ##### Typical example[​](#typical-example-1 "Direct link to Typical example") ###### Create a database and a table[​](#create-a-database-and-a-table-1 "Direct link to Create a database and a table") Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand (we recommend that the table have the same schema as the Parquet file you want to load from HDFS): ```sql CREATE TABLE user_behavior_replica ( UserID int(11), ItemID int(11), CategoryID int(11), BehaviorType varchar(65533), Timestamp varbinary ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID); ``` ###### Start a Pipe job[​](#start-a-pipe-job "Direct link to Start a Pipe job") Run the following command to start a Pipe job that loads data from the data file `/user/amber/user_behavior_ten_million_rows.parquet` to the `user_behavior_replica` table: ```sql CREATE PIPE user_behavior_replica PROPERTIES ( "AUTO_INGEST" = "TRUE" ) AS INSERT INTO user_behavior_replica SELECT * FROM FILES ( "path" = "hdfs://:/user/amber/user_behavior_ten_million_rows.parquet", "format" = "parquet", "hadoop.security.authentication" = "simple", "username" = "", "password" = "" ); ``` This job has four main sections: * `pipe_name`: The name of the pipe. The pipe name must be unique within the database to which the pipe belongs. * `INSERT_SQL`: The INSERT INTO SELECT FROM FILES statement that is used to load data from the specified source data file to the destination table. * `PROPERTIES`: A set of optional parameters that specify how to execute the pipe. These include `AUTO_INGEST`, `POLL_INTERVAL`, `BATCH_SIZE`, and `BATCH_FILES`. Specify these properties in the `"key" = "value"` format. For detailed syntax and parameter descriptions, see [CREATE PIPE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/CREATE_PIPE.md). ###### Check load progress[​](#check-load-progress-2 "Direct link to Check load progress") * Query the progress of Pipe jobs by using [SHOW PIPES](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/SHOW_PIPES.md). ```sql SHOW PIPES; ``` If you have submitted multiple load jobs, you can filter on the `NAME` associated with the job. Example: ```sql SHOW PIPES WHERE NAME = 'user_behavior_replica' \G *************************** 1. row *************************** DATABASE_NAME: mydatabase PIPE_ID: 10252 PIPE_NAME: user_behavior_replica STATE: RUNNING TABLE_NAME: mydatabase.user_behavior_replica LOAD_STATUS: {"loadedFiles":1,"loadedBytes":132251298,"loadingFiles":0,"lastLoadedTime":"2023-11-17 16:13:22"} LAST_ERROR: NULL CREATED_TIME: 2023-11-17 16:13:15 1 row in set (0.00 sec) ``` * Query the progress of Pipe jobs from the [`pipes`](https://docs.starrocks.io/docs/sql-reference/information_schema/pipes.md) view in the StarRocks Information Schema. ```sql SELECT * FROM information_schema.pipes; ``` If you have submitted multiple load jobs, you can filter on the `PIPE_NAME` associated with the job. Example: ```sql SELECT * FROM information_schema.pipes WHERE pipe_name = 'user_behavior_replica' \G *************************** 1. row *************************** DATABASE_NAME: mydatabase PIPE_ID: 10252 PIPE_NAME: user_behavior_replica STATE: RUNNING TABLE_NAME: mydatabase.user_behavior_replica LOAD_STATUS: {"loadedFiles":1,"loadedBytes":132251298,"loadingFiles":0,"lastLoadedTime":"2023-11-17 16:13:22"} LAST_ERROR: CREATED_TIME: 2023-11-17 16:13:15 1 row in set (0.00 sec) ``` ###### Check file status[​](#check-file-status "Direct link to Check file status") You can query the load status of the files loaded from the [`pipe_files`](https://docs.starrocks.io/docs/sql-reference/information_schema/pipe_files.md) view in the StarRocks Information Schema. ```sql SELECT * FROM information_schema.pipe_files; ``` If you have submitted multiple load jobs, you can filter on the `PIPE_NAME` associated with the job. Example: ```sql SELECT * FROM information_schema.pipe_files WHERE pipe_name = 'user_behavior_replica' \G *************************** 1. row *************************** DATABASE_NAME: mydatabase PIPE_ID: 10252 PIPE_NAME: user_behavior_replica FILE_NAME: hdfs://172.26.195.67:9000/user/amber/user_behavior_ten_million_rows.parquet FILE_VERSION: 1700035418838 FILE_SIZE: 132251298 LAST_MODIFIED: 2023-11-15 08:03:38 LOAD_STATE: FINISHED STAGED_TIME: 2023-11-17 16:13:16 START_LOAD_TIME: 2023-11-17 16:13:17 FINISH_LOAD_TIME: 2023-11-17 16:13:22 ERROR_MSG: 1 row in set (0.02 sec) ``` ###### Manage Pipes[​](#manage-pipes "Direct link to Manage Pipes") You can alter, suspend or resume, drop, or query the pipes you have created and retry to load specific data files. For more information, see [ALTER PIPE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/ALTER_PIPE.md), [SUSPEND or RESUME PIPE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/SUSPEND_or_RESUME_PIPE.md), [DROP PIPE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/DROP_PIPE.md), [SHOW PIPES](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/SHOW_PIPES.md), and [RETRY FILE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/RETRY_FILE.md). --- ### S3 Load ### Load data from AWS S3 StarRocks provides the following options for loading data from AWS S3: * Synchronous loading using [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md)+[`FILES()`](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md) * Asynchronous loading using [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) * Continuous asynchronous loading using [Pipe](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/CREATE_PIPE.md) Each of these options has its own advantages, which are detailed in the following sections. In most cases, we recommend that you use the INSERT+`FILES()` method, which is much easier to use. However, the INSERT+`FILES()` method currently supports only the Parquet, ORC, and CSV file formats. Therefore, if you need to load data of other file formats such as JSON, or perform data changes such as DELETE during data loading, you can resort to Broker Load. If you need to load a large number of data files with a significant data volume in total (for example, more than 100 GB or even 1 TB), we recommend that you use the Pipe method. Pipe can split the files based on their number or size, breaking down the load job into smaller, sequential tasks. This approach ensures that errors in one file do not impact the entire load job and minimizes the need for retries due to data errors. #### Before you begin[​](#before-you-begin "Direct link to Before you begin") ##### Make source data ready[​](#make-source-data-ready "Direct link to Make source data ready") Make sure the source data you want to load into StarRocks is properly stored in an S3 bucket. You may also consider where the data and the database are located, because data transfer costs are much lower when your bucket and your StarRocks cluster are located in the same region. In this topic, we provide you with a sample dataset in an S3 bucket, `s3://starrocks-examples/user-behavior-10-million-rows.parquet`. You can access that dataset with any valid credentials as the object is readable by any AWS authenticated user. ##### Check privileges[​](#check-privileges "Direct link to Check privileges") You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. ##### Gather authentication details[​](#gather-authentication-details "Direct link to Gather authentication details") The examples in this topic use IAM user-based authentication. To ensure that you have permission to read data from AWS S3, we recommend that you read [Preparation for IAM user-based authentication](https://docs.starrocks.io/docs/integrations/authenticate_to_aws_resources.md) and follow the instructions to create an IAM user with proper [IAM policies](https://docs.starrocks.io/docs/sql-reference/aws_iam_policies.md) configured. In a nutshell, if you practice IAM user-based authentication, you need to gather information about the following AWS resources: * The S3 bucket that stores your data. * The S3 object key (object name) if accessing a specific object in the bucket. Note that the object key can include a prefix if your S3 objects are stored in sub-folders. * The AWS region to which the S3 bucket belongs. * The access key and secret key used as access credentials. For information about all the authentication methods available, see [Authenticate to AWS resources](https://docs.starrocks.io/docs/integrations/authenticate_to_aws_resources.md). #### Use INSERT+FILES()[​](#use-insertfiles "Direct link to Use INSERT+FILES()") This method is available from v3.1 onwards and currently supports only the Parquet, ORC, and CSV (from v3.3.0 onwards) file formats. ##### Advantages of INSERT+FILES()[​](#advantages-of-insertfiles "Direct link to Advantages of INSERT+FILES()") [`FILES()`](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md) can read the file stored in cloud storage based on the path-related properties you specify, infer the table schema of the data in the file, and then return the data from the file as data rows. With `FILES()`, you can: * Query the data directly from S3 using [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md). * Create and load a table using [CREATE TABLE AS SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE_AS_SELECT.md) (CTAS). * Load the data into an existing table using [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md). ##### Typical examples[​](#typical-examples "Direct link to Typical examples") ###### Querying directly from S3 using SELECT[​](#querying-directly-from-s3-using-select "Direct link to Querying directly from S3 using SELECT") Querying directly from S3 using SELECT+`FILES()` can give a good preview of the content of a dataset before you create a table. For example: * Get a preview of the dataset without storing the data. * Query for the min and max values and decide what data types to use. * Check for `NULL` values. The following example queries the sample dataset `s3://starrocks-examples/user-behavior-10-million-rows.parquet`: ```sql SELECT * FROM FILES ( "path" = "s3://starrocks-examples/user-behavior-10-million-rows.parquet", "format" = "parquet", "aws.s3.region" = "us-east-1", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB" ) LIMIT 3; ``` > **NOTE** > > Substitute your credentials for `AAA` and `BBB` in the above command. Any valid `aws.s3.access_key` and `aws.s3.secret_key` can be used, as the object is readable by any AWS authenticated user. The system returns the following query result: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 1 | 2576651 | 149192 | pv | 2017-11-25 01:21:25 | | 1 | 3830808 | 4181361 | pv | 2017-11-25 07:04:53 | | 1 | 4365585 | 2520377 | pv | 2017-11-25 07:49:06 | +--------+---------+------------+--------------+---------------------+ ``` > **NOTE** > > Notice that the column names as returned above are provided by the Parquet file. ###### Creating and loading a table using CTAS[​](#creating-and-loading-a-table-using-ctas "Direct link to Creating and loading a table using CTAS") This is a continuation of the previous example. The previous query is wrapped in CREATE TABLE AS SELECT (CTAS) to automate the table creation using schema inference. This means StarRocks will infer the table schema, create the table you want, and then load the data into the table. The column names and types are not required to create a table when using the `FILES()` table function with Parquet files as the Parquet format includes the column names. > **NOTE** > > The syntax of CREATE TABLE when using schema inference does not allow setting the number of replicas, so set it before creating the table. The example below is for a system with one replica: > > ```sql > ADMIN SET FRONTEND CONFIG ('default_replication_num' = "1"); > > ``` Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Use CTAS to create a table and load the data of the sample dataset `s3://starrocks-examples/user-behavior-10-million-rows.parquet` into the table: ```sql CREATE TABLE user_behavior_inferred AS SELECT * FROM FILES ( "path" = "s3://starrocks-examples/user-behavior-10-million-rows.parquet", "format" = "parquet", "aws.s3.region" = "us-east-1", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB" ); ``` > **NOTE** > > Substitute your credentials for `AAA` and `BBB` in the above command. Any valid `aws.s3.access_key` and `aws.s3.secret_key` can be used, as the object is readable by any AWS authenticated user. After creating the table, you can view its schema by using [DESCRIBE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DESCRIBE.md): ```sql DESCRIBE user_behavior_inferred; ``` The system returns the following query result: ```plain +--------------+------------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------+------------------+------+-------+---------+-------+ | UserID | bigint | YES | true | NULL | | | ItemID | bigint | YES | true | NULL | | | CategoryID | bigint | YES | true | NULL | | | BehaviorType | varchar(1048576) | YES | false | NULL | | | Timestamp | varchar(1048576) | YES | false | NULL | | +--------------+------------------+------+-------+---------+-------+ ``` Query the table to verify that the data has been loaded into it. Example: ```sql SELECT * from user_behavior_inferred LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 225586 | 3694958 | 1040727 | pv | 2017-12-01 00:58:40 | | 225586 | 3726324 | 965809 | pv | 2017-12-01 02:16:02 | | 225586 | 3732495 | 1488813 | pv | 2017-12-01 00:59:46 | +--------+---------+------------+--------------+---------------------+ ``` ###### Loading into an existing table using INSERT[​](#loading-into-an-existing-table-using-insert "Direct link to Loading into an existing table using INSERT") You may want to customize the table that you are inserting into, for example, the: * column data type, nullable setting, or default values * key types and columns * data partitioning and bucketing > **NOTE** > > Creating the most efficient table structure requires knowledge of how the data will be used and the content of the columns. This topic does not cover table design. For information about table design, see [Table types](https://docs.starrocks.io/docs/table_design/StarRocks_table_design.md). In this example, we are creating a table based on knowledge of how the table will be queried and the data in the Parquet file. The knowledge of the data in the Parquet file can be gained by querying the file directly in S3. * Since a query of the dataset in S3 indicates that the `Timestamp` column contains data that matches a VARCHAR data type, and StarRocks can cast from VARCHAR to DATETIME, the data type is changed to DATETIME in the following DDL. * By querying the data in S3, you can find that there are no `NULL` values in the dataset, so the DDL could also set all columns as non-nullable. * Based on knowledge of the expected query types, the sort key and bucketing column are set to the column `UserID`. Your use case might be different for this data, so you might decide to use `ItemID` in addition to, or instead of, `UserID` for the sort key. Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand: ```sql CREATE TABLE user_behavior_declared ( UserID int(11), ItemID int(11), CategoryID int(11), BehaviorType varchar(65533), Timestamp datetime ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID); ``` Display the schema so that you can compare it with the inferred schema produced by the `FILES()` table function: ```sql DESCRIBE user_behavior_declared; ``` ```plaintext +--------------+----------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------+----------------+------+-------+---------+-------+ | UserID | int | YES | true | NULL | | | ItemID | int | YES | false | NULL | | | CategoryID | int | YES | false | NULL | | | BehaviorType | varchar(65533) | YES | false | NULL | | | Timestamp | datetime | YES | false | NULL | | +--------------+----------------+------+-------+---------+-------+ ``` tip Compare the schema you just created with the schema inferred earlier using the `FILES()` table function. Look at: * data types * nullable * key fields To better control the schema of the destination table and for better query performance, we recommend that you specify the table schema by hand in production environments. After creating the table, you can load it with INSERT INTO SELECT FROM FILES(): ```sql INSERT INTO user_behavior_declared SELECT * FROM FILES ( "path" = "s3://starrocks-examples/user-behavior-10-million-rows.parquet", "format" = "parquet", "aws.s3.region" = "us-east-1", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB" ); ``` > **NOTE** > > Substitute your credentials for `AAA` and `BBB` in the above command. Any valid `aws.s3.access_key` and `aws.s3.secret_key` can be used, as the object is readable by any AWS authenticated user. After the load is complete, you can query the table to verify that the data has been loaded into it. Example: ```sql SELECT * from user_behavior_declared LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 393529 | 3715112 | 883960 | pv | 2017-12-02 02:45:44 | | 393529 | 2650583 | 883960 | pv | 2017-12-02 02:45:59 | | 393529 | 3715112 | 883960 | pv | 2017-12-02 03:00:56 | +--------+---------+------------+--------------+---------------------+ ``` ###### Check load progress[​](#check-load-progress "Direct link to Check load progress") You can query the progress of INSERT jobs from the [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md) view in the StarRocks Information Schema. This feature is supported from v3.1 onwards. Example: ```sql SELECT * FROM information_schema.loads ORDER BY JOB_ID DESC; ``` For information about the fields provided in the `loads` view, see [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md). If you have submitted multiple load jobs, you can filter on the `LABEL` associated with the job. Example: ```sql SELECT * FROM information_schema.loads WHERE LABEL = 'insert_e3b882f5-7eb3-11ee-ae77-00163e267b60' \G *************************** 1. row *************************** JOB_ID: 10243 LABEL: insert_e3b882f5-7eb3-11ee-ae77-00163e267b60 DATABASE_NAME: mydatabase STATE: FINISHED PROGRESS: ETL:100%; LOAD:100% TYPE: INSERT PRIORITY: NORMAL SCAN_ROWS: 10000000 FILTERED_ROWS: 0 UNSELECTED_ROWS: 0 SINK_ROWS: 10000000 ETL_INFO: TASK_INFO: resource:N/A; timeout(s):300; max_filter_ratio:0.0 CREATE_TIME: 2023-11-09 11:56:01 ETL_START_TIME: 2023-11-09 11:56:01 ETL_FINISH_TIME: 2023-11-09 11:56:01 LOAD_START_TIME: 2023-11-09 11:56:01 LOAD_FINISH_TIME: 2023-11-09 11:56:44 JOB_DETAILS: {"All backends":{"e3b882f5-7eb3-11ee-ae77-00163e267b60":[10142]},"FileNumber":0,"FileSize":0,"InternalTableLoadBytes":311710786,"InternalTableLoadRows":10000000,"ScanBytes":581574034,"ScanRows":10000000,"TaskNumber":1,"Unfinished backends":{"e3b882f5-7eb3-11ee-ae77-00163e267b60":[]}} ERROR_MSG: NULL TRACKING_URL: NULL TRACKING_SQL: NULL REJECTED_RECORD_PATH: NULL ``` > **NOTE** > > INSERT is a synchronous command. If an INSERT job is still running, you need to open another session to check its execution status. #### Use Broker Load[​](#use-broker-load "Direct link to Use Broker Load") An asynchronous Broker Load process handles making the connection to S3, pulling the data, and storing the data in StarRocks. This method supports the following file formats: * Parquet * ORC * CSV * JSON (supported from v3.2.3 onwards) ##### Advantages of Broker Load[​](#advantages-of-broker-load "Direct link to Advantages of Broker Load") * Broker Load runs in the background and clients do not need to stay connected for the job to continue. * Broker Load is preferred for long-running jobs, with the default timeout spanning 4 hours. * In addition to Parquet and ORC file format, Broker Load supports CSV file format and JSON file format (JSON file format is supported from v3.2.3 onwards). ##### Data flow[​](#data-flow "Direct link to Data flow") ![Workflow of Broker Load](/assets/images/broker_load_how-to-work_en-bb36de70866e6366b2b21808f0f77be8.png) 1. The user creates a load job. 2. The frontend (FE) creates a query plan and distributes the plan to the backend nodes (BEs) or compute nodes (CNs). 3. The BEs or CNs pull the data from the source and load the data into StarRocks. ##### Typical example[​](#typical-example "Direct link to Typical example") Create a table, start a load process that pulls the sample dataset `s3://starrocks-examples/user-behavior-10-million-rows.parquet` from S3, and verify the progress and success of the data loading. ###### Create a database and a table[​](#create-a-database-and-a-table "Direct link to Create a database and a table") Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand (we recommend that the table has the same schema as the Parquet file that you want to load from AWS S3): ```sql CREATE TABLE user_behavior ( UserID int(11), ItemID int(11), CategoryID int(11), BehaviorType varchar(65533), Timestamp datetime ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID); ``` ###### Start a Broker Load[​](#start-a-broker-load "Direct link to Start a Broker Load") Run the following command to start a Broker Load job that loads data from the sample dataset `s3://starrocks-examples/user-behavior-10-million-rows.parquet` to the `user_behavior` table: ```sql LOAD LABEL user_behavior ( DATA INFILE("s3://starrocks-examples/user-behavior-10-million-rows.parquet") INTO TABLE user_behavior FORMAT AS "parquet" ) WITH BROKER ( "aws.s3.enable_ssl" = "true", "aws.s3.use_instance_profile" = "false", "aws.s3.region" = "us-east-1", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB" ) PROPERTIES ( "timeout" = "72000" ); ``` > **NOTE** > > Substitute your credentials for `AAA` and `BBB` in the above command. Any valid `aws.s3.access_key` and `aws.s3.secret_key` can be used, as the object is readable by any AWS authenticated user. This job has four main sections: * `LABEL`: A string used when querying the state of the load job. * `LOAD` declaration: The source URI, source data format, and destination table name. * `BROKER`: The connection details for the source. * `PROPERTIES`: The timeout value and any other properties to apply to the load job. For detailed syntax and parameter descriptions, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). ###### Check load progress[​](#check-load-progress-1 "Direct link to Check load progress") You can query the progress of the Broker Load job from the [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md) view in the StarRocks Information Schema. This feature is supported from v3.1 onwards. ```sql SELECT * FROM information_schema.loads WHERE LABEL = 'user_behavior'; ``` For information about the fields provided in the `loads` view, see [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md). This record shows a state of `LOADING`, and the progress is 39%. If you see something similar, then run the command again until you see a state of `FINISHED`. ```plaintext JOB_ID: 10466 LABEL: user_behavior DATABASE_NAME: mydatabase STATE: LOADING PROGRESS: ETL:100%; LOAD:39% TYPE: BROKER PRIORITY: NORMAL SCAN_ROWS: 4620288 FILTERED_ROWS: 0 UNSELECTED_ROWS: 0 SINK_ROWS: 4620288 ETL_INFO: TASK_INFO: resource:N/A; timeout(s):72000; max_filter_ratio:0.0 CREATE_TIME: 2024-02-28 22:11:36 ETL_START_TIME: 2024-02-28 22:11:41 ETL_FINISH_TIME: 2024-02-28 22:11:41 LOAD_START_TIME: 2024-02-28 22:11:41 LOAD_FINISH_TIME: NULL JOB_DETAILS: {"All backends":{"2fb97223-b14c-404b-9be1-83aa9b3a7715":[10004]},"FileNumber":1,"FileSize":136901706,"InternalTableLoadBytes":144032784,"InternalTableLoadRows":4620288,"ScanBytes":143969616,"ScanRows":4620288,"TaskNumber":1,"Unfinished backends":{"2fb97223-b14c-404b-9be1-83aa9b3a7715":[10004]}} ERROR_MSG: NULL TRACKING_URL: NULL TRACKING_SQL: NULL REJECTED_RECORD_PATH: NULL ``` After you confirm that the load job has finished, you can check a subset of the destination table to see if the data has been successfully loaded. Example: ```sql SELECT * from user_behavior LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 34 | 856384 | 1029459 | pv | 2017-11-27 14:43:27 | | 34 | 5079705 | 1029459 | pv | 2017-11-27 14:44:13 | | 34 | 4451615 | 1029459 | pv | 2017-11-27 14:45:52 | +--------+---------+------------+--------------+---------------------+ ``` #### Use Pipe[​](#use-pipe "Direct link to Use Pipe") Starting from v3.2, StarRocks provides the Pipe loading method, which currently supports only the Parquet and ORC file formats. ##### Advantages of Pipe[​](#advantages-of-pipe "Direct link to Advantages of Pipe") Pipe is ideal for continuous data loading and large-scale data loading: * **Large-scale data loading in micro-batches helps reduce the cost of retries caused by data errors.** With the help of Pipe, StarRocks enables the efficient loading of a large number of data files with a significant data volume in total. Pipe automatically splits the files based on their number or size, breaking down the load job into smaller, sequential tasks. This approach ensures that errors in one file do not impact the entire load job. The load status of each file is recorded by Pipe, allowing you to easily identify and fix files that contain errors. By minimizing the need for retries due to data errors, this approach helps to reduce costs. * **Continuous data loading helps reduce manpower.** Pipe helps you write new or updated data files to a specific location and continuously load the new data from these files into StarRocks. After you create a Pipe job with `"AUTO_INGEST" = "TRUE"` specified, it will constantly monitor changes to the data files stored in the specified path and automatically load new or updated data from the data files into the destination StarRocks table. Additionally, Pipe performs file uniqueness checks to help prevent duplicate data loading.During the loading process, Pipe checks the uniqueness of each data file based on the file name and digest. If a file with a specific file name and digest has already been processed by a Pipe job, the Pipe job will skip all subsequent files with the same file name and digest. Note that object storage like AWS S3 uses ETag as file digest. The load status of each data file is recorded and saved to the `information_schema.pipe_files` view. After a Pipe job associated with the view is deleted, the records about the files loaded in that job will also be deleted. ##### Data flow[​](#data-flow "Direct link to Data flow") ![Pipe data flow](/assets/images/pipe_data_flow-2a4dc0b44a06c987d9afc0ecf632f5d9.png) Pipe is ideal for continuous data loading and large-scale data loading: * **Large-scale data loading in micro-batches helps reduce the cost of retries caused by data errors.** With the help of Pipe, StarRocks enables the efficient loading of a large number of data files with a significant data volume in total. Pipe automatically splits the files based on their number or size, breaking down the load job into smaller, sequential tasks. This approach ensures that errors in one file do not impact the entire load job. The load status of each file is recorded by Pipe, allowing you to easily identify and fix files that contain errors. By minimizing the need for retries due to data errors, this approach helps to reduce costs. * **Continuous data loading helps reduce manpower.** Pipe helps you write new or updated data files to a specific location and continuously load the new data from these files into StarRocks. After you create a Pipe job with `"AUTO_INGEST" = "TRUE"` specified, it will constantly monitor changes to the data files stored in the specified path and automatically load new or updated data from the data files into the destination StarRocks table. Additionally, Pipe performs file uniqueness checks to help prevent duplicate data loading.During the loading process, Pipe checks the uniqueness of each data file based on the file name and digest. If a file with a specific file name and digest has already been processed by a Pipe job, the Pipe job will skip all subsequent files with the same file name and digest. Note that object storage like AWS S3 uses `ETag` as file digest. The load status of each data file is recorded and saved to the `information_schema.pipe_files` view. After a Pipe job associated with the view is deleted, the records about the files loaded in that job will also be deleted. ##### Differences between Pipe and INSERT+FILES()[​](#differences-between-pipe-and-insertfiles "Direct link to Differences between Pipe and INSERT+FILES()") A Pipe job is split into one or more transactions based on the size and number of rows in each data file. Users can query the intermediate results during the loading process. In contrast, an INSERT+`FILES()` job is processed as a single transaction, and users are unable to view the data during the loading process. ##### File loading sequence[​](#file-loading-sequence "Direct link to File loading sequence") For each Pipe job, StarRocks maintains a file queue, from which it fetches and loads data files as micro-batches. Pipe does not ensure that the data files are loaded in the same order as they are uploaded. Therefore, newer data may be loaded prior to older data. ##### Typical example[​](#typical-example-1 "Direct link to Typical example") ###### Create a database and a table[​](#create-a-database-and-a-table-1 "Direct link to Create a database and a table") Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand (we recommend that the table have the same schema as the Parquet file you want to load from AWS S3): ```sql CREATE TABLE user_behavior_from_pipe ( UserID int(11), ItemID int(11), CategoryID int(11), BehaviorType varchar(65533), Timestamp datetime ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID); ``` ###### Start a Pipe job[​](#start-a-pipe-job "Direct link to Start a Pipe job") Run the following command to start a Pipe job that loads data from the sample dataset `s3://starrocks-examples/user-behavior-10-million-rows/` to the `user_behavior_from_pipe` table. This pipe job uses both micro batches, and continuous loading (described above) pipe-specific features. The other examples in this guide load a single Parquet file with 10 million rows. For the pipe example, the same dataset is split into 57 separate files, and these are all stored in one S3 folder. Note in the `CREATE PIPE` command below the `path` is the URI for an S3 folder and rather than providing a filename the URI ends in `/*`. By setting `AUTO_INGEST` and specifying a folder rather than an individual file the pipe job will poll the S3 folder for new files and ingest them as they are added to the folder. ```sql CREATE PIPE user_behavior_pipe PROPERTIES ( "AUTO_INGEST" = "TRUE" ) AS INSERT INTO user_behavior_from_pipe SELECT * FROM FILES ( "path" = "s3://starrocks-examples/user-behavior-10-million-rows/*", "format" = "parquet", "aws.s3.region" = "us-east-1", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB" ); ``` > **NOTE** > > Substitute your credentials for `AAA` and `BBB` in the above command. Any valid `aws.s3.access_key` and `aws.s3.secret_key` can be used, as the object is readable by any AWS authenticated user. This job has four main sections: * `pipe_name`: The name of the pipe. The pipe name must be unique within the database to which the pipe belongs. * `INSERT_SQL`: The INSERT INTO SELECT FROM FILES statement that is used to load data from the specified source data file to the destination table. * `PROPERTIES`: A set of optional parameters that specify how to execute the pipe. These include `AUTO_INGEST`, `POLL_INTERVAL`, `BATCH_SIZE`, and `BATCH_FILES`. Specify these properties in the `"key" = "value"` format. For detailed syntax and parameter descriptions, see [CREATE PIPE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/CREATE_PIPE.md). ###### Check load progress[​](#check-load-progress-2 "Direct link to Check load progress") * Query the progress of the Pipe job by using [SHOW PIPES](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/SHOW_PIPES.md) in the current database to which the Pipe job belongs. ```sql SHOW PIPES WHERE NAME = 'user_behavior_pipe' \G ``` The following result is returned: tip In the output shown below the pipe is in the `RUNNING` state. A pipe will stay in the `RUNNING` state until you manually stop it. The output also shows the number of files loaded (57) and the last time that a file was loaded. ```sql *************************** 1. row *************************** DATABASE_NAME: mydatabase PIPE_ID: 10476 PIPE_NAME: user_behavior_pipe STATE: RUNNING TABLE_NAME: mydatabase.user_behavior_from_pipe LOAD_STATUS: {"loadedFiles":57,"loadedBytes":295345637,"loadingFiles":0,"lastLoadedTime":"2024-02-28 22:14:19"} LAST_ERROR: NULL CREATED_TIME: 2024-02-28 22:13:41 1 row in set (0.02 sec) ``` * Query the progress of the Pipe job from the [`pipes`](https://docs.starrocks.io/docs/sql-reference/information_schema/pipes.md) view in the StarRocks Information Schema. ```sql SELECT * FROM information_schema.pipes WHERE pipe_name = 'user_behavior_replica' \G ``` The following result is returned: tip Some of the queries in this guide end in `\G` instead of a semicolon (`;`). This causes the MySQL client to output the results in vertical format. If you are using DBeaver or another client you may need to use a semicolon (`;`) rather than `\G`. ```sql *************************** 1. row *************************** DATABASE_NAME: mydatabase PIPE_ID: 10217 PIPE_NAME: user_behavior_replica STATE: RUNNING TABLE_NAME: mydatabase.user_behavior_replica LOAD_STATUS: {"loadedFiles":1,"loadedBytes":132251298,"loadingFiles":0,"lastLoadedTime":"2023-11-09 15:35:42"} LAST_ERROR: CREATED_TIME: 9891-01-15 07:51:45 1 row in set (0.01 sec) ``` ###### Check file status[​](#check-file-status "Direct link to Check file status") You can query the load status of the files loaded from the [`pipe_files`](https://docs.starrocks.io/docs/sql-reference/information_schema/pipe_files.md) view in the StarRocks Information Schema. ```sql SELECT * FROM information_schema.pipe_files WHERE pipe_name = 'user_behavior_replica' \G ``` The following result is returned: ```sql *************************** 1. row *************************** DATABASE_NAME: mydatabase PIPE_ID: 10217 PIPE_NAME: user_behavior_replica FILE_NAME: s3://starrocks-examples/user-behavior-10-million-rows.parquet FILE_VERSION: e29daa86b1120fea58ad0d047e671787-8 FILE_SIZE: 132251298 LAST_MODIFIED: 2023-11-06 13:25:17 LOAD_STATE: FINISHED STAGED_TIME: 2023-11-09 15:35:02 START_LOAD_TIME: 2023-11-09 15:35:03 FINISH_LOAD_TIME: 2023-11-09 15:35:42 ERROR_MSG: 1 row in set (0.03 sec) ``` ###### Manage Pipe jobs[​](#manage-pipe-jobs "Direct link to Manage Pipe jobs") You can alter, suspend or resume, drop, or query the pipes you have created and retry to load specific data files. For more information, see [ALTER PIPE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/ALTER_PIPE.md), [SUSPEND or RESUME PIPE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/SUSPEND_or_RESUME_PIPE.md), [DROP PIPE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/DROP_PIPE.md), [SHOW PIPES](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/SHOW_PIPES.md), and [RETRY FILE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/RETRY_FILE.md). --- ### Sink data from RisingWave to StarRocks RisingWave is a distributed SQL streaming database that enables simple, efficient, and reliable processing of streaming data. To quickly get started with RisingWave, see [Get started](https://docs.risingwave.com/get-started/quickstart/). RisingWave provides the data sinking feature to enable users to directly sink data to StarRocks without requiring any other third-party components. This feature can work with all StarRocks table types: Duplicate Key, Primary Key, Aggregate, and Unique Key tables. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") * You have a running RisingWave cluster of v1.7 or later. * You can access the target StarRocks table and the StarRocks version is v2.5 or later. * To sink data into a StarRocks table, you must have the SELECT and INSERT privileges on the target table. To grant the privileges, see [GRANT](https://docs.starrocks.io/zh/docs/sql-reference/sql-statements/account-management/GRANT/). tip RisingWave only supports at-least-once semantics for StarRocks Sink, which means that in case of failures, duplicate data may be written. You are recommended to use [StarRocks Primary Key tables](https://docs.starrocks.io/zh/docs/table_design/table_types/primary_key_table/), which can deduplicate data and achieve end-to-end idempotent writes. #### Parameters[​](#parameters "Direct link to Parameters") The following table describes the parameters you need to configure when you sink data from RisingWave to StarRocks. All parameters are required unless otherwise specified. | Parameters | Description | | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | connector | Set it to `starrocks`. | | starrocks.host | The IP address of the StarRocks FE node. | | starrocks.query\_port | The query port the FE node. | | starrocks.http\_port | The HTTP port of the FE node. | | starrocks.user | The username used to access the StarRocks cluster. | | starrocks.password | The password associated with the username. | | starrocks.database | The StarRocks database where the target table is located. | | starrocks.table | The StarRocks table to which you want to sink data. | | starrocks.partial\_update | (Optional) Whether to enable the StarRocks partial update feature. Enabling this feature can increase the Sink performance when only a few columns need to be updated. | | type | The data operation type during sink.- `append-only`: Performs only INSERT operations.
- `upsert`: Performs Upsert operations. If this setting is used, the StarRocks target table must be a Primary Key table. | | force\_append\_only | (Optional) When `type` is set to `append-only` but there are also Upsert and Delete operations in the sink process, this setting can force the Sink task to generate append-only data and discard Upsert and Delete data. | | primary\_key | (Optional) The primary key of the StarRocks table. Required if `type` is `upsert`. | #### Data type mapping[​](#data-type-mapping "Direct link to Data type mapping") The following table provides the data type mapping between RisingWave and StarRocks. | RisingWave | StarRocks | | ----------------------------------------------------------------------------- | ------------- | | BOOLEAN | BOOLEAN | | SMALLINT | SMALLINT | | INTEGER | INT | | BIGINT | BIGINT | | REAL | FLOAT | | DOUBLE | DOUBLE | | DECIMAL | DECIMAL | | DATE | DATE | | VARCHAR | VARCHAR | | TIME
(Cast to VARCHAR before sinking to StarRocks) | Not supported | | TIMESTAMP | DATETIME | | TIMESTAMP WITH TIME ZONE
(Cast to TIMESTAMP before sinking to StarRocks) | Not supported | | INTERVAL
(Cast to VARCHAR before sinking to StarRocks) | Not supported | | STRUCT | JSON | | ARRAY | ARRAY | | BYTEA
(Cast to VARCHAR before sinking to StarRocks) | Not supported | | JSONB | JSON | | SERIAL | BIGINT | #### Examples[​](#examples "Direct link to Examples") 1. Create a database `demo` in StarRocks and create a Primary Key table `score_board` in this database. ```sql CREATE DATABASE demo; USE demo; CREATE TABLE demo.score_board( id int(11) NOT NULL COMMENT "", name varchar(65533) NULL DEFAULT "" COMMENT "", score int(11) NOT NULL DEFAULT "0" COMMENT "" ) PRIMARY KEY(id) DISTRIBUTED BY HASH(id); ``` 2. Sink data from RisingWave to StarRocks. ```sql -- Create a table in RisingWave. CREATE TABLE score_board ( id INT PRIMARY KEY, name VARCHAR, score INT ); -- Insert data into the table. INSERT INTO score_board VALUES (1, 'starrocks', 100), (2, 'risingwave', 100); -- Sink data from this table to the StarRocks table. CREATE SINK score_board_sink FROM score_board WITH ( connector = 'starrocks', type = 'upsert', starrocks.host = 'starrocks-fe', starrocks.mysqlport = '9030', starrocks.httpport = '8030', starrocks.user = 'users', starrocks.password = '123456', starrocks.database = 'demo', starrocks.table = 'score_board', primary_key = 'id' ); ``` --- ### Superset Support [Apache Superset](https://superset.apache.org) is a modern data exploration and visualization platform. It uses [SQLAlchemy](https://github.com/StarRocks/starrocks/tree/main/contrib/starrocks-python-client/starrocks) to query data. Although Mysql Dialect can be used, it does not support `largeint`. So we developed [StarRocks Dialect](https://github.com/StarRocks/starrocks/tree/main/contrib/starrocks-python-client/starrocks/). #### Environment[​](#environment "Direct link to Environment") * Python 3.x * mysqlclient (pip install mysqlclient) * [Apache Superset](https://superset.apache.org) Notice: If `mysqlclient` is not installed, an exception will be thrown: ```plain No module named 'MySQLdb' ``` #### Installation[​](#installation "Direct link to Installation") Since `dialect` does not contribute to `SQLAlchemy`, it needs to be installed from the source code. If you install `superset` with Docker, install `sqlalchemy-starrocks` with `root`. Install from [Source Code](https://github.com/StarRocks/starrocks/tree/main/contrib/starrocks-python-client/starrocks) ```shell pip install . ``` Uninstall ```shell pip uninstall sqlalchemy-starrocks ``` #### Usage[​](#usage "Direct link to Usage") To connect to StarRocks with SQLAlchemy, the following URL pattern can be used: ```shell starrocks://:@:/[?charset=utf8] ``` #### Basic Example[​](#basic-example "Direct link to Basic Example") ##### Sqlalchemy Example[​](#sqlalchemy-example "Direct link to Sqlalchemy Example") It is recommended to use python 3.x to connect to the StarRocks database, eg: ```python from sqlalchemy import create_engine import pandas as pd conn = create_engine('starrocks://root:@x.x.x.x:9030/superset_db?charset=utf8') sql = """select * from xxx""" df = pd.read_sql(sql, conn) ``` ##### Superset Example[​](#superset-example "Direct link to Superset Example") In superset, use `Other` database, and set url as: ```shell starrocks://root:@x.x.x.x:9030/superset_db?charset=utf8 ``` --- ## Introduction ### Architecture StarRocks has a simple architecture. The entire system consists of only two types of components; frontends and backends. The frontend nodes are called **FE**s. There are two types of backend nodes, **BE**s, and **CN**s (Compute Nodes). BEs are deployed when local storage for data is used, and CNs are deployed when data is stored on object storage or HDFS. StarRocks does not rely on any external components, simplifying deployment and maintenance. Nodes can be horizontally scaled without service downtime. In addition, StarRocks has a replica mechanism for metadata and service data, which increases data reliability and efficiently prevents single points of failure (SPOFs). StarRocks is compatible with MySQL protocols and supports standard SQL. Users can easily connect to StarRocks from MySQL clients to gain instant and valuable insights. #### Architecture choices[​](#architecture-choices "Direct link to Architecture choices") StarRocks supports shared-nothing (Each BE has a portion of the data on its local storage) and shared-data (all data on object storage or HDFS and each CN has only cache on local storage). You can decide where the data is stored based on your needs. ![Architecture choices](/assets/images/architecture_choices-ddd6ad78779f16a6691ec53a1f86ef06.png) ##### Shared-nothing[​](#shared-nothing "Direct link to Shared-nothing") Local storage provides improved query latency for real-time queries. As a typical massively parallel processing (MPP) database StarRocks supports the shared-nothing architecture. In this architecture, BEs are responsible for both data storage and computation. Direct access to local data on the BE mode allows for local computation, avoiding data transfer and data copying, and providing ultra-fast query and analytics performance. This architecture supports multi-replica data storage, enhancing the cluster's ability to handle high concurrency queries and ensuring data reliability. It is well-suited for scenarios that pursue optimal query performance. ![shared-data-arch](/assets/images/shared-nothing-f5cbe29195b0d7e9c66069d356208cd2.png) ###### Nodes[​](#nodes "Direct link to Nodes") In the shared-nothing architecture, StarRocks consists of two types of nodes: FEs and BEs. * FEs are responsible for metadata management and constructing execution plans. * BEs execute query plans and store data. BEs utilize local storage to accelerate queries and the multi-replica mechanism to ensure high data availability. ###### FE[​](#fe "Direct link to FE") FEs are responsible for metadata management, client connection management, query planning, and query scheduling. Each FE uses BDB JE (Berkeley DB Java Edition) to store and maintain a complete copy of the metadata in its memory, ensuring consistent services across all FEs. FEs can work as the leader, followers, and observers. If the leader node crashes, with followers electing a leader based on the Raft protocol. | **FE Role** | **Metadata management** | **Leader election** | | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Leader | The leader FE reads and writes metadata. Follower and observer FEs can only read metadata. They route metadata write requests to the leader FE. The leader FE updates the metadata and then uses the Raft protocol to synchronize the metadata changes to the follower and observer FEs. Data writes are considered successful only after the metadata changes are synchronized to more than half of the follower FEs. | The leader FE, technically speaking, is also a follower node and is elected from follower FEs. To perform leader election, more than half of the follower FEs in the cluster must be active. When the leader FE fails, follower FEs will start another round of leader election. | | Follower | Followers can only read metadata. They synchronize and replay logs from the leader FE to update metadata. | Followers participate in leader election, which requires more than half of the followers in the cluster be active. | | Observer | Observers synchronize and replay logs from the leader FE to update metadata. | Observers are mainly used to increase the query concurrency of the cluster. Observers do not participate in leader election and therefore, will not add leader selection pressure to the cluster. | ###### BE[​](#be "Direct link to BE") BEs are responsible for data storage and SQL execution. * Data storage: BEs have equivalent data storage capabilities. FEs distribute data to BEs based on predefined rules. BEs transform the ingested data, write the data into the required format, and generate indexes for the data. * SQL execution: FEs parse each SQL query into a logical execution plan according to the semantics of the query, and then transform the logical plan into physical execution plans that can be executed on BEs. BEs that store the destination data execute the query. This eliminates the need for data transmission and copy, achieving high query performance. ##### Shared-data[​](#shared-data "Direct link to Shared-data") Object storage and HDFS provide cost, reliability, and scalability benefits. In addition to the scalability of storage, CN nodes can be added and removed without the need to rebalance data since storage and compute are separate. In the shared-data architecture, BEs are replaced with "compute nodes (CNs)" which are responsible only for data compute tasks and caching hot data. Data is stored in low-cost and reliable remote storage systems such as Amazon S3, Google Cloud Storage, Azure Blob Storage, MinIO, etc. When the cache is hit, query performance is comparable to that of the shared-nothing architecture. CN nodes can be added or removed on demand within seconds. This architecture reduces storage cost, ensures better resource isolation, and high elasticity and scalability. The shared-data architecture maintains as simple an architecture as its shared-nothing counterpart. It consists of only two types of nodes: FE and CN. The only difference is users have to provision backend object storage. ![shared-data-arch](/assets/images/shared-data-1fac1b5ab7d46bf34f67c93ecc8e6c28.png) ###### Nodes[​](#nodes-1 "Direct link to Nodes") FEs in the shared-data architecture provide the same functions as in the shared-nothing architecture. BEs are replaced with CNs (Compute Nodes), and the storage function is offloaded to object storage or HDFS. CNs are stateless compute nodes that perform all the functions of BEs, except for the storage of data. ###### Storage[​](#storage "Direct link to Storage") StarRocks shared-data clusters support two storage solutions: object storage (for example, AWS S3, Google GCS, Azure Blob Storage, or MinIO) and HDFS. In a shared-data cluster, the data file format remains consistent with that of a shared-nothing cluster (featuring coupled storage and compute). Data is organized into segment files, and various indexing technologies are reused in cloud-native tables, which are tables used specifically in shared-data clusters. ###### Cache[​](#cache "Direct link to Cache") StarRocks shared-data clusters decouple data storage and computation, allowing each to scale independently, thereby reducing costs and enhancing elasticity. However, this architecture can affect query performance. To mitigate the impact, StarRocks establishes a multi-tiered data access system encompassing memory, local disk, and remote storage to better meet various business needs. Queries against hot data scan the cache directly and then the local disk, while cold data needs to be loaded from the object storage into the local cache to accelerate subsequent queries. By keeping hot data close to compute units, StarRocks achieves truly high-performance computation and cost-effective storage. Moreover, access to cold data has been optimized with data prefetch strategies, effectively eliminating performance limits for queries. Caching can be enabled when creating tables. If caching is enabled, data will be written to both the local disk and backend object storage. During queries, the CN nodes first read data from the local disk. If the data is not found, it will be retrieved from the backend object storage and simultaneously cached on the local disk. #### Learn by doing[​](#learn-by-doing "Direct link to Learn by doing") Try the [Quick Starts](https://docs.starrocks.io/docs/quick_start.md) to get an overview of using StarRocks with realistic scenarios. --- ### Feature Differences between Shared-nothing and Shared-data Clusters This topic lists the differences of features supported in shared-nothing and shared-data clusters. #### Real-time Analytics and Primary Key Tables[​](#real-time-analytics-and-primary-key-tables "Direct link to Real-time Analytics and Primary Key Tables") | Feature | Shared-nothing | Shared-data | | ---------------------------------------------------- | -------------- | --------------- | | Conditional Update | v2.5+ | v3.1+ | | Partial Update Column mode | v3.1 | To be supported | | Partial Update Row mode | v2.3+ | v3.1+ | | Partial update with Conditional Update | v3.1+ | v3.4.1+ | | Primary Key Index Persistence | v2.3+ | v3.2+ | | Decoupled ORDER BY columns from Primary Keys | v3.0+ | v3.1+ | | Rich UPDATE and DELETE syntax for Primary Key tables | v3.0+ | v3.1+ | #### Storage Engine[​](#storage-engine "Direct link to Storage Engine") | Feature | Shared-nothing | Shared-data | | ----------------------------- | -------------- | --------------- | | Fast Schema Evolution | v3.2+ | v4.0+ | | File Bundling | Not applicable | v4.0+ | | Inverted Index | v3.3+ | v4.1+ | | Manual Compaction | v3.1+ | v3.3+ | | Row Store | v3.2+ | To be supported | | Adding/dropping STRUCT fields | v3.3.2+ | v3.3.5+ | | Vector index | v3.4+ | To be supported | #### Data Distribution[​](#data-distribution "Direct link to Data Distribution") | Feature | Shared-nothing | Shared-data | | --------------------------------------------------- | -------------- | -------------- | | Expression Partitioning | v3.0+ | v3.1.1+ | | List Partitioning | v3.1+ | v3.1.1+ | | Random Bucketing | v3.1+ | v3.2+ | | Random Bucketing Optimization (with Sub-partition) | v3.2+ | v3.2+ | | Unified Syntax for Multi-level Expression Partition | v3.4+ | v3.4+ | | Optimize Table: Change Bucketing | v3.2+ | v3.3+ | | Optimize Table: Change Bucketing Online | v3.3.3+ | Not applicable | | Range-based Distribution | v4.1+ | v4.1+ | #### Query Performance[​](#query-performance "Direct link to Query Performance") | Feature | Shared-nothing | Shared-data | | ---------------------------------------- | -------------- | ----------- | | Cross-node Data Cache Sharing | Not applicable | v3.5.1+ | | Generated Column | v3.1+ | v3.5+ | | Query Cache | v2.5+ | v3.4+ | | Rollup and Synchronous Materialized view | v1.x+ | v3.3+ | #### Disaster Recovery[​](#disaster-recovery "Direct link to Disaster Recovery") | Feature | Shared-nothing | Shared-data | | --------------------------------- | -------------- | -------------- | | Backup & Restore | v1.x+ | Not applicable | | BE/CN Blacklist | v3.3+ | v4.0+ | | Cross-cluster Data Migration Tool | v3.3+ | Not applicable | | Cluster Snapshot | Not applicable | v3.4.2+ | #### Security and Authentication[​](#security-and-authentication "Direct link to Security and Authentication") | Feature | Shared-nothing | Shared-data | | ------------------------ | -------------- | ----------- | | Kerberos Support on HDFS | Not applicable | v3.2+ | --- ### Database Features StarRocks offers a rich set of features to deliver a blazing-fast, real-time analytics experience on data at scale. #### MPP framework[​](#mpp-framework "Direct link to MPP framework") StarRocks adopts the massively parallel processing (MPP) framework. One query request is split into multiple physical computing units that can be executed in parallel on multiple machines. Each machine has dedicated CPU and memory resources. The MPP framework fully uses the resources of all CPU cores and machines. The performance of a single query can continuously increase as the cluster is scaled out. ![MPP](/assets/images/1.1-3-mpp-243454cd702a6fac41bbc619a80e0c90.png) In the preceding figure, StarRocks parses an SQL statement into multiple logical execution units (query fragments) according to the semantics of the statement. Each fragment is then implemented by one or multiple physical execution units (fragment instances) based on the computing complexity. A physical execution unit is the smallest scheduling unit in StarRocks. They will be scheduled to backends (BEs) for execution. One logical execution unit can contain one or more operators, such as the Scan, Project, and Agg operators, as shown on the right side of the figure. Each physical execution unit processes only part of the data and the result will be merged to generate the final data. **Parallel execution of logical execution units fully utilizes the resources of all CPU cores and physical machines and accelerates the query speed.** ![MPP](/assets/images/1.1-4-mpp-6e666188e05796fccef4a7f92e97d6c7.png) Unlike the Scatter-Gather framework used by many other data analytics systems, the MPP framework can utilize more resources to process query requests. In the Scatter-Gather framework, only the Gather node can perform the final merge operation. In the MPP framework, data is shuffled to multiple nodes for merge operations. For complex queries, such as Group By on high-cardinality fields and large table joins, StarRocks' MPP framework has noticeable performance advantages over the Scatter-Gather framework. #### Fully vectorized execution engine[​](#fully-vectorized-execution-engine "Direct link to Fully vectorized execution engine") The fully vectorized execution engine makes more efficient use of CPU processing power because this engine organizes and processes data in a columnar manner. Specifically, StarRocks stores data, organizes data in memory, and computes SQL operators all in a columnar manner. Columnar organization makes full use of CPU cache. Columnar computing reduces the number of virtual function calls and branch judgments, resulting in more sufficient CPU instruction flows. The vectorized execution engine also makes full use of SIMD instructions. This engine can complete more data operations with fewer instructions. Tests against standard datasets show that this engine enhances the overall performance of operators by 3 to 10 times. In addition to operator vectorization, StarRocks has implemented other optimizations for the query engine. For example, StarRocks uses the Operation on Encoded Data technology to directly execute operators on encoded strings, without the need for decoding. This noticeably reduces SQL complexity and increases the query speed by more than 2 times. #### Separation of storage and compute[​](#separation-of-storage-and-compute "Direct link to Separation of storage and compute") The [storage-compute separation architecture](https://docs.starrocks.io/docs/introduction/Architecture.md) was introduced from 3.0. In this architecture, computing and storage are decoupled to achieve resource isolation, elastic scaling of compute nodes, and high-performance queries. Storage-compute separation equips StarRocks with better flexibility, higher performance and data availability, and lower cost. ![shared-data](/assets/images/share_data_arch-2aee703af4e8afcf46e5bf9866cd2f2a.png) In storage-compute separation mode, computing and storage are decoupled and can be scaled independently, which eliminates the cost that long exists in the storage-compute coupled mode where storage has to be scaled anytime users want to add computing nodes. In addition, computing can dynamically scale within seconds, improving resource utilization, especially when there are noticeable traffic peaks and valleys. The storage layer leverages the nearly unlimited capacity and high reliability of object storage to achieve massive data storage and data persistence. StarRocks can work with various object storage systems such as AWS S3, Google Cloud Storage, Azure Blob Storage, HDFS, and other S3-compatible storage like MinIO. Users can choose to deploy StarRocks in public clouds, private clouds, or on-premises data centers. StarRocks supports Kubernetes-based deployments and provides an Operator for automated deployment of storage-compute decoupled clusters. StarRocks in storage-compute separation mode provides the same functionalities as the storage-compute coupled mode. The data write and hot data query performance are also the same. Users can perform data updates, data lake analytics, and materialized view acceleration as they do in storage-compute coupled mode. #### Cost-based optimizer[​](#cost-based-optimizer "Direct link to Cost-based optimizer") ![CBO](/assets/images/1.1-5-cbo-3baa9959e4ca41775a33d67400e8c480.png) Performance of multi-table join queries is difficult to optimize. Execution engines alone cannot deliver superior performance because the complexity of execution plans may vary by several orders of magnitude in multi-table join query scenarios. The more the associated tables, the more the execution plans, which makes it NP-hard to choose an optimal plan. Only a query optimizer excellent enough can choose a relatively optimal query plan for efficient multi-table analytics. StarRocks designs a brand-new [CBO](https://docs.starrocks.io/docs/using_starrocks/Cost_based_optimizer.md) from scratch. This CBO adopts the cascades-like framework and is deeply customized for the vectorized execution engine with a number of optimizations and innovations. These optimizations include the reuse of common table expressions (CTEs), rewriting of subqueries, Lateral Join, Join Reorder, strategy selection for distributed Join execution, and low-cardinality optimization. The CBO supports a total of 99 TPC-DS SQL statements. The CBO enables StarRocks to deliver better multi-table join query performance than competitors, especially in complex multi-table join queries. #### Real-time, updatable columnar storage engine[​](#real-time-updatable-columnar-storage-engine "Direct link to Real-time, updatable columnar storage engine") StarRocks is a columnar storage engine that allows data of the same type to be stored continuously. In columnar storage, data can be encoded in a more efficient way, increasing the compression ratio and lowering the storage cost. Columnar storage also reduces the total data read I/Os, improving query performance. In addition, in most OLAP scenarios, only specific columns are queried. Columnar storage enables users to query only part of the columns, significantly reducing disk I/Os. StarRocks can load data within seconds for near-real-time analytics. StarRocks' storage engine guarantees the atomicity, consistency, isolation, and durability (ACID) of each data ingestion operation. For a data loading transaction, the entire transaction either succeeds or fails. Concurrent transactions do not affect each other, providing transaction-level isolation. ![Realtime](/assets/images/1.1-6-realtime-81890333b24620f95ea35397d23af561.png) StarRocks' storage engine uses the Delete-and-insert pattern, which allows for efficient Partial Update and Upsert operations. The storage engine can quickly filter data using primary key indexes, eliminating the need for Sort and Merge operations at data reading. The engine can also make full use of secondary indexes. It delivers fast and predictable query performance even on huge volume of data updates. #### Intelligent materialized view[​](#intelligent-materialized-view "Direct link to Intelligent materialized view") StarRocks uses intelligent [materialized views](https://docs.starrocks.io/docs/using_starrocks/async_mv/Materialized_view.md) to accelerate queries and data warehouse layering. Different from materialized views of other similar products that requires manual data synchronization with the base table, StarRocks' materialized views automatically update data according to the data changes in the base table without requiring additional maintenance operations. In addition, the selection of materialized views is also automatic. If StarRocks identifies a suitable materialized view (MV) to improve query performance, it will automatically rewrite the query to utilize the MV. This intelligent process significantly enhances query efficiency without requiring manual intervention. StarRocks' MV can replace the traditional ETL data modeling process: Instead of transforming data in the upstream applications, you now have the option to transform data with MV within StarRocks, simplifying the data processing pipeline. For example, in the figure, raw data on data lake can be used to create a normalized table based on an external MV. A denormalized table can be created from normalized tables through asynchronous materialized views. Another MV can be created from normalized tables to support high concurrency queries and better query performance. ![MV](/assets/images/1.1-7-mv-7fb0ae6f680ae698965d29b4a7a6911d.png) #### Data lake analytics[​](#data-lake-analytics "Direct link to Data lake analytics") ![DLA](/assets/images/1.1-8-dla-c67d601d709b092317fa11eb64ac1783.png) In addition to efficient analytics of local data, StarRocks can work as the compute engine to analyze data stored in [data lakes](https://docs.starrocks.io/docs/data_source/catalog/catalog_overview.md) such as Apache Hive, Apache Iceberg, Apache Hudi, and Delta Lake. One of the key features of StarRocks is its external catalog, which acts as a linkage to an externally maintained metastore. This functionality provides users with the capability to query external data sources seamlessly, eliminating the need for data migration. As such, users can analyze data from different systems such as HDFS and Amazon S3, in various file formats such as Parquet, ORC, and CSV, etc. The preceding figure shows a data lake analytics scenario where StarRocks is responsible for data computing and analysis, and the data lake is responsible for data storage, organization, and maintenance. Data lakes allow users to store data in open storage formats and use flexible schemas to produce reports on "single source of truth" for various BI, AI, ad-hoc, and reporting use cases. StarRocks fully leverages the advantages of its vectorization engine and CBO, significantly improving the performance of data lake analytics. --- ### Beta and experimental features StarRocks features have five potential maturity levels: * Experimental * Beta * Generally Available (GA) * Deprecated * Removed Most of the StarRocks features are GA, and if there is no label in the documentation to indicate that a feature is Experimental, Beta, or Deprecated—then the feature is GA. #### Experimental features[​](#experimental-features "Direct link to Experimental features") * **Stability**: Possibly buggy, with minor known issues. * **Maturity**: Low * **Interface**: The interface may be changed in the future. This includes command syntax, configuration parameters, defaults, feature removal, etc. * **Availability**: Experimental features are off by default, and need to be allowed by setting a parameter with SQL or in a configuration file. * **Production readiness**: Experimental features should not be used in production. * **Support**: Please open a [GitHub issue](https://github.com/StarRocks/starrocks/issues) or ask questions in [Slack](https://docs.starrocks.io/join/) and the StarRocks Engineering team will try to help you. #### Beta features[​](#beta-features "Direct link to Beta features") * **Stability**: Well tested. May be not good for corner cases. * **Maturity**: Core functionality is complete, may not be performance-optimized. * **Interface**: The interface may be changed in the future. May be not backward compatible. * **Availability**: Beta features are off by default, and need to be allowed by setting a parameter with SQL or in a configuration file. * **Production readiness**: Beta features are not recommended for production use. * **Support**: Please open a [GitHub issue](https://github.com/StarRocks/starrocks/issues) or ask questions in [Slack](https://docs.starrocks.io/join/) and the StarRocks Engineering team will try to help you. #### GA features[​](#ga-features "Direct link to GA features") * **Stability**: Comprehensively tested. * **Maturity**: High. * **Interface**: Stable API. * **Availability**: GA features are on by default. * **Production readiness**: Production ready. * **Support**: The support team provides support to customers. Open-source community members should open a [GitHub issue](https://github.com/StarRocks/starrocks/issues) or ask questions in [Slack](https://docs.starrocks.io/join/) and the StarRocks Engineering team will try to help you. #### Deprecated features[​](#deprecated-features "Direct link to Deprecated features") Some features are deprecated—marked for removal, because they are replaced with other features or the features were not being used. Generally when a feature is deprecated we will suggest an alternative in the documentation. --- ### StarRocks StarRocks is a next-gen, high-performance analytical data warehouse that enables real-time, multi-dimensional, and highly concurrent data analysis. StarRocks has an MPP architecture and is equipped with a fully vectorized execution engine, a columnar storage engine that supports real-time updates, and is powered by a rich set of features including a fully-customized cost-based optimizer (CBO), intelligent materialized view and more. StarRocks supports real-time and batch data ingestion from a variety of data sources. It also allows you to directly analyze data stored in data lakes with zero data migration. StarRocks is also compatible with MySQL protocols and can be easily connected using MySQL clients and popular BI tools. StarRocks is highly scalable, available, and easy to maintain. It is widely adopted in the industry, powering a variety of OLAP scenarios, such as real-time analytics, ad-hoc queries, data lake analytics and more. [StarRocks](https://github.com/StarRocks/starrocks/tree/main) is licensed under Apache 2.0, available at the StarRocks GitHub repository (see the [StarRocks license](https://github.com/StarRocks/starrocks/blob/main/LICENSE.txt)). StarRocks (i) links to or calls functions from third party software libraries, the licenses of which are available in the folder [licenses-binary](https://github.com/StarRocks/starrocks/tree/main/licenses-binary); and (ii) incorporates third party software code, the licenses of which are available in the folder [licenses](https://github.com/StarRocks/starrocks/tree/main/licenses). Join our [Slack channel](https://docs.starrocks.com/join/) for asking general questions and for chat. For community news, read the [StarRocks.io Blog](https://www.starrocks.io/blog). You can also follow us on [LinkedIn](https://www.linkedin.com/company/starrocks) to get first-hand updates on new features, events, and sharing. *** ##### Popular topics[​](#popular-topics "Direct link to Popular topics") ##### [Introduction](https://docs.starrocks.io/docs/introduction) [OLAP, features, architecture](https://docs.starrocks.io/docs/introduction) ##### [Quick Start](https://docs.starrocks.io/docs/quick_start.md) [Get up and running quickly.](https://docs.starrocks.io/docs/quick_start.md) ##### [Data Loading](https://docs.starrocks.io/docs/loading/Loading_intro.md) [Clean, transform, and load](https://docs.starrocks.io/docs/loading/Loading_intro.md) ##### [Table Design](https://docs.starrocks.io/docs/table_design/StarRocks_table_design.md) [Tables, indexing, acceleration](https://docs.starrocks.io/docs/table_design/StarRocks_table_design.md) ##### [Data Lakes](https://docs.starrocks.io/docs/data_source/data_lakes.md) [Iceberg, Hive, Delta Lake, …](https://docs.starrocks.io/docs/data_source/data_lakes.md) ##### [Work with semi-structured data](https://docs.starrocks.io/docs/category/semi-structured) [JSON, map, struct, array](https://docs.starrocks.io/docs/category/semi-structured) ##### [Integrations](https://docs.starrocks.io/docs/integrations) [BI tools, IDEs, Cloud authentication, …](https://docs.starrocks.io/docs/integrations) ##### [Administration](https://docs.starrocks.io/docs/administration) [Scale, backups, roles and privileges, …](https://docs.starrocks.io/docs/administration) ##### [Reference](https://docs.starrocks.io/docs/category/reference) [SQL, functions, error codes, …](https://docs.starrocks.io/docs/category/reference) ##### [FAQs](https://docs.starrocks.io/docs/faq) [Frequently asked questions.](https://docs.starrocks.io/docs/faq) ##### [Benchmarks](https://docs.starrocks.io/docs/benchmarking) [DB performance comparison benchmarks.](https://docs.starrocks.io/docs/benchmarking) *** --- ### Version naming Purpose: To provide a unified and clear explanation of the current naming conventions for StarRocks software packages. #### Numbering[​](#numbering "Direct link to Numbering") The version numbering format for released versions is in the form of `MAJOR.MINOR.PATCH[-PRERELEASE]`, where PRERELEASE indicates the type and number of the prerelease version. Detailed explanations are as follows: ##### MAJOR[​](#major "Direct link to MAJOR") The major version is incremented when incompatible API changes are made or there are major adjustments to the product's overall strategy. ##### MINOR[​](#minor "Direct link to MINOR") The minor version is incremented when functionality is added in a backward-compatible manner. This typically refers to the addition of new features or improvements that do not break the API of the current major version or the main behavioral patterns of the software. ##### PATCH[​](#patch "Direct link to PATCH") The patch version is incremented when backward-compatible bug fixes are made. This is usually for bug fixes or minor improvements that do not affect the software's main functions or API. ##### PRERELEASE[​](#prerelease "Direct link to PRERELEASE") Adding a prerelease identifier to the version indicates that the build is for testing by early adopters. The prerelease identifier will indicate the version number of the prerelease, such as `rc01`, `rc02`, etc. Currently, the only prerelease tag in use is `rc` followed by a two-digit number such as `rc01`. note Release Candidates are only built for the initial minor and major versions. For example, `3.4.0-rc01` or `4.0.0-rc01`. #### Version examples[​](#version-examples "Direct link to Version examples") * `3.4.0`: The official release version. * `3.4.0-rc01`: The first prerelease version of `3.4.0`. For instance, version `3.3` also had a second prerelease version `3.3.0-rc02`. * `3.4.1`, `3.4.2`: Bug fix versions. #### Software package naming rules[​](#software-package-naming-rules "Direct link to Software package naming rules") The naming convention for software packages is: `StarRocks-x.y.z[-rcxx]{-os}{-arch}.tar.gz`. (That is, the main form is `Product-version-os-arch`) 1. x.y.z: follows the three-digit version numbering rule. `-rcxx` is an optional prerelease version number, such as `-rc01`, `-rc02`. 2. OS includes: `ubuntu`, `centos`. Mandatory. 3. ARCH includes: `amd64`. Mandatory. (There is no community edition for `arm64` yet.) ###### Example software package names[​](#example-software-package-names "Direct link to Example software package names") 1. `StarRocks-3.4.0-rc01-ubuntu-amd64.tar.gz`: The prerelease version of 3.4.0, built for Ubuntu Linux. 2. `StarRocks-3.4.1-centos-amd64.tar.gz`: The PATCH revision built for Red Hat Enterprise Linux / CentOS. --- ### What is StarRocks? StarRocks is a next-generation, blazing-fast massively parallel processing (MPP) database designed to make real-time analytics easy for enterprises. It is built to power sub-second queries at scale. StarRocks has an elegant design. It encompasses a rich set of features including fully vectorized engine, newly designed cost-based optimizer (CBO), and intelligent materialized view. As such, StarRocks can deliver a query speed far exceeding database products of its kind, especially for multi-table joins. StarRocks is ideal for real-time analytics on fresh data. Data can be ingested at a high speed and updated and deleted in real time. StarRocks empowers users to create tables that use various schemas, such as flat, star, and snowflake schemas. Compatible with MySQL protocols and standard SQL, StarRocks has out-of-the-box support for all major Business Intelligence (BI) tools, such as Tableau and Power BI. StarRocks does not rely on any external components. It is an integrated data analytics platform that allows for high scalability, high availability, and simplified management and maintenance. [StarRocks](https://github.com/StarRocks/starrocks/tree/main) is licensed under Apache 2.0, available at the StarRocks GitHub repository (see the [StarRocks license](https://github.com/StarRocks/starrocks/blob/main/LICENSE.txt)). StarRocks (i) links to or calls functions from third party software libraries, the licenses of which are available in the folder [licenses-binary](https://github.com/StarRocks/starrocks/tree/main/licenses-binary); and (ii) incorporates third party software code, the licenses of which are available in the folder [licenses](https://github.com/StarRocks/starrocks/tree/main/licenses). #### Scenarios[​](#scenarios "Direct link to Scenarios") StarRocks meets varied enterprise analytics requirements, including OLAP (Online Analytical Processing) multi-dimensional analytics, real-time analytics, high concurrency analytics, customized reporting, ad-hoc queries, and unified analytics. ##### OLAP multi-dimensional analytics[​](#olap-multi-dimensional-analytics "Direct link to OLAP multi-dimensional analytics") The MPP framework and vectorized execution engine enable users to choose between various schemas to develop multi-dimensional analytical reports. Scenarios: * User behavior analysis * User profiling, label analysis, user tagging * High-dimensional metrics report * Self-service dashboard * Service anomaly probing and analysis * Cross-theme analysis * Financial data analysis * System monitoring analysis ##### Real-time analytics[​](#real-time-analytics "Direct link to Real-time analytics") StarRocks uses the Primary Key table to implement real-time updates. Data changes in a TP (Transaction Processing) database can be synchronized to StarRocks in a matter of seconds to build a real-time warehouse. Scenarios: * Online promotion analysis * Logistics tracking and analysis * Performance analysis and metrics computation for the financial industry * Quality analysis for livestreaming * Ad placement analysis * Cockpit management * Application Performance Management (APM) ##### High concurrency analytics[​](#high-concurrency-analytics "Direct link to High concurrency analytics") StarRocks leverages performant data distribution, flexible indexing, and intelligent materialized views to facilitate user-facing analytics at high concurrency: * Advertiser report analysis * Channel analysis for the retail industry * User-facing analysis for SaaS * Multi-tabbed dashboard analysis ##### Unified analytics[​](#unified-analytics "Direct link to Unified analytics") StarRocks provides a unified data analytics experience. * One system can power various analytical scenarios, reducing system complexity and lowering Total Cost of Ownership (TCO). * StarRocks unifies data lakes and data warehouses. Data in a lakehouse can be managed all in StarRocks. Latency-sensitive queries that require high concurrency can run on StarRocks. Data in data lakes can be accessed by using external catalogs or external tables provided by StarRocks. --- ## Kafka_connector ### Releases of StarRocks Connector for Kafka #### Notifications[​](#notifications "Direct link to Notifications") **User guide:** [Load data using Kafka connector](https://docs.starrocks.io/docs/loading/Kafka-connector-starrocks/) **Source codes:** [starrocks-connector-for-kafka](https://github.com/StarRocks/starrocks-connector-for-kafka) **Naming format of the compressed file:** `starrocks-kafka-connector-${connector_version}.tar.gz` **Download link of the compressed file:** [starrocks-kafka-connector](https://github.com/StarRocks/starrocks-connector-for-kafka/releases) **Version requirements:** | Kafka Connector | StarRocks | Java | | --------------- | ------------- | ---- | | 1.0.3 | 2.1 and later | 8 | | 1.0.2 | 2.1 and later | 8 | | 1.0.1 | 2.1 and later | 8 | | 1.0.0 | 2.1 and later | 8 | #### Release notes[​](#release-notes "Direct link to Release notes") ##### 1.0[​](#10 "Direct link to 1.0") ###### 1.0.3[​](#103 "Direct link to 1.0.3") Release date: December 19, 2023 **Features** Added Apache License as the open-source software license. [#9](https://github.com/StarRocks/starrocks-connector-for-kafka/pull/9) ###### 1.0.2[​](#102 "Direct link to 1.0.2") Release date: December 14, 2023 **Features** Supports the source data to be of DECIMAL type. [#7](https://github.com/StarRocks/starrocks-connector-for-kafka/pull/7) ###### 1.0.1[​](#101 "Direct link to 1.0.1") Release date: November 28, 2023 **Features** * Supports loading Debezium data into Primary Key tables. [#4](https://github.com/StarRocks/starrocks-connector-for-kafka/pull/4) * Supports parsing JSON data without schema registry. [#6](https://github.com/StarRocks/starrocks-connector-for-kafka/pull/6) ###### 1.0.0[​](#100 "Direct link to 1.0.0") Release date: June 25, 2023 **Features** * Supports loading CSV, JSON, Avro, and Protobuf data. * Supports loading data from a self-managed Apache Kafka cluster or Confluent cloud. --- ## Loading ### alibaba --- ### AutoMQ Kafka [AutoMQ for Kafka](https://docs.automq.com/automq/what-is-automq/overview) is a cloud-native version of Kafka redesigned for cloud environments. AutoMQ Kafka is [open source](https://github.com/AutoMQ/automq-for-kafka) and fully compatible with the Kafka protocol, fully leveraging cloud benefits. Compared to self-managed Apache Kafka, AutoMQ Kafka, with its cloud-native architecture, offers features like capacity auto scaling, self-balancing of network traffic, move partition in seconds. These features contribute to a significantly lower Total Cost of Ownership (TCO) for users. This article will guide you through importing data into AutoMQ Kafka using StarRocks Routine Load. For an understanding of the basic principles of Routine Load, refer to the section on Routine Load Fundamentals. #### Prepare Environment[​](#prepare-environment "Direct link to Prepare Environment") ##### Prepare StarRocks and test data[​](#prepare-starrocks-and-test-data "Direct link to Prepare StarRocks and test data") Ensure you have a running StarRocks cluster. Creating a database and a Primary Key table for testing: ```sql create database automq_db; create table users ( id bigint NOT NULL, name string NOT NULL, timestamp string NULL, status string NULL ) PRIMARY KEY (id) DISTRIBUTED BY HASH(id) PROPERTIES ( "enable_persistent_index" = "true" ); ``` note If a StarRocks cluster in a staging environment contains only one BE, the number of replicas can be set to `1` in the `PROPERTIES` clause, such as `PROPERTIES( "replication_num" = "1" )`. The default number of replicas is 3, which is also the number recommended for production StarRocks clusters. If you want to use the default number, you do not need to configure the `replication_num` parameter. #### Prepare AutoMQ Kafka and test data[​](#prepare-automq-kafka-and-test-data "Direct link to Prepare AutoMQ Kafka and test data") To prepare your AutoMQ Kafka environment and test data, follow the AutoMQ [Quick Start](https://docs.automq.com/automq/what-is-automq/overview) guide to deploy your AutoMQ Kafka cluster. Ensure that StarRocks can directly connect to your AutoMQ Kafka server. To quickly create a topic named `example_topic` in AutoMQ Kafka and write a test JSON data into it, follow these steps: ##### Create a topic[​](#create-a-topic "Direct link to Create a topic") Use Kafka’s command-line tools to create a topic. Ensure you have access to the Kafka environment and the Kafka service is running. Here is the command to create a topic: ```shell ./kafka-topics.sh --create --topic example_topic --bootstrap-server 10.0.96.4:9092 --partitions 1 --replication-factor 1 ``` > Note: Replace `topic` and `bootstrap-server` with your Kafka server address. To check the result of the topic creation, use this command: ```shell ./kafka-topics.sh --describe example_topic --bootstrap-server 10.0.96.4:9092 ``` ##### Generate test data[​](#generate-test-data "Direct link to Generate test data") Generate a simple JSON format test data ```json { "id": 1, "name": "testuser", "timestamp": "2023-11-10T12:00:00", "status": "active" } ``` ##### Write Test Data[​](#write-test-data "Direct link to Write Test Data") Use Kafka’s command-line tools or programming methods to write test data into example\_topic. Here is an example using command-line tools: ```shell echo '{"id": 1, "name": "testuser", "timestamp": "2023-11-10T12:00:00", "status": "active"}' | sh kafka-console-producer.sh --broker-list 10.0.96.4:9092 --topic example_topic ``` > Note: Replace `topic` and `bootstrap-server` with your Kafka server address. To view the recently written topic data, use the following command: ```shell sh kafka-console-consumer.sh --bootstrap-server 10.0.96.4:9092 --topic example_topic --from-beginning ``` #### Creating a Routine Load Task[​](#creating-a-routine-load-task "Direct link to Creating a Routine Load Task") In the StarRocks command line, create a Routine Load task to continuously import data from the AutoMQ Kafka topic: ```sql CREATE ROUTINE LOAD automq_example_load ON users COLUMNS(id, name, timestamp, status) PROPERTIES ( "desired_concurrent_number" = "5", "format" = "json", "jsonpaths" = "[\"$.id\",\"$.name\",\"$.timestamp\",\"$.status\"]" ) FROM KAFKA ( "kafka_broker_list" = "10.0.96.4:9092", "kafka_topic" = "example_topic", "kafka_partitions" = "0", "property.kafka_default_offsets" = "OFFSET_BEGINNING" ); ``` > Note: Replace `kafka_broker_list` with your Kafka server address. ##### Explanation of Parameters[​](#explanation-of-parameters "Direct link to Explanation of Parameters") ###### Data Format[​](#data-format "Direct link to Data Format") Specify the data format as JSON in the "format" = "json" of the PROPERTIES clause. ###### Data Extraction and Transformation[​](#data-extraction-and-transformation "Direct link to Data Extraction and Transformation") To specify the mapping and transformation relationship between the source data and the target table, configure the COLUMNS and jsonpaths parameters. The column names in COLUMNS correspond to the column names of the target table, and their order corresponds to the column order in the source data. The jsonpaths parameter is used to extract the required field data from JSON data, similar to newly generated CSV data. Then the COLUMNS parameter temporarily names the fields in jsonpaths in order. For more explanations on data transformation, please see [Data Transformation during Import](https://docs.starrocks.io/docs/loading/Etl_in_loading.md). > Note: If each JSON object per line has key names and quantities (order is not required) that correspond to the columns of the target table, there is no need to configure COLUMNS. #### Verifying Data Import[​](#verifying-data-import "Direct link to Verifying Data Import") First, we check the Routine Load import job and confirm the Routine Load import task status is in RUNNING status. ```sql show routine load\G ``` Then, querying the corresponding table in the StarRocks database, we can observe that the data has been successfully imported. ```sql StarRocks > select * from users; +------+--------------+---------------------+--------+ | id | name | timestamp | status | +------+--------------+---------------------+--------+ | 1 | testuser | 2023-11-10T12:00:00 | active | | 2 | testuser | 2023-11-10T12:00:00 | active | +------+--------------+---------------------+--------+ 2 rows in set (0.01 sec) ``` --- ### Load data from Microsoft Azure Storage StarRocks provides the following options for loading data from Azure: * Synchronous loading using [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md)+[`FILES()`](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md) * Asynchronous loading using [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) Each of these options has its own advantages, which are detailed in the following sections. In most cases, we recommend that you use the INSERT+`FILES()` method, which is much easier to use. However, the INSERT+`FILES()` method currently supports only the Parquet, ORC, and CSV file formats. Therefore, if you need to load data of other file formats such as JSON, or [perform data changes such as DELETE during data loading](https://docs.starrocks.io/docs/loading/Load_to_Primary_Key_tables.md), you can resort to Broker Load. #### Before you begin[​](#before-you-begin "Direct link to Before you begin") ##### Make source data ready[​](#make-source-data-ready "Direct link to Make source data ready") Make sure that the source data you want to load into StarRocks is properly stored in a container within your Azure storage account. In this topic, suppose you want to load the data of a Parquet-formatted sample dataset (`user_behavior_ten_million_rows.parquet`) stored in the root directory of a container (`starrocks-container`) within an Azure Data Lake Storage Gen2 (ADLS Gen2) storage account (`starrocks`). ##### Check privileges[​](#check-privileges "Direct link to Check privileges") You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. ##### Gather authentication details[​](#gather-authentication-details "Direct link to Gather authentication details") The examples in this topic use the Shared Key authentication method. To ensure that you have permission to read data from ADLS Gen2, we recommend that you read [Azure Data Lake Storage Gen2 > Shared Key (access key of storage account)](https://docs.starrocks.io/docs/integrations/authenticate_to_azure_storage.md#service-principal-1) to understand the authentication parameters that you need to configure. In a nutshell, if you practice Shared Key authentication, you need to gather the following information: * The username of your ADLS Gen2 storage account * The shared key of your ADLS Gen2 storage account For information about all the authentication methods available, see [Authenticate to Azure cloud storage](https://docs.starrocks.io/docs/integrations/authenticate_to_azure_storage.md). #### Use INSERT+FILES()[​](#use-insertfiles "Direct link to Use INSERT+FILES()") This method is available from v3.2 onwards and currently supports only the Parquet, ORC, and CSV (from v3.3.0 onwards) file formats. ##### Advantages of INSERT+FILES()[​](#advantages-of-insertfiles "Direct link to Advantages of INSERT+FILES()") `FILES()` can read the file stored in cloud storage based on the path-related properties you specify, infer the table schema of the data in the file, and then return the data from the file as data rows. With `FILES()`, you can: * Query the data directly from Azure using [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md). * Create and load a table using [CREATE TABLE AS SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE_AS_SELECT.md) (CTAS). * Load the data into an existing table using [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md). ##### Typical examples[​](#typical-examples "Direct link to Typical examples") ###### Querying directly from Azure using SELECT[​](#querying-directly-from-azure-using-select "Direct link to Querying directly from Azure using SELECT") Querying directly from Azure using SELECT+`FILES()` can give a good preview of the content of a dataset before you create a table. For example: * Get a preview of the dataset without storing the data. * Query for the min and max values and decide what data types to use. * Check for `NULL` values. The following example queries the sample dataset `user_behavior_ten_million_rows.parquet` stored in the container `starrocks-container` within your storage account `starrocks`: ```sql SELECT * FROM FILES ( "path" = "abfss://starrocks-container@starrocks.dfs.core.windows.net/user_behavior_ten_million_rows.parquet", "format" = "parquet", "azure.adls2.storage_account" = "starrocks", "azure.adls2.shared_key" = "xxxxxxxxxxxxxxxxxx" ) LIMIT 3; ``` The system returns a query result similar to the following: ```plain +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 543711 | 829192 | 2355072 | pv | 2017-11-27 08:22:37 | | 543711 | 2056618 | 3645362 | pv | 2017-11-27 10:16:46 | | 543711 | 1165492 | 3645362 | pv | 2017-11-27 10:17:00 | +--------+---------+------------+--------------+---------------------+ ``` > **NOTE** > > Notice that the column names as returned above are provided by the Parquet file. ###### Creating and loading a table using CTAS[​](#creating-and-loading-a-table-using-ctas "Direct link to Creating and loading a table using CTAS") This is a continuation of the previous example. The previous query is wrapped in CREATE TABLE AS SELECT (CTAS) to automate the table creation using schema inference. This means StarRocks will infer the table schema, create the table you want, and then load the data into the table. The column names and types are not required to create a table when using the `FILES()` table function with Parquet files as the Parquet format includes the column names. > **NOTE** > > The syntax of CREATE TABLE when using schema inference does not allow setting the number of replicas. If you are using a StarRocks shared-nothing cluster, set the number of replicas before creating the table. The example below is for a system with three replicas: > > ```sql > ADMIN SET FRONTEND CONFIG ('default_replication_num' = "3"); > > ``` Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Use CTAS to create a table and load the data of the sample dataset `user_behavior_ten_million_rows.parquet`, which is stored in the container `starrocks-container` within your storage account `starrocks`, into the table: ```sql CREATE TABLE user_behavior_inferred AS SELECT * FROM FILES ( "path" = "abfss://starrocks-container@starrocks.dfs.core.windows.net/user_behavior_ten_million_rows.parquet", "format" = "parquet", "azure.adls2.storage_account" = "starrocks", "azure.adls2.shared_key" = "xxxxxxxxxxxxxxxxxx" ); ``` After creating the table, you can view its schema by using [DESCRIBE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DESCRIBE.md): ```sql DESCRIBE user_behavior_inferred; ``` The system returns the following query result: ```plain +--------------+-----------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------+-----------+------+-------+---------+-------+ | UserID | bigint | YES | true | NULL | | | ItemID | bigint | YES | true | NULL | | | CategoryID | bigint | YES | true | NULL | | | BehaviorType | varbinary | YES | false | NULL | | | Timestamp | varbinary | YES | false | NULL | | +--------------+-----------+------+-------+---------+-------+ ``` Query the table to verify that the data has been loaded into it. Example: ```sql SELECT * from user_behavior_inferred LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plain +--------+--------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+--------+------------+--------------+---------------------+ | 84 | 162325 | 2939262 | pv | 2017-12-02 05:41:41 | | 84 | 232622 | 4148053 | pv | 2017-11-27 04:36:10 | | 84 | 595303 | 903809 | pv | 2017-11-26 08:03:59 | +--------+--------+------------+--------------+---------------------+ ``` ###### Loading into an existing table using INSERT[​](#loading-into-an-existing-table-using-insert "Direct link to Loading into an existing table using INSERT") You may want to customize the table that you are inserting into, for example, the: * column data type, nullable setting, or default values * key types and columns * data partitioning and bucketing > **NOTE** > > Creating the most efficient table structure requires knowledge of how the data will be used and the content of the columns. This topic does not cover table design. For information about table design, see [Table types](https://docs.starrocks.io/docs/table_design/StarRocks_table_design.md). In this example, we are creating a table based on knowledge of how the table will be queried and the data in the Parquet file. The knowledge of the data in the Parquet file can be gained by querying the file directly in Azure. * Since a query of the dataset in Azure indicates that the `Timestamp` column contains data that matches a VARBINARY data type, the column type is specified in the following DDL. * By querying the data in Azure, you can find that there are no `NULL` values in the dataset, so the DDL does not set any columns as nullable. * Based on knowledge of the expected query types, the sort key and bucketing column are set to the column `UserID`. Your use case might be different for this data, so you might decide to use `ItemID` in addition to or instead of `UserID` for the sort key. Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand (we recommend that the table have the same schema as the Parquet file you want to load from Azure): ```sql CREATE TABLE user_behavior_declared ( UserID int(11), ItemID int(11), CategoryID int(11), BehaviorType varchar(65533), Timestamp varbinary ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID); ``` Display the schema so that you can compare it with the inferred schema produced by the `FILES()` table function: ```sql DESCRIBE user_behavior_declared; ``` ```plaintext +--------------+----------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------+----------------+------+-------+---------+-------+ | UserID | int | NO | true | NULL | | | ItemID | int | NO | false | NULL | | | CategoryID | int | NO | false | NULL | | | BehaviorType | varchar(65533) | NO | false | NULL | | | Timestamp | varbinary | NO | false | NULL | | +--------------+----------------+------+-------+---------+-------+ 5 rows in set (0.00 sec) ``` tip Compare the schema you just created with the schema inferred earlier using the `FILES()` table function. Look at: * data types * nullable * key fields To better control the schema of the destination table and for better query performance, we recommend that you specify the table schema by hand in production environments. After creating the table, you can load it with INSERT INTO SELECT FROM FILES(): ```sql INSERT INTO user_behavior_declared SELECT * FROM FILES ( "path" = "abfss://starrocks-container@starrocks.dfs.core.windows.net/user_behavior_ten_million_rows.parquet", "format" = "parquet", "azure.adls2.storage_account" = "starrocks", "azure.adls2.shared_key" = "xxxxxxxxxxxxxxxxxx" ); ``` After the load is complete, you can query the table to verify that the data has been loaded into it. Example: ```sql SELECT * from user_behavior_declared LIMIT 3; ``` The system returns a query result similar to the following, indicating that the data has been successfully loaded: ```plain +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 142 | 2869980 | 2939262 | pv | 2017-11-25 03:43:22 | | 142 | 2522236 | 1669167 | pv | 2017-11-25 15:14:12 | | 142 | 3031639 | 3607361 | pv | 2017-11-25 15:19:25 | +--------+---------+------------+--------------+---------------------+ ``` ###### Check load progress[​](#check-load-progress "Direct link to Check load progress") You can query the progress of INSERT jobs from the [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md) view in the StarRocks Information Schema. This feature is supported from v3.1 onwards. Example: ```sql SELECT * FROM information_schema.loads ORDER BY JOB_ID DESC; ``` For information about the fields provided in the `loads` view, see [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md). If you have submitted multiple load jobs, you can filter on the `LABEL` associated with the job. Example: ```sql SELECT * FROM information_schema.loads WHERE LABEL = 'insert_f3fc2298-a553-11ee-92f4-00163e0842bd' \G *************************** 1. row *************************** JOB_ID: 10193 LABEL: insert_f3fc2298-a553-11ee-92f4-00163e0842bd DATABASE_NAME: mydatabase STATE: FINISHED PROGRESS: ETL:100%; LOAD:100% TYPE: INSERT PRIORITY: NORMAL SCAN_ROWS: 10000000 FILTERED_ROWS: 0 UNSELECTED_ROWS: 0 SINK_ROWS: 10000000 ETL_INFO: TASK_INFO: resource:N/A; timeout(s):300; max_filter_ratio:0.0 CREATE_TIME: 2023-12-28 15:37:38 ETL_START_TIME: 2023-12-28 15:37:38 ETL_FINISH_TIME: 2023-12-28 15:37:38 LOAD_START_TIME: 2023-12-28 15:37:38 LOAD_FINISH_TIME: 2023-12-28 15:39:35 JOB_DETAILS: {"All backends":{"f3fc2298-a553-11ee-92f4-00163e0842bd":[10120]},"FileNumber":0,"FileSize":0,"InternalTableLoadBytes":581730322,"InternalTableLoadRows":10000000,"ScanBytes":581574034,"ScanRows":10000000,"TaskNumber":1,"Unfinished backends":{"f3fc2298-a553-11ee-92f4-00163e0842bd":[]}} ERROR_MSG: NULL TRACKING_URL: NULL TRACKING_SQL: NULL REJECTED_RECORD_PATH: NULL ``` > **NOTE** > > INSERT is a synchronous command. If an INSERT job is still running, you need to open another session to check its execution status. #### Use Broker Load[​](#use-broker-load "Direct link to Use Broker Load") An asynchronous Broker Load process handles making the connection to Azure, pulling the data, and storing the data in StarRocks. This method supports the following file formats: * Parquet * ORC * CSV * JSON (supported from v3.2.3 onwards) ##### Advantages of Broker Load[​](#advantages-of-broker-load "Direct link to Advantages of Broker Load") * Broker Load runs in the background and clients don't need to stay connected for the job to continue. * Broker Load is preferred for long running jobs, the default timeout is 4 hours. * In addition to Parquet and ORC file format, Broker Load supports CSV file format and JSON file format (JSON file format is supported from v3.2.3 onwards). ##### Data flow[​](#data-flow "Direct link to Data flow") ![Workflow of Broker Load](/assets/images/broker_load_how-to-work_en-bb36de70866e6366b2b21808f0f77be8.png) 1. The user creates a load job. 2. The frontend (FE) creates a query plan and distributes the plan to the backend nodes (BEs) or compute nodes (CNs). 3. The BEs or CNs pull the data from the source and load the data into StarRocks. ##### Typical example[​](#typical-example "Direct link to Typical example") Create a table, start a load process that pulls the sample dataset `user_behavior_ten_million_rows.parquet` from Azure, and verify the progress and success of the data loading. ###### Create a database and a table[​](#create-a-database-and-a-table "Direct link to Create a database and a table") Connect to your StarRocks cluster. Then, create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand (we recommend that the table have the same schema as the Parquet file you want to load from Azure): ```sql CREATE TABLE user_behavior ( UserID int(11), ItemID int(11), CategoryID int(11), BehaviorType varchar(65533), Timestamp varbinary ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID); ``` ###### Start a Broker Load[​](#start-a-broker-load "Direct link to Start a Broker Load") Run the following command to start a Broker Load job that loads data from the sample dataset `user_behavior_ten_million_rows.parquet` to the `user_behavior` table: ```sql LOAD LABEL user_behavior ( DATA INFILE("abfss://starrocks-container@starrocks.dfs.core.windows.net/user_behavior_ten_million_rows.parquet") INTO TABLE user_behavior FORMAT AS "parquet" ) WITH BROKER ( "azure.adls2.storage_account" = "starrocks", "azure.adls2.shared_key" = "xxxxxxxxxxxxxxxxxx" ) PROPERTIES ( "timeout" = "3600" ); ``` This job has four main sections: * `LABEL`: A string used when querying the state of the load job. * `LOAD` declaration: The source URI, source data format, and destination table name. * `BROKER`: The connection details for the source. * `PROPERTIES`: The timeout value and any other properties to apply to the load job. For detailed syntax and parameter descriptions, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). ###### Check load progress[​](#check-load-progress-1 "Direct link to Check load progress") You can query the progress of Broker Load jobs from the [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md) view in the StarRocks Information Schema. This feature is supported from v3.1 onwards. ```sql SELECT * FROM information_schema.loads \G ``` For information about the fields provided in the `loads` view, see [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md). If you have submitted multiple load jobs, you can filter on the `LABEL` associated with the job: ```sql SELECT * FROM information_schema.loads WHERE LABEL = 'user_behavior' \G *************************** 1. row *************************** JOB_ID: 10250 LABEL: user_behavior DATABASE_NAME: mydatabase STATE: FINISHED PROGRESS: ETL:100%; LOAD:100% TYPE: BROKER PRIORITY: NORMAL SCAN_ROWS: 10000000 FILTERED_ROWS: 0 UNSELECTED_ROWS: 0 SINK_ROWS: 10000000 ETL_INFO: TASK_INFO: resource:N/A; timeout(s):3600; max_filter_ratio:0.0 CREATE_TIME: 2023-12-28 16:15:19 ETL_START_TIME: 2023-12-28 16:15:25 ETL_FINISH_TIME: 2023-12-28 16:15:25 LOAD_START_TIME: 2023-12-28 16:15:25 LOAD_FINISH_TIME: 2023-12-28 16:16:31 JOB_DETAILS: {"All backends":{"6a8ef4c0-1009-48c9-8d18-c4061d2255bf":[10121]},"FileNumber":1,"FileSize":132251298,"InternalTableLoadBytes":311710786,"InternalTableLoadRows":10000000,"ScanBytes":132251298,"ScanRows":10000000,"TaskNumber":1,"Unfinished backends":{"6a8ef4c0-1009-48c9-8d18-c4061d2255bf":[]}} ERROR_MSG: NULL TRACKING_URL: NULL TRACKING_SQL: NULL REJECTED_RECORD_PATH: NULL ``` After you confirm that the load job has finished, you can check a subset of the destination table to see if the data has been successfully loaded. Example: ```sql SELECT * from user_behavior LIMIT 3; ``` The system returns a query result similar to the following, indicating that the data has been successfully loaded: ```plain +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 142 | 2869980 | 2939262 | pv | 2017-11-25 03:43:22 | | 142 | 2522236 | 1669167 | pv | 2017-11-25 15:14:12 | | 142 | 3031639 | 3607361 | pv | 2017-11-25 15:19:25 | +--------+---------+------------+--------------+---------------------+ ``` --- ### Load data from HDFS or cloud storage StarRocks provides the loading method MySQL-based Broker Load to help you load a large amount of data from HDFS or cloud storage into StarRocks. Broker Load runs in asynchronous loading mode. After you submit a load job, StarRocks asynchronously runs the job. You need to use the [SHOW LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SHOW_LOAD.md) statement or the `curl` command to check the result of the job. Broker Load supports single-table loads and multi-table loads. You can load one or multiple data files into one or multiple destination tables by running one Broker Load job. Broker Load ensures the transactional atomicity of each load job that is run to load multiple data files. Atomicity means that the loading of multiple data files in one load job must all succeed or fail. It never happens that the loading of some data files succeeds while the loading of the other files fails. Broker Load supports data transformation at data loading and supports data changes made by UPSERT and DELETE operations during data loading. For more information, see [Transform data at loading](https://docs.starrocks.io/docs/loading/Etl_in_loading.md) and [Change data through loading](https://docs.starrocks.io/docs/loading/Load_to_Primary_Key_tables.md). You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. #### Background information[​](#background-information "Direct link to Background information") In v2.4 and earlier, StarRocks depends on brokers to set up connections between your StarRocks cluster and your external storage system when it runs a Broker Load job. Therefore, you need to input `WITH BROKER ""` to specify the broker you want to use in the load statement. This is called "broker-based loading." A broker is an independent, stateless service that is integrated with a file-system interface. With brokers, StarRocks can access and read data files that are stored in your external storage system, and can use its own computing resources to pre-process and load the data of these data files. From v2.5 onwards, StarRocks no longer depends on brokers to set up connections between your StarRocks cluster and your external storage system when it runs a Broker Load job. Therefore, you no longer need to specify a broker in the load statement, but you still need to retain the `WITH BROKER` keyword. This is called "broker-free loading." When your data is stored in HDFS, you may encounter situations where broker-free loading does not work. This can happen when your data is stored across multiple HDFS clusters or when you have configured multiple Kerberos users. In these situations, you can resort to using broker-based loading instead. To do this successfully, make sure that at least one independent broker group is deployed. For information about how to specify authentication configuration and HA configuration in these situations, see [HDFS](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md#hdfs). #### Supported data file formats[​](#supported-data-file-formats "Direct link to Supported data file formats") Broker Load supports the following data file formats: * CSV * Parquet * ORC > **NOTE** > > For CSV data, take note of the following points: > > * You can use a UTF-8 string, such as a comma (,), tab, or pipe (|), whose length does not exceed 50 bytes as a text delimiter. > * Null values are denoted by using `\N`. For example, a data file consists of three columns, and a record from that data file holds data in the first and third columns but no data in the second column. In this situation, you need to use `\N` in the second column to denote a null value. This means the record must be compiled as `a,\N,b` instead of `a,,b`. `a,,b` denotes that the second column of the record holds an empty string. #### Supported storage systems[​](#supported-storage-systems "Direct link to Supported storage systems") Broker Load supports the following storage systems: * HDFS * AWS S3 * Google GCS * Other S3-compatible storage system such as MinIO * Microsoft Azure Storage #### How it works[​](#how-it-works "Direct link to How it works") After you submit a load job to an FE, the FE generates a query plan, splits the query plan into portions based on the number of available BEs and the size of the data file you want to load, and then assigns each portion of the query plan to an available BE. During the load, each involved BE pulls the data of the data file from your HDFS or cloud storage system, pre-processes the data, and then loads the data into your StarRocks cluster. After all BEs finish their portions of the query plan, the FE determines whether the load job is successful. The following figure shows the workflow of a Broker Load job. ![Workflow of Broker Load](/assets/images/broker_load_how-to-work_en-bb36de70866e6366b2b21808f0f77be8.png) #### Basic operations[​](#basic-operations "Direct link to Basic operations") ##### Create a multi-table load job[​](#create-a-multi-table-load-job "Direct link to Create a multi-table load job") This topic uses CSV as an example to describe how to load multiple data files into multiple tables. For information about how to load data in other file formats and about the syntax and parameter descriptions for Broker Load, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). Note that in StarRocks some literals are used as reserved keywords by the SQL language. Do not directly use these keywords in SQL statements. If you want to use such a keyword in an SQL statement, enclose it in a pair of backticks (\`). See [Keywords](https://docs.starrocks.io/docs/sql-reference/sql-statements/keywords.md). ###### Data examples[​](#data-examples "Direct link to Data examples") 1. Create CSV files in your local file system. a. Create a CSV file named `file1.csv`. The file consists of three columns, which represent user ID, user name, and user score in sequence. ```plain 1,Lily,23 2,Rose,23 3,Alice,24 4,Julia,25 ``` b. Create a CSV file named `file2.csv`. The file consists of two columns, which represent city ID and city name in sequence. ```plain 200,'Beijing' ``` 2. Create StarRocks tables in your StarRocks database `test_db`. > **NOTE** > > Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). a. Create a Primary Key table named `table1`. The table consists of three columns: `id`, `name`, and `score`, of which `id` is the primary key. ```sql CREATE TABLE `table1` ( `id` int(11) NOT NULL COMMENT "user ID", `name` varchar(65533) NULL DEFAULT "" COMMENT "user name", `score` int(11) NOT NULL DEFAULT "0" COMMENT "user score" ) ENGINE=OLAP PRIMARY KEY(`id`) DISTRIBUTED BY HASH(`id`); ``` b. Create a Primary Key table named `table2`. The table consists of two columns: `id` and `city`, of which `id` is the primary key. ```sql CREATE TABLE `table2` ( `id` int(11) NOT NULL COMMENT "city ID", `city` varchar(65533) NULL DEFAULT "" COMMENT "city name" ) ENGINE=OLAP PRIMARY KEY(`id`) DISTRIBUTED BY HASH(`id`); ``` 3. Upload `file1.csv` and `file2.csv` to the `/user/starrocks/` path of your HDFS cluster, to the `input` folder of your AWS S3 bucket `bucket_s3`, to the `input` folder of your Google GCS bucket `bucket_gcs`, to the `input` folder of your MinIO bucket `bucket_minio`, and to the specified paths of your Azure Storage. ###### Load data from HDFS[​](#load-data-from-hdfs "Direct link to Load data from HDFS") Execute the following statement to load `file1.csv` and `file2.csv` from the `/user/starrocks` path of your HDFS cluster into `table1` and `table2`, respectively: ```sql LOAD LABEL test_db.label1 ( DATA INFILE("hdfs://:/user/starrocks/file1.csv") INTO TABLE table1 COLUMNS TERMINATED BY "," (id, name, score) , DATA INFILE("hdfs://:/user/starrocks/file2.csv") INTO TABLE table2 COLUMNS TERMINATED BY "," (id, city) ) WITH BROKER ( StorageCredentialParams ) PROPERTIES ( "timeout" = "3600" ); ``` In the preceding example, `StorageCredentialParams` represents a group of authentication parameters which vary depending on the authentication method you choose. For more information, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md#hdfs). ###### Load data from AWS S3[​](#load-data-from-aws-s3 "Direct link to Load data from AWS S3") Execute the following statement to load `file1.csv` and `file2.csv` from the `input` folder of your AWS S3 bucket `bucket_s3` into `table1` and `table2`, respectively: ```sql LOAD LABEL test_db.label2 ( DATA INFILE("s3a://bucket_s3/input/file1.csv") INTO TABLE table1 COLUMNS TERMINATED BY "," (id, name, score) , DATA INFILE("s3a://bucket_s3/input/file2.csv") INTO TABLE table2 COLUMNS TERMINATED BY "," (id, city) ) WITH BROKER ( StorageCredentialParams ); ``` > **NOTE** > > Broker Load supports accessing AWS S3 only according to the S3A protocol. Therefore, when you load data from AWS S3, you must replace `s3://` in the S3 URI you pass as the file path with `s3a://`. In the preceding example, `StorageCredentialParams` represents a group of authentication parameters which vary depending on the authentication method you choose. For more information, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md#aws-s3). From v3.1 onwards, StarRocks supports directly loading the data of Parquet-formatted or ORC-formatted files from AWS S3 by using the INSERT command and the TABLE keyword, saving you from the trouble of creating an external table first. For more information, see [Load data using INSERT > Insert data directly from files in an external source using TABLE keyword](https://docs.starrocks.io/docs/loading/InsertInto.md#insert-data-directly-from-files-in-an-external-source-using-files). ###### Load data from Google GCS[​](#load-data-from-google-gcs "Direct link to Load data from Google GCS") Execute the following statement to load `file1.csv` and `file2.csv` from the `input` folder of your Google GCS bucket `bucket_gcs` into `table1` and `table2`, respectively: ```sql LOAD LABEL test_db.label3 ( DATA INFILE("gs://bucket_gcs/input/file1.csv") INTO TABLE table1 COLUMNS TERMINATED BY "," (id, name, score) , DATA INFILE("gs://bucket_gcs/input/file2.csv") INTO TABLE table2 COLUMNS TERMINATED BY "," (id, city) ) WITH BROKER ( StorageCredentialParams ); ``` > **NOTE** > > Broker Load supports accessing Google GCS only according to the gs protocol. Therefore, when you load data from Google GCS, you must include `gs://` as the prefix in the GCS URI that you pass as the file path. In the preceding example, `StorageCredentialParams` represents a group of authentication parameters which vary depending on the authentication method you choose. For more information, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md#google-gcs). ###### Load data from other S3-compatible storage system[​](#load-data-from-other-s3-compatible-storage-system "Direct link to Load data from other S3-compatible storage system") Use MinIO as an example. You can execute the following statement to load `file1.csv` and `file2.csv` from the `input` folder of your MinIO bucket `bucket_minio` into `table1` and `table2`, respectively: ```sql LOAD LABEL test_db.label7 ( DATA INFILE("s3://bucket_minio/input/file1.csv") INTO TABLE table1 COLUMNS TERMINATED BY "," (id, name, score) , DATA INFILE("s3://bucket_minio/input/file2.csv") INTO TABLE table2 COLUMNS TERMINATED BY "," (id, city) ) WITH BROKER ( StorageCredentialParams ); ``` In the preceding example, `StorageCredentialParams` represents a group of authentication parameters which vary depending on the authentication method you choose. For more information, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md#other-s3-compatible-storage-system). ###### Load data from Microsoft Azure Storage[​](#load-data-from-microsoft-azure-storage "Direct link to Load data from Microsoft Azure Storage") Execute the following statement to load `file1.csv` and `file2.csv` from the specified paths of your Azure Storage: ```sql LOAD LABEL test_db.label8 ( DATA INFILE("wasb[s]://@.blob.core.windows.net//file1.csv") INTO TABLE table1 COLUMNS TERMINATED BY "," (id, name, score) , DATA INFILE("wasb[s]://@.blob.core.windows.net//file2.csv") INTO TABLE table2 COLUMNS TERMINATED BY "," (id, city) ) WITH BROKER ( StorageCredentialParams ); ``` > **NOTICE** > > When you load data from Azure Storage, you need to determine which prefix to use based on the access protocol and specific storage service that you use. The preceding example uses Blob Storage as an example. > > * When you load data from Blob Storage, you must include `wasb://` or `wasbs://` as a prefix in the file path based on the protocol that is used to access your storage account: > > > > * If your Blob Storage allows access only through HTTP, use `wasb://` as the prefix, for example, `wasb://@.blob.core.windows.net///*`. > * If your Blob Storage allows access only through HTTPS, use `wasbs://` as the prefix, for example, `wasbs://@.blob.core.windows.net///*` > > * When you load data from Data Lake Storage Gen1, you must include `adl://` as a prefix in the file path, for example, `adl://.azuredatalakestore.net//`. > > * When you load data from Data Lake Storage Gen2, you must include `abfs://` or `abfss://` as a prefix in the file path based on the protocol that is used to access your storage account: > > > > * If your Data Lake Storage Gen2 allows access only via HTTP, use `abfs://` as the prefix, for example, `abfs://@.dfs.core.windows.net/`. > * If your Data Lake Storage Gen2 allows access only via HTTPS, use `abfss://` as the prefix, for example, `abfss://@.dfs.core.windows.net/`. In the preceding example, `StorageCredentialParams` represents a group of authentication parameters which vary depending on the authentication method you choose. For more information, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md#microsoft-azure-storage). ###### Query data[​](#query-data "Direct link to Query data") After the load of data from your HDFS cluster, AWS S3 bucket, or Google GCS bucket is complete, you can use the SELECT statement to query the data of the StarRocks tables to verify that the load is successful. 1. Execute the following statement to query the data of `table1`: ```sql MySQL [test_db]> SELECT * FROM table1; +------+-------+-------+ | id | name | score | +------+-------+-------+ | 1 | Lily | 23 | | 2 | Rose | 23 | | 3 | Alice | 24 | | 4 | Julia | 25 | +------+-------+-------+ 4 rows in set (0.00 sec) ``` 2. Execute the following statement to query the data of `table2`: ```sql MySQL [test_db]> SELECT * FROM table2; +------+--------+ | id | city | +------+--------+ | 200 | Beijing| +------+--------+ 4 rows in set (0.01 sec) ``` ##### Create a single-table load job[​](#create-a-single-table-load-job "Direct link to Create a single-table load job") You can also load a single data file or all data files from a specified path into a single destination table. Suppose your AWS S3 bucket `bucket_s3` contains a folder named `input`. The `input` folder contains multiple data files, one of which is named `file1.csv`. These data files consist of the same number of columns as `table1` and the columns from each of these data files can be mapped one on one in sequence to the columns from `table1`. To load `file1.csv` into `table1`, execute the following statement: ```sql LOAD LABEL test_db.label_7 ( DATA INFILE("s3a://bucket_s3/input/file1.csv") INTO TABLE table1 COLUMNS TERMINATED BY "," FORMAT AS "CSV" ) WITH BROKER ( StorageCredentialParams ); ``` To load all data files from the `input` folder into `table1`, execute the following statement: ```sql LOAD LABEL test_db.label_8 ( DATA INFILE("s3a://bucket_s3/input/*") INTO TABLE table1 COLUMNS TERMINATED BY "," FORMAT AS "CSV" ) WITH BROKER ( StorageCredentialParams ); ``` In the preceding examples, `StorageCredentialParams` represents a group of authentication parameters which vary depending on the authentication method you choose. For more information, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md#aws-s3). ##### View a load job[​](#view-a-load-job "Direct link to View a load job") Broker Load allows you to view a lob job by using the SHOW LOAD statement or the `curl` command. ###### Use SHOW LOAD[​](#use-show-load "Direct link to Use SHOW LOAD") For more information, see [SHOW LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SHOW_LOAD.md). ###### Use curl[​](#use-curl "Direct link to Use curl") The syntax is as follows: ```bash curl --location-trusted -u : \ 'http://:/api//_load_info?label=' ``` > **NOTE** > > If you use an account for which no password is set, you need to input only `:`. For example, you can run the following command to view the information about a load job, whose label is `label1`, in the `test_db` database: ```bash curl --location-trusted -u : \ 'http://:/api/test_db/_load_info?label=label1' ``` The `curl` command returns the information about the most recently executed load job with the specified label as a JSON object `jobInfo`: ```json {"jobInfo":{"dbName":"default_cluster:test_db","tblNames":["table1_simple"],"label":"label1","state":"FINISHED","failMsg":"","trackingUrl":""},"status":"OK","msg":"Success"}% ``` The following table describes the parameters in `jobInfo`. | **Parameter** | **Description** | | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | dbName | The name of the database into which data is loaded | | tblNames | The name of the table into which data is loaded. | | label | The label of the load job. | | state | The status of the load job. Valid values:- `PENDING`: The load job is in queue waiting to be scheduled.
- `QUEUEING`: The load job is in the queue waiting to be scheduled.
- `LOADING`: The load job is running.
- `PREPARED`: The transaction has been committed.
- `FINISHED`: The load job succeeded.
- `CANCELLED`: The load job failed.For more information, see the "Asynchronous loading" section in [Loading concepts](https://docs.starrocks.io/docs/loading/loading_introduction/loading_concepts.md). | | failMsg | The reason why the load job failed. If the `state` value for the load job is `PENDING`, `LOADING`, or `FINISHED`, `NULL` is returned for the `failMsg` parameter. If the `state` value for the load job is `CANCELLED`, the value returned for the `failMsg` parameter consists of two parts: `type` and `msg`.- The `type` part can be any of the following values:
- The `msg` part provides the detailed cause of the load failure. | | trackingUrl | The URL that is used to access the unqualified data detected in the load job. You can use the `curl` or `wget` command to access the URL and obtain the unqualified data. If no unqualified data is detected, `NULL` is returned for the `trackingUrl` parameter. | | status | The status of the HTTP request for the load job. Valid values: `OK` and `Fail`. | | msg | The error information of the HTTP request for the load job. | ##### Cancel a load job[​](#cancel-a-load-job "Direct link to Cancel a load job") When a load job is not in the **CANCELLED** or **FINISHED** stage, you can use the [CANCEL LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/CANCEL_LOAD.md) statement to cancel the job. For example, you can execute the following statement to cancel a load job, whose label is `label1`, in the database `test_db`: ```sql CANCEL LOAD FROM test_db WHERE LABEL = "label"; ``` #### Job splitting and concurrent running[​](#job-splitting-and-concurrent-running "Direct link to Job splitting and concurrent running") A Broker Load job can be split into one or more tasks that concurrently run. The tasks within a load job are run within a single transaction. They must all succeed or fail. StarRocks splits each load job based on how you declare `data_desc` in the `LOAD` statement: * If you declare multiple `data_desc` parameters, each of which specifies a distinct table, a task is generated to load the data of each table. * If you declare multiple `data_desc` parameters, each of which specifies a distinct partition for the same table, a task is generated to load the data of each partition. Additionally, each task can be further split into one or more instances, which are evenly distributed to and concurrently run on the BEs of your StarRocks cluster. StarRocks splits each task based on the following [FE configurations](https://docs.starrocks.io/docs/administration/management/FE_configuration.md): * `min_bytes_per_broker_scanner`: the minimum amount of data processed by each instance. The default amount is 64 MB. * `load_parallel_instance_num`: the number of concurrent instances allowed in each load job on an individual BE. The default number is 1. You can use the following formula to calculate the number of instances in an individual task: **Number of instances in an individual task = min(Amount of data to be loaded by an individual task/`min_bytes_per_broker_scanner`,`load_parallel_instance_num` x Number of BEs)** In most cases, only one `data_desc` is declared for each load job, each load job is split into only one task, and the task is split into the same number of instances as the number of BEs. #### Related configuration items[​](#related-configuration-items "Direct link to Related configuration items") The [FE configuration item](https://docs.starrocks.io/docs/administration/management/FE_configuration.md) `max_broker_load_job_concurrency` specifies the maximum number of Broker Load jobs that can be concurrently run within your StarRocks cluster. In StarRocks v2.4 and earlier, if the total number of Broker Load jobs that are submitted within a specific period of time exceeds the maximum number, excessive jobs are queued and scheduled based on their submission time. Since StarRocks v2.5, if the total number of Broker Load jobs that are submitted within a specific period of time exceeds the maximum number, excessive jobs are queued and scheduled based on their priorities. You can specify a priority for a job by using the `priority` parameter at job creation. See [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md#opt_properties). You can also use [ALTER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/ALTER_LOAD.md) to modify the priority of an existing job that is in the **QUEUEING** or **LOADING** state. --- ### Transform data at loading StarRocks supports data transformation at loading. This feature supports [Stream Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md), [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md), and [Routine Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md) but does not support [Spark Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SPARK_LOAD.md). You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. This topic uses CSV data as an example to describe how to extract and transform data at loading. The data file formats that are supported vary depending on the loading method of your choice. > **NOTE** > > For CSV data, you can use a UTF-8 string, such as a comma (,), tab, or pipe (|), whose length does not exceed 50 bytes as a text delimiter. #### Scenarios[​](#scenarios "Direct link to Scenarios") When you load a data file into a StarRocks table, the data of the data file may not be completely mapped onto the data of the StarRocks table. In this situation, you do not need to extract or transform the data before you load it into the StarRocks table. StarRocks can help you extract and transform the data during loading: * Skip columns that do not need to be loaded. You can skip the columns that do not need to be loaded. Additionally, if the columns of the data file are in a different order than the columns of the StarRocks table, you can create a column mapping between the data file and the StarRocks table. * Filter out rows you do not want to load. You can specify filter conditions based on which StarRocks filters out the rows that you do not want to load. * Generate new columns from original columns. Generated columns are special columns that are computed from the original columns of the data file. You can map the generated columns onto the columns of the StarRocks table. * Extract partition field values from a file path. If the data file is generated from Apache Hive™, you can extract partition field values from the file path. #### Data examples[​](#data-examples "Direct link to Data examples") 1. Create data files in your local file system. a. Create a data file named `file1.csv`. The file consists of four columns, which represent user ID, user gender, event date, and event type in sequence. ```plain 354,female,2020-05-20,1 465,male,2020-05-21,2 576,female,2020-05-22,1 687,male,2020-05-23,2 ``` b. Create a data file named `file2.csv`. The file consists of only one column, which represents date. ```plain 2020-05-20 2020-05-21 2020-05-22 2020-05-23 ``` 2. Create tables in your StarRocks database `test_db`. > **NOTE** > > Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). a. Create a table named `table1`, which consists of three columns: `event_date`, `event_type`, and `user_id`. ```sql MySQL [test_db]> CREATE TABLE table1 ( `event_date` DATE COMMENT "event date", `event_type` TINYINT COMMENT "event type", `user_id` BIGINT COMMENT "user ID" ) DISTRIBUTED BY HASH(user_id); ``` b. Create a table named `table2`, which consists of four columns: `date`, `year`, `month`, and `day`. ```sql MySQL [test_db]> CREATE TABLE table2 ( `date` DATE COMMENT "date", `year` INT COMMENT "year", `month` TINYINT COMMENT "month", `day` TINYINT COMMENT "day" ) DISTRIBUTED BY HASH(date); ``` 3. Upload `file1.csv` and `file2.csv` to the `/user/starrocks/data/input/` path of your HDFS cluster, publish the data of `file1.csv` to `topic1` of your Kafka cluster, and publish the data of `file2.csv` to `topic2` of your Kafka cluster. #### Skip columns that do not need to be loaded[​](#skip-columns-that-do-not-need-to-be-loaded "Direct link to Skip columns that do not need to be loaded") The data file that you want to load into a StarRocks table may contain some columns that cannot be mapped to any columns of the StarRocks table. In this situation, StarRocks supports loading only the columns that can be mapped from the data file onto the columns of the StarRocks table. This feature supports loading data from the following data sources: * Local file system * HDFS and cloud storage > **NOTE** > > This section uses HDFS as an example. * Kafka In most cases, the columns of a CSV file are not named. For some CSV files, the first row is composed of column names, but StarRocks processes the content of the first row as common data rather than column names. Therefore, when you load a CSV file, you must temporarily name the columns of the CSV file **in sequence** in the job creation statement or command. These temporarily named columns are mapped **by name** onto the columns of the StarRocks table. Take note of the following points about the columns of the data file: * The data of the columns that can be mapped onto and are temporarily named by using the names of the columns in the StarRocks table is directly loaded. * The columns that cannot be mapped onto the columns of the StarRocks table are ignored, the data of these columns are not loaded. * If some columns can be mapped onto the columns of the StarRocks table but are not temporarily named in the job creation statement or command, the load job reports errors. This section uses `file1.csv` and `table1` as an example. The four columns of `file1.csv` are temporarily named as `user_id`, `user_gender`, `event_date`, and `event_type` in sequence. Among the temporarily named columns of `file1.csv`, `user_id`, `event_date`, and `event_type` can be mapped onto specific columns of `table1`, whereas `user_gender` cannot be mapped onto any column of `table1`. Therefore, `user_id`, `event_date`, and `event_type` are loaded into `table1`, but `user_gender` is not. ##### Load data[​](#load-data "Direct link to Load data") ###### Load data from a local file system[​](#load-data-from-a-local-file-system "Direct link to Load data from a local file system") If `file1.csv` is stored in your local file system, run the following command to create a [Stream Load](https://docs.starrocks.io/docs/loading/StreamLoad.md) job: ```bash curl --location-trusted -u : \ -H "Expect:100-continue" \ -H "column_separator:," \ -H "columns: user_id, user_gender, event_date, event_type" \ -T file1.csv -XPUT \ http://:/api/test_db/table1/_stream_load ``` > **NOTE** > > If you choose Stream Load, you must use the `columns` parameter to temporarily name the columns of the data file to create a column mapping between the data file and the StarRocks table. For detailed syntax and parameter descriptions, see [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). ###### Load data from an HDFS cluster[​](#load-data-from-an-hdfs-cluster "Direct link to Load data from an HDFS cluster") If `file1.csv` is stored in your HDFS cluster, execute the following statement to create a [Broker Load](https://docs.starrocks.io/docs/loading/hdfs_load.md) job: ```sql LOAD LABEL test_db.label1 ( DATA INFILE("hdfs://:/user/starrocks/data/input/file1.csv") INTO TABLE `table1` FORMAT AS "csv" COLUMNS TERMINATED BY "," (user_id, user_gender, event_date, event_type) ) WITH BROKER; ``` > **NOTE** > > If you choose Broker Load, you must use the `column_list` parameter to temporarily name the columns of the data file to create a column mapping between the data file and the StarRocks table. For detailed syntax and parameter descriptions, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). ###### Load data from a Kafka cluster[​](#load-data-from-a-kafka-cluster "Direct link to Load data from a Kafka cluster") If the data of `file1.csv` is published to `topic1` of your Kafka cluster, execute the following statement to create a [Routine Load](https://docs.starrocks.io/docs/loading/RoutineLoad.md) job: ```sql CREATE ROUTINE LOAD test_db.table101 ON table1 COLUMNS TERMINATED BY ",", COLUMNS(user_id, user_gender, event_date, event_type) FROM KAFKA ( "kafka_broker_list" = ":", "kafka_topic" = "topic1", "property.kafka_default_offsets" = "OFFSET_BEGINNING" ); ``` > **NOTE** > > If you choose Routine Load, you must use the `COLUMNS` parameter to temporarily name the columns of the data file to create a column mapping between the data file and the StarRocks table. For detailed syntax and parameter descriptions, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). ##### Query data[​](#query-data "Direct link to Query data") After the load of data from your local file system, HDFS cluster, or Kafka cluster is complete, query the data of `table1` to verify that the load is successful: ```sql MySQL [test_db]> SELECT * FROM table1; +------------+------------+---------+ | event_date | event_type | user_id | +------------+------------+---------+ | 2020-05-22 | 1 | 576 | | 2020-05-20 | 1 | 354 | | 2020-05-21 | 2 | 465 | | 2020-05-23 | 2 | 687 | +------------+------------+---------+ 4 rows in set (0.01 sec) ``` #### Filter out rows that you do not want to load[​](#filter-out-rows-that-you-do-not-want-to-load "Direct link to Filter out rows that you do not want to load") When you load a data file into a StarRocks table, you may not want to load specific rows of the data file. In this situation, you can use the WHERE clause to specify the rows that you want to load. StarRocks filters out all rows that do not meet the filter conditions specified in the WHERE clause. This feature supports loading data from the following data sources: * Local file system * HDFS and cloud storage > **NOTE** > > This section uses HDFS as an example. * Kafka This section uses `file1.csv` and `table1` as an example. If you want to load only the rows whose event type is `1` from `file1.csv` into `table1`, you can use the WHERE clause to specify a filter condition `event_type = 1`. ##### Load data[​](#load-data-1 "Direct link to Load data") ###### Load data from a local file system[​](#load-data-from-a-local-file-system-1 "Direct link to Load data from a local file system") If `file1.csv` is stored in your local file system, run the following command to create a [Stream Load](https://docs.starrocks.io/docs/loading/StreamLoad.md) job: ```bash curl --location-trusted -u : \ -H "Expect:100-continue" \ -H "column_separator:," \ -H "columns: user_id, user_gender, event_date, event_type" \ -H "where: event_type=1" \ -T file1.csv -XPUT \ http://:/api/test_db/table1/_stream_load ``` For detailed syntax and parameter descriptions, see [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). ###### Load data from an HDFS cluster[​](#load-data-from-an-hdfs-cluster-1 "Direct link to Load data from an HDFS cluster") If `file1.csv` is stored in your HDFS cluster, execute the following statement to create a [Broker Load](https://docs.starrocks.io/docs/loading/hdfs_load.md) job: ```sql LOAD LABEL test_db.label2 ( DATA INFILE("hdfs://:/user/starrocks/data/input/file1.csv") INTO TABLE `table1` FORMAT AS "csv" COLUMNS TERMINATED BY "," (user_id, user_gender, event_date, event_type) WHERE event_type = 1 ) WITH BROKER; ``` For detailed syntax and parameter descriptions, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). ###### Load data from a Kafka cluster[​](#load-data-from-a-kafka-cluster-1 "Direct link to Load data from a Kafka cluster") If the data of `file1.csv` is published to `topic1` of your Kafka cluster, execute the following statement to create a [Routine Load](https://docs.starrocks.io/docs/loading/RoutineLoad.md) job: ```sql CREATE ROUTINE LOAD test_db.table102 ON table1 COLUMNS TERMINATED BY ",", COLUMNS (user_id, user_gender, event_date, event_type), WHERE event_type = 1 FROM KAFKA ( "kafka_broker_list" = ":", "kafka_topic" = "topic1", "property.kafka_default_offsets" = "OFFSET_BEGINNING" ); ``` For detailed syntax and parameter descriptions, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). ##### Query data[​](#query-data-1 "Direct link to Query data") After the load of data from your local file system, HDFS cluster, or Kafka cluster is complete, query the data of `table1` to verify that the load is successful: ```sql MySQL [test_db]> SELECT * FROM table1; +------------+------------+---------+ | event_date | event_type | user_id | +------------+------------+---------+ | 2020-05-20 | 1 | 354 | | 2020-05-22 | 1 | 576 | +------------+------------+---------+ 2 rows in set (0.01 sec) ``` #### Generate new columns from original columns[​](#generate-new-columns-from-original-columns "Direct link to Generate new columns from original columns") When you load a data file into a StarRocks table, some data of the data file may require conversions before the data can be loaded into the StarRocks table. In this situation, you can use functions or expressions in the job creation command or statement to implement data conversions. This feature supports loading data from the following data sources: * Local file system * HDFS and cloud storage > **NOTE** > > This section uses HDFS as an example. * Kafka This section uses `file2.csv` and `table2` as an example. `file2.csv` consists of only one column that represents date. You can use the [year](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/year.md), [month](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/month.md), and [day](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/day.md) functions to extract the year, month, and day in each date from `file2.csv` and load the extracted data into the `year`, `month`, and `day` columns of `table2`. ##### Load data[​](#load-data-2 "Direct link to Load data") ###### Load data from a local file system[​](#load-data-from-a-local-file-system-2 "Direct link to Load data from a local file system") If `file2.csv` is stored in your local file system, run the following command to create a [Stream Load](https://docs.starrocks.io/docs/loading/StreamLoad.md) job: ```bash curl --location-trusted -u : \ -H "Expect:100-continue" \ -H "column_separator:," \ -H "columns:date,year=year(date),month=month(date),day=day(date)" \ -T file2.csv -XPUT \ http://:/api/test_db/table2/_stream_load ``` > **NOTE** > > * In the `columns` parameter, you must first temporarily name **all columns** of the data file, and then temporarily name the new columns that you want to generate from the original columns of the data file. As shown in the preceding example, the only column of `file2.csv` is temporarily named as `date`, and then the `year=year(date)`, `month=month(date)`, and `day=day(date)` functions are invoked to generate three new columns, which are temporarily named as `year`, `month`, and `day`. > > * Stream Load does not support `column_name = function(column_name)` but supports `column_name = function(column_name)`. For detailed syntax and parameter descriptions, see [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). ###### Load data from an HDFS cluster[​](#load-data-from-an-hdfs-cluster-2 "Direct link to Load data from an HDFS cluster") If `file2.csv` is stored in your HDFS cluster, execute the following statement to create a [Broker Load](https://docs.starrocks.io/docs/loading/hdfs_load.md) job: ```sql LOAD LABEL test_db.label3 ( DATA INFILE("hdfs://:/user/starrocks/data/input/file2.csv") INTO TABLE `table2` FORMAT AS "csv" COLUMNS TERMINATED BY "," (date) SET(year=year(date), month=month(date), day=day(date)) ) WITH BROKER; ``` > **NOTE** > > You must first use the `column_list` parameter to temporarily name **all columns** of the data file, and then use the SET clause to temporarily name the new columns that you want to generate from the original columns of the data file. As shown in the preceding example, the only column of `file2.csv` is temporarily named as `date` in the `column_list` parameter, and then the `year=year(date)`, `month=month(date)`, and `day=day(date)` functions are invoked in the SET clause to generate three new columns, which are temporarily named as `year`, `month`, and `day`. For detailed syntax and parameter descriptions, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). ###### Load data from a Kafka cluster[​](#load-data-from-a-kafka-cluster-2 "Direct link to Load data from a Kafka cluster") If the data of `file2.csv` is published to `topic2` of your Kafka cluster, execute the following statement to create a [Routine Load](https://docs.starrocks.io/docs/loading/RoutineLoad.md) job: ```sql CREATE ROUTINE LOAD test_db.table201 ON table2 COLUMNS TERMINATED BY ",", COLUMNS(date,year=year(date),month=month(date),day=day(date)) FROM KAFKA ( "kafka_broker_list" = ":", "kafka_topic" = "topic2", "property.kafka_default_offsets" = "OFFSET_BEGINNING" ); ``` > **NOTE** > > In the `COLUMNS` parameter, you must first temporarily name **all columns** of the data file, and then temporarily name the new columns that you want to generate from the original columns of the data file. As shown in the preceding example, the only column of `file2.csv` is temporarily named as `date`, and then the `year=year(date)`, `month=month(date)`, and `day=day(date)` functions are invoked to generate three new columns, which are temporarily named as `year`, `month`, and `day`. For detailed syntax and parameter descriptions, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). ##### Query data[​](#query-data-2 "Direct link to Query data") After the load of data from your local file system, HDFS cluster, or Kafka cluster is complete, query the data of `table2` to verify that the load is successful: ```sql MySQL [test_db]> SELECT * FROM table2; +------------+------+-------+------+ | date | year | month | day | +------------+------+-------+------+ | 2020-05-20 | 2020 | 5 | 20 | | 2020-05-21 | 2020 | 5 | 21 | | 2020-05-22 | 2020 | 5 | 22 | | 2020-05-23 | 2020 | 5 | 23 | +------------+------+-------+------+ 4 rows in set (0.01 sec) ``` #### Extract partition field values from a file path[​](#extract-partition-field-values-from-a-file-path "Direct link to Extract partition field values from a file path") If the file path that you specify contains partition fields, you can use the `COLUMNS FROM PATH AS` parameter to specify the partition fields you want to extract from the file paths. The partition fields in file paths are equivalent to the columns in data files. The `COLUMNS FROM PATH AS` parameter is supported only when you load data from an HDFS cluster. For example, you want to load the following four data files generated from Hive: ```plain /user/starrocks/data/input/date=2020-05-20/data 1,354 /user/starrocks/data/input/date=2020-05-21/data 2,465 /user/starrocks/data/input/date=2020-05-22/data 1,576 /user/starrocks/data/input/date=2020-05-23/data 2,687 ``` The four data files are stored in the `/user/starrocks/data/input/` path of your HDFS cluster. Each of these data files is partitioned by partition field `date` and consists of two columns, which represent event type and user ID in sequence. ##### Load data from an HDFS cluster[​](#load-data-from-an-hdfs-cluster-3 "Direct link to Load data from an HDFS cluster") Execute the following statement to create a [Broker Load](https://docs.starrocks.io/docs/loading/hdfs_load.md) job, which enables you to extract the `date` partition field values from the `/user/starrocks/data/input/` file path and use a wildcard (\*) to specify that you want to load all data files in the file path to `table1`: ```sql LOAD LABEL test_db.label4 ( DATA INFILE("hdfs://:/user/starrocks/data/input/date=*/*") INTO TABLE `table1` FORMAT AS "csv" COLUMNS TERMINATED BY "," (event_type, user_id) COLUMNS FROM PATH AS (date) SET(event_date = date) ) WITH BROKER; ``` > **NOTE** > > In the preceding example, the `date` partition field in the specified file path is equivalent to the `event_date` column of `table1`. Therefore, you need to use the SET clause to map the `date` partition field onto the `event_date` column. If the partition field in the specified file path has the same name as a column of the StarRocks table, you do not need to use the SET clause to create a mapping. For detailed syntax and parameter descriptions, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). ##### Query data[​](#query-data-3 "Direct link to Query data") After the load of data from your HDFS cluster is complete, query the data of `table1` to verify that the load is successful: ```sql MySQL [test_db]> SELECT * FROM table1; +------------+------------+---------+ | event_date | event_type | user_id | +------------+------------+---------+ | 2020-05-22 | 1 | 576 | | 2020-05-20 | 1 | 354 | | 2020-05-21 | 2 | 465 | | 2020-05-23 | 2 | 687 | +------------+------------+---------+ 4 rows in set (0.01 sec) ``` --- ### Realtime synchronization from MySQL StarRocks supports multiple methods to synchronize data from MySQL to StarRocks in real time, delivering low latency real-time analytics of massive data. This topic describes how to synchronize data from MySQL to StarRocks in real-time (within seconds) through Apache Flink®. You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. #### How it works[​](#how-it-works "Direct link to How it works") tip Flink CDC is used in the synchronization from MySQL to Flink. This topic uses Flink CDC whose version is less than 3.0, so SMT is used to synchronize table schemas. However, if Flink CDC 3.0 is used, it is not necessary to use SMT to synchronize table schemas to StarRocks. Flink CDC 3.0 can even synchronize the schemas of the entire MySQL database, the sharded databases and tables, and also supports schema changes synchronization. For detailed usage, see [Streaming ELT from MySQL to StarRocks](https://nightlies.apache.org/flink/flink-cdc-docs-release-3.4/docs/get-started/quickstart/mysql-to-starrocks/). The following figure illustrates the entire synchronization process. ![img](/assets/images/4.9.2-ef6bd371aaac43f2d8b55e26499a3ee7.png) Real-time synchronization from MySQL through Flink to StarRocks is implemented in two stages: synchronizing database & table schema and synchronizing data. First, the SMT converts MySQL database & table schema into table creation statements for StarRocks. Then, the Flink cluster runs Flink jobs to synchronize full and incremental MySQL data to StarRocks. info The synchronization process guarantees exactly-once semantics. **Synchronization process**: 1. Synchronize database & table schema. The SMT reads the schema of the MySQL database & table to be synchronized and generates SQL files for creating a destination database & table in StarRocks. This operation is based on the MySQL and StarRocks information in SMT's configuration file. 2. Synchronize data. a. The Flink SQL client executes the data loading statement `INSERT INTO SELECT` to submit one or more Flink jobs to the Flink cluster. b. The Flink cluster runs the Flink jobs to obtain data. The Flink CDC connector first reads full historical data from the source database, then seamlessly switches to incremental reading, and sends the data to flink-connector-starrocks. c. flink-connector-starrocks accumulates data in mini-batches, and synchronizes each batch of data to StarRocks. info Only data manipulation language (DML) operations in MySQL can be synchronized to StarRocks. Data definition language (DDL) operations cannot be synchronized. #### Scenarios[​](#scenarios "Direct link to Scenarios") Real-time synchronization from MySQL has a broad range of use cases where data is constantly changed. Take a real-world use case "real-time ranking of commodity sales" as an example. Flink calculates the real-time ranking of commodity sales based on the original order table in MySQL and synchronizes the ranking to StarRocks' Primary Key table in real time. Users can connect a visualization tool to StarRocks to view the ranking in real time to gain on-demand operational insights. #### Preparations[​](#preparations "Direct link to Preparations") ##### Download and install synchronization tools[​](#download-and-install-synchronization-tools "Direct link to Download and install synchronization tools") To synchronize data from MySQL, you need to install the following tools: SMT, Flink, Flink CDC connector, and flink-connector-starrocks. 1. Download and install Flink, and start the Flink cluster. You can also perform this step by following the instructions in [Flink official documentation](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/try-flink/local_installation/). a. Install Java 8 or Java 11 in your operating system before you run Flink. You can run the following command to check the installed Java version. ```bash # View the Java version. java -version # Java 8 is installed if the following output is returned. java version "1.8.0_301" Java(TM) SE Runtime Environment (build 1.8.0_301-b09) Java HotSpot(TM) 64-Bit Server VM (build 25.301-b09, mixed mode) ``` b. Download the [Flink installation package](https://flink.apache.org/downloads/) and decompress it. We recommend that you use Flink 1.14 or later. The minimum allowed version is Flink 1.11. This topic uses Flink 1.14.5. ```bash # Download Flink. wget https://archive.apache.org/dist/flink/flink-1.14.5/flink-1.14.5-bin-scala_2.11.tgz # Decompress Flink. tar -xzf flink-1.14.5-bin-scala_2.11.tgz # Go to the Flink directory. cd flink-1.14.5 ``` c. Start the Flink cluster. ```bash # Start the Flink cluster. ./bin/start-cluster.sh # The Flink cluster is started if the following output is returned. Starting cluster. Starting standalonesession daemon on host. Starting taskexecutor daemon on host. ``` 2. Download [Flink CDC connector](https://github.com/ververica/flink-cdc-connectors/releases). This topic uses MySQL as the data source and therefore, `flink-sql-connector-mysql-cdc-x.x.x.jar` is downloaded. The connector version must match the [Flink](https://github.com/ververica/flink-cdc-connectors/releases) version. This topic uses Flink 1.14.5 and you can download `flink-sql-connector-mysql-cdc-2.2.0.jar`. ```bash wget https://repo1.maven.org/maven2/com/ververica/flink-sql-connector-mysql-cdc/2.1.1/flink-sql-connector-mysql-cdc-2.2.0.jar ``` 3. Download [flink-connector-starrocks](https://search.maven.org/artifact/com.starrocks/flink-connector-starrocks). The version must match the Flink version. > The flink-connector-starrocks package `x.x.x_flink-y.yy _ z.zz.jar` contains three version numbers: > > * `x.x.x` is the version number of flink-connector-starrocks. > * `y.yy` is the supported Flink version. > * `z.zz` is the Scala version supported by Flink. If the Flink version is 1.14.x or earlier, you must download a package that has the Scala version. > > This topic uses Flink 1.14.5 and Scala 2.11. Therefore, you can download the following package: `1.2.3_flink-14_2.11.jar`. 4. Move the JAR packages of Flink CDC connector (`flink-sql-connector-mysql-cdc-2.2.0.jar`) and flink-connector-starrocks (`1.2.3_flink-1.14_2.11.jar`) to the `lib` directory of Flink. > **Note** > > If a Flink cluster is already running in your system, you must stop the Flink cluster and restart it to load and validate the JAR packages. > > ```bash > ./bin/stop-cluster.sh > ./bin/start-cluster.sh > > ``` 5. Download and decompress the [SMT package](https://www.starrocks.io/download/community) and place it in the `flink-1.14.5` directory. StarRocks provides SMT packages for Linux x86 and macos ARM64. You can choose one based on your operating system and CPU. ```bash # for Linux x86 wget https://releases.starrocks.io/resources/smt.tar.gz # for macOS ARM64 wget https://releases.starrocks.io/resources/smt_darwin_arm64.tar.gz ``` ##### Enable MySQL binary log[​](#enable-mysql-binary-log "Direct link to Enable MySQL binary log") To synchronize data from MySQL in real time, the system needs to read data from MySQL binary log (binlog), parse the data, and then synchronize the data to StarRocks. Make sure that MySQL binary log is enabled. 1. Edit the MySQL configuration file `my.cnf` (default path: `/etc/my.cnf`) to enable MySQL binary log. ```bash # Enable MySQL Binlog. log_bin = ON # Configure the save path for the Binlog. log_bin =/var/lib/mysql/mysql-bin # Configure server_id. # If server_id is not configured for MySQL 5.7.3 or later, the MySQL service cannot be used. server_id = 1 # Set the Binlog format to ROW. binlog_format = ROW # The base name of the Binlog file. An identifier is appended to identify each Binlog file. log_bin_basename =/var/lib/mysql/mysql-bin # The index file of Binlog files, which manages the directory of all Binlog files. log_bin_index =/var/lib/mysql/mysql-bin.index ``` 2. Run one of the following commands to restart MySQL for the modified configuration file to take effect. ```bash # Use service to restart MySQL. service mysqld restart # Use mysqld script to restart MySQL. /etc/init.d/mysqld restart ``` 3. Connect to MySQL and check whether MySQL binary log is enabled. ```plain -- Connect to MySQL. mysql -h xxx.xx.xxx.xx -P 3306 -u root -pxxxxxx -- Check whether MySQL binary log is enabled. mysql> SHOW VARIABLES LIKE 'log_bin'; +---------------+-------+ | Variable_name | Value | +---------------+-------+ | log_bin | ON | +---------------+-------+ 1 row in set (0.00 sec) ``` #### Synchronize database & table schema[​](#synchronize-database--table-schema "Direct link to Synchronize database & table schema") 1. Edit the SMT configuration file. Go to the SMT `conf` directory and edit the configuration file `config_prod.conf`, such as MySQL connection information, the matching rules of the database & table to be synchronized, and configuration information of flink-connector-starrocks. ```bash [db] type = mysql host = xxx.xx.xxx.xx port = 3306 user = user1 password = xxxxxx [other] # Number of BEs in StarRocks be_num = 3 # `decimal_v3` is supported since StarRocks-1.18.1. use_decimal_v3 = true # File to save the converted DDL SQL output_dir = ./result [table-rule.1] # Pattern to match databases for setting properties database = ^demo.*$ # Pattern to match tables for setting properties table = ^.*$ ############################################ ### Flink sink configurations ### DO NOT set `connector`, `table-name`, `database-name`. They are auto-generated. ############################################ flink.starrocks.jdbc-url=jdbc:mysql://: flink.starrocks.load-url= : flink.starrocks.username=user2 flink.starrocks.password=xxxxxx flink.starrocks.sink.properties.format=csv flink.starrocks.sink.properties.column_separator=\x01 flink.starrocks.sink.properties.row_delimiter=\x02 flink.starrocks.sink.buffer-flush.interval-ms=15000 ``` * `[db]`: information used to access the source database. * `type`: type of the source database. In this topic, the source database is `mysql`. * `host`: IP address of the MySQL server. * `port`: port number of the MySQL database, defaults to `3306` * `user`: username for accessing the MySQL database * `password`: password of the username * `[table-rule]`: database & table matching rules and the corresponding flink-connector-starrocks configuration. * `Database`, `table`: the names of the database & table in MySQL. Regular expressions are supported. * `flink.starrocks.*`: configuration information of flink-connector-starrocks. For more configurations and information, see [flink-connector-starrocks](https://docs.starrocks.io/docs/loading/Flink-connector-starrocks.md). > If you need to use different flink-connector-starrocks configurations for different tables. For example, if some tables are frequently updated and you need to accelerate data loading, see [Use different flink-connector-starrocks configurations for different tables](#use-different-flink-connector-starrocks-configurations-for-different-tables). If you need to load multiple tables obtained from MySQL sharding into the same StarRocks table, see [Synchronize multiple tables after MySQL sharding to one table in StarRocks](#synchronize-multiple-tables-after-mysql-sharding-to-one-table-in-starrocks). * `[other]`: other information * `be_num`: The number of BEs in your StarRocks cluster (This parameter will be used for setting a reasonable number of tablets in subsequent StarRocks table creation). * `use_decimal_v3`: Whether to enable [Decimal V3](https://docs.starrocks.io/docs/sql-reference/data-types/numeric/DECIMAL.md). After Decimal V3 is enabled, MySQL decimal data will be converted into Decimal V3 data when data is synchronized to StarRocks. * `output_dir`: The path to save the SQL files to be generated. The SQL files will be used to create a database & table in StarRocks and submit a Flink job to the Flink cluster. The default path is `./result` and we recommend that you retain the default settings. 2. Run the SMT to read the database & table schema in MySQL and generate SQL files in the `./result` directory based on the configuration file. The `starrocks-create.all.sql` file is used to create a database & table in StarRocks and the `flink-create.all.sql` file is used to submit a Flink job to the Flink cluster. ```bash # Run the SMT. ./starrocks-migrate-tool # Go to the result directory and check the files in this directory. cd result ls result flink-create.1.sql smt.tar.gz starrocks-create.all.sql flink-create.all.sql starrocks-create.1.sql ``` 3. Run the following command to connect to StarRocks and execute the `starrocks-create.all.sql` file to create a database and table in StarRocks. We recommend that you use the default table creation statement in the SQL file to create a table of the [Primary Key table](https://docs.starrocks.io/docs/table_design/table_types/primary_key_table.md). > **Note** > > You can also modify the table creation statement based on your business needs and create a table that does not use the Primary Key table. However, the DELETE operation in the source MySQL database cannot be synchronized to the non- Primary Key table. Exercise caution when you create such a table. ```bash mysql -h -P -u user2 -pxxxxxx < starrocks-create.all.sql ``` If the data needs to be processed by Flink before it is written to the destination StarRocks table, the table schema will be different between the source and destination tables. In this case, you must modify the table creation statement. In this example, the destination table requires only the `product_id` and `product_name` columns and real-time ranking of commodity sales. You can use the following table creation statement. ```bash CREATE DATABASE IF NOT EXISTS `demo`; CREATE TABLE IF NOT EXISTS `demo`.`orders` ( `product_id` INT(11) NOT NULL COMMENT "", `product_name` STRING NOT NULL COMMENT "", `sales_cnt` BIGINT NOT NULL COMMENT "" ) ENGINE=olap PRIMARY KEY(`product_id`) DISTRIBUTED BY HASH(`product_id`) PROPERTIES ( "replication_num" = "3" ); ``` > **NOTICE** > > Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). #### Synchronize data[​](#synchronize-data "Direct link to Synchronize data") Run the Flink cluster and submit a Flink job to continuously synchronize full and incremental data from MySQL to StarRocks. 1. Go to the Flink directory and run the following command to run the `flink-create.all.sql` file on your Flink SQL client. ```bash ./bin/sql-client.sh -f flink-create.all.sql ``` This SQL file defines dynamic tables `source table` and `sink table`, query statement `INSERT INTO SELECT`, and specifies the connector, source database, and destination database. After this file is executed, a Flink job is submitted to the Flink cluster to start data synchronization. > **Note** > > * Make sure that the Flink cluster has been started. You can start the Flink cluster by running `flink/bin/start-cluster.sh`. > * If your Flink version is earlier than 1.13, you may not be able to directly run the SQL file `flink-create.all.sql`. You need to execute SQL statements one by one in this file in the command line interface (CLI) of the SQL client. You also need to escape the `\` character. > > ```bash > 'sink.properties.column_separator' = '\\x01' > 'sink.properties.row_delimiter' = '\\x02' > > ``` **Process data during synchronization**: If you need to process data during synchronization, such as performing GROUP BY or JOIN on the data, you can modify the `flink-create.all.sql` file. The following example calculates real-time ranking of commodity sales by executing COUNT (\*) and GROUP BY. ```bash $ ./bin/sql-client.sh -f flink-create.all.sql No default environment is specified. Searching for '/home/disk1/flink-1.13.6/conf/sql-client-defaults.yaml'...not found. [INFO] Executing SQL from file. Flink SQL> CREATE DATABASE IF NOT EXISTS `default_catalog`.`demo`; [INFO] Execute statement succeed. -- Create a dynamic table `source table` based on the order table in MySQL. Flink SQL> CREATE TABLE IF NOT EXISTS `default_catalog`.`demo`.`orders_src` (`order_id` BIGINT NOT NULL, `product_id` INT NULL, `order_date` TIMESTAMP NOT NULL, `customer_name` STRING NOT NULL, `product_name` STRING NOT NULL, `price` DECIMAL(10, 5) NULL, PRIMARY KEY(`order_id`) NOT ENFORCED ) with ('connector' = 'mysql-cdc', 'hostname' = 'xxx.xx.xxx.xxx', 'port' = '3306', 'username' = 'root', 'password' = '', 'database-name' = 'demo', 'table-name' = 'orders' ); [INFO] Execute statement succeed. -- Create a dynamic table `sink table`. Flink SQL> CREATE TABLE IF NOT EXISTS `default_catalog`.`demo`.`orders_sink` (`product_id` INT NOT NULL, `product_name` STRING NOT NULL, `sales_cnt` BIGINT NOT NULL, PRIMARY KEY(`product_id`) NOT ENFORCED ) with ('sink.max-retries' = '10', 'jdbc-url' = 'jdbc:mysql://:', 'password' = '', 'sink.properties.strip_outer_array' = 'true', 'sink.properties.format' = 'json', 'load-url' = ':', 'username' = 'root', 'sink.buffer-flush.interval-ms' = '15000', 'connector' = 'starrocks', 'database-name' = 'demo', 'table-name' = 'orders' ); [INFO] Execute statement succeed. -- Implement real-time ranking of commodity sales, where `sink table` is dynamically updated to reflect data changes in `source table`. Flink SQL> INSERT INTO `default_catalog`.`demo`.`orders_sink` select product_id,product_name, count(*) as cnt from `default_catalog`.`demo`.`orders_src` group by product_id,product_name; [INFO] Submitting SQL update statement to the cluster... [INFO] SQL update statement has been successfully submitted to the cluster: Job ID: 5ae005c4b3425d8bb13fe660260a35da ``` If you only need to synchronize only a portion of the data, such as data whose payment time is later than December 21, 2021, you can use the `WHERE` clause in `INSERT INTO SELECT` to set a filter condition, such as `WHERE pay_dt > '2021-12-21'`. Data that does not meet this condition will not be synchronized to StarRocks. If the following result is returned, the Flink job has been submitted for full and incremental synchronization. ```sql [INFO] Submitting SQL update statement to the cluster... [INFO] SQL update statement has been successfully submitted to the cluster: Job ID: 5ae005c4b3425d8bb13fe660260a35da ``` 2. You can use the [Flink WebUI](https://nightlies.apache.org/flink/flink-docs-release-2.2/docs/try-flink/flink-operations-playground/#flink-webui) or run the `bin/flink list -running` command on your Flink SQL client to view Flink jobs that are running in the Flink cluster and the job IDs. * Flink WebUI ![img](/assets/images/4.9.3-9e843337839f2cf335718301cd414239.png) * `bin/flink list -running` ```bash $ bin/flink list -running Waiting for response... ------------------ Running/Restarting Jobs ------------------- 13.10.2022 15:03:54 : 040a846f8b58e82eb99c8663424294d5 : insert-into_default_catalog.lily.example_tbl1_sink (RUNNING) -------------------------------------------------------------- ``` > **Note** > > If the job is abnormal, you can perform troubleshooting by using Flink WebUI or by viewing the log file in the `/log` directory of Flink 1.14.5. #### FAQ[​](#faq "Direct link to FAQ") ##### Use different flink-connector-starrocks configurations for different tables[​](#use-different-flink-connector-starrocks-configurations-for-different-tables "Direct link to Use different flink-connector-starrocks configurations for different tables") If some tables in the data source are frequently updated and you want to accelerate the loading speed of flink-connector-starrocks, you must set a separate flink-connector-starrocks configuration for each table in the SMT configuration file `config_prod.conf`. ```bash [table-rule.1] # Pattern to match databases for setting properties database = ^order.*$ # Pattern to match tables for setting properties table = ^.*$ ############################################ ### Flink sink configurations ### DO NOT set `connector`, `table-name`, `database-name`. They are auto-generated ############################################ flink.starrocks.jdbc-url=jdbc:mysql://: flink.starrocks.load-url= : flink.starrocks.username=user2 flink.starrocks.password=xxxxxx flink.starrocks.sink.properties.format=csv flink.starrocks.sink.properties.column_separator=\x01 flink.starrocks.sink.properties.row_delimiter=\x02 flink.starrocks.sink.buffer-flush.interval-ms=15000 [table-rule.2] # Pattern to match databases for setting properties database = ^order2.*$ # Pattern to match tables for setting properties table = ^.*$ ############################################ ### Flink sink configurations ### DO NOT set `connector`, `table-name`, `database-name`. They are auto-generated ############################################ flink.starrocks.jdbc-url=jdbc:mysql://: flink.starrocks.load-url= : flink.starrocks.username=user2 flink.starrocks.password=xxxxxx flink.starrocks.sink.properties.format=csv flink.starrocks.sink.properties.column_separator=\x01 flink.starrocks.sink.properties.row_delimiter=\x02 flink.starrocks.sink.buffer-flush.interval-ms=10000 ``` ##### Synchronize multiple tables after MySQL sharding to one table in StarRocks[​](#synchronize-multiple-tables-after-mysql-sharding-to-one-table-in-starrocks "Direct link to Synchronize multiple tables after MySQL sharding to one table in StarRocks") After sharding is performed, data in one MySQL table may be split into multiple tables or even distributed to multiple databases. All the tables have the same schema. In this case, you can set `[table-rule]` to synchronize these tables to one StarRocks table. For example, MySQL has two databases `edu_db_1` and `edu_db_2`, each of which has two tables `course_1 and course_2`, and the schema of all tables is the same. You can use the following `[table-rule]` configuration to synchronize all the tables to one StarRocks table. > **Note** > > The name of the StarRocks table defaults to `course__auto_shard`. If you need to use a different name, you can modify it in the SQL files `starrocks-create.all.sql` and `flink-create.all.sql` ```bash [table-rule.1] # Pattern to match databases for setting properties database = ^edu_db_[0-9]*$ # Pattern to match tables for setting properties table = ^course_[0-9]*$ ############################################ ### Flink sink configurations ### DO NOT set `connector`, `table-name`, `database-name`. They are auto-generated ############################################ flink.starrocks.jdbc-url = jdbc: mysql://xxx.xxx.x.x:xxxx flink.starrocks.load-url = xxx.xxx.x.x:xxxx flink.starrocks.username = user2 flink.starrocks.password = xxxxxx flink.starrocks.sink.properties.format=csv flink.starrocks.sink.properties.column_separator =\x01 flink.starrocks.sink.properties.row_delimiter =\x02 flink.starrocks.sink.buffer-flush.interval-ms = 5000 ``` ##### Import data in JSON format[​](#import-data-in-json-format "Direct link to Import data in JSON format") Data in the preceding example is imported in CSV format. If you are unable to choose a suitable delimiter, you need to replace the following parameters of `flink.starrocks.*` in `[table-rule]`. ```plain flink.starrocks.sink.properties.format=csv flink.starrocks.sink.properties.column_separator =\x01 flink.starrocks.sink.properties.row_delimiter =\x02 ``` Data is imported in JSON format after the following parameters are passed in. ```plain flink.starrocks.sink.properties.format=json flink.starrocks.sink.properties.strip_outer_array=true ``` > **Note** > > This method slightly slows down the loading speed. ##### Execute multiple INSERT INTO statements as one Flink job[​](#execute-multiple-insert-into-statements-as-one-flink-job "Direct link to Execute multiple INSERT INTO statements as one Flink job") You can use the [STATEMENT SET](https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sqlclient/#execute-a-set-of-sql-statements) syntax in the `flink-create.all.sql` file to execute multiple INSERT INTO statements as one Flink job, which prevents multiple statements from taking up too many Flink job resources and improves the efficiency of executing multiple queries. > **Note** > > Flink supports the STATEMENT SET syntax from 1.13 onwards. 1. Open the `result/flink-create.all.sql` file. 2. Modify the SQL statements in the file. Move all the INSERT INTO statements to the end of the file. Place `EXECUTE STATEMENT SET BEGIN` before the first INSERT INTO statement and place `END;` after the last INSERT INTO statement. > **Note** > > The positions of CREATE DATABASE and CREATE TABLE remain unchanged. ```sql CREATE DATABASE IF NOT EXISTS db; CREATE TABLE IF NOT EXISTS db.a1; CREATE TABLE IF NOT EXISTS db.b1; CREATE TABLE IF NOT EXISTS db.a2; CREATE TABLE IF NOT EXISTS db.b2; EXECUTE STATEMENT SET BEGIN-- one or more INSERT INTO statements INSERT INTO db.a1 SELECT * FROM db.b1; INSERT INTO db.a2 SELECT * FROM db.b2; END; ``` --- ### Load data from Apache Flink® with Multi-table Transaction StarRocks Flink Connector supports Multi-table Transaction to load data from Flink into multiple tables atomically. #### Use Cases[​](#use-cases "Direct link to Use Cases") When a single Flink job writes to multiple tables within the same StarRocks database in one processing cycle, enabling multi-table transaction guarantees: * **Cross-table atomic commit**: Data written to different table within the same commit cycle becomes visible atomically — all or nothing. * **Source transaction integrity**: A complete upstream transaction (for example, from Kafka) is never split across two StarRocks transactions. * **Sub-second data freshness**: Data continuously flows into StarRocks via `/api/transaction/load`, and is committed at the interval configured by `sink.buffer-flush.interval-ms`. Typical scenarios: * Synchronous writes to a general table and a detail table (for example, `orders` and `order_items`) * Event routing to different partition tables (for example, `events_202601`, `events_202602`) * A single job maintaining multiple interrelated downstream result tables Prerequisites To enable Multi-table Transaction, you must running your cluster on StarRocks v4.0 and later (with the Multi-table Transaction Stream Load support), and StarRocks Flink Connector on v1.2.9 and later. #### Core Capabilities[​](#core-capabilities "Direct link to Core Capabilities") | Capability | Description | | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | Cross-table atomic commit | All tables within the same flush cycle share one StarRocks transaction label. The Prepare and Commit operations are unified. | | Source transaction integrity | Commit timing is controlled by the `transactionEnd` flag. Commit only occurs at complete source transaction boundaries. | | Sub-second data visibility | Data is periodically flushed to StarRocks (`/api/transaction/load`). It is committed when the `transactionEnd` and the timer conditions are met | | N:1 transaction mapping | Multiple source transactions can accumulate in a single StarRocks transaction. They do not have to be mapped 1:1. | | Within-partition ordering | `keyBy(sourcePartition)` ensures transactions from the same partition are processed in order within the same sink subtask. | #### Configurations[​](#configurations "Direct link to Configurations") ##### Multi-table Transaction Configurations[​](#multi-table-transaction-configurations "Direct link to Multi-table Transaction Configurations") ###### `sink.transaction.multi-table.enabled`[​](#sinktransactionmulti-tableenabled "Direct link to sinktransactionmulti-tableenabled") * Type: Boolean * Default: `false` * Description: Whether to enable the Multi-table Atomic Transaction mode. ###### `sink.transaction.multi-table.buffer-size`[​](#sinktransactionmulti-tablebuffer-size "Direct link to sinktransactionmulti-tablebuffer-size") * Type: Long * Default: `134217728` (128 MB) * Unit: Bytes * Description: Global buffer size in bytes for the Multi-table Transaction mode. When the total buffered data across all tables reaches this threshold, a flush is triggered. ##### Load-related Configurations[​](#load-related-configurations "Direct link to Load-related Configurations") ###### `sink.version`[​](#sinkversion "Direct link to sinkversion") * Recommended Value: `V2` * Description: Required. `V1` does not support the transaction Stream Load interface. ###### `sink.semantic`[​](#sinksemantic "Direct link to sinksemantic") * Recommended Value: `at-least-once` * Description: Multi-table mode currently supports `at-least-once` only. ###### `database-name`[​](#database-name "Direct link to database-name") * Recommended Value: `*` * Description: Wildcard to enable dynamic multi-table routing. ###### `table-name`[​](#table-name "Direct link to table-name") * Recommended Value: `*` * Description: Wildcard to enable dynamic multi-table routing. ###### `sink.buffer-flush.interval-ms`[​](#sinkbuffer-flushinterval-ms "Direct link to sinkbuffer-flushinterval-ms") * Recommended Value: `1000` * Description: Controls the commit cycle. You can set it to `1000` to achieve the freshness of approximately one second. ###### `sink.properties.format`[​](#sinkpropertiesformat "Direct link to sinkpropertiesformat") * Recommended Value: `json` * Description: The data format. ###### `sink.properties.strip_outer_array`[​](#sinkpropertiesstrip_outer_array "Direct link to sinkpropertiesstrip_outer_array") * Recommended Value: `true` * Description: Whether to strip the outermost array structure. #### Interfaces[​](#interfaces "Direct link to Interfaces") ##### `StarRocksRowData`[​](#starrocksrowdata "Direct link to starrocksrowdata") ```java public interface StarRocksRowData { String getUniqueKey(); // Region routing key (nullable; auto-derived from database.table) String getDatabase(); // Target database String getTable(); // Target table String getRow(); // Row data in JSON format /** * Indicates this is the last row of a source transaction batch. * Used by multi-table transaction mode to determine safe commit points: * the connector only commits when the most recent write had this flag set, * ensuring no partial source transaction is committed. */ default boolean isTransactionEnd() { return false; } /** * Returns the source partition ID for this row. * Used by multi-table transaction mode to track per-partition transaction * boundaries. Returns -1 when partition tracking is not applicable. */ default int getSourcePartition() { return -1; } } ``` ##### `DefaultStarRocksRowData`[​](#defaultstarrocksrowdata "Direct link to defaultstarrocksrowdata") ```java public class DefaultStarRocksRowData implements StarRocksRowData { // Basic fields private String uniqueKey; private String database; private String table; private String row; // Multi-table transaction fields private boolean transactionEnd; // Source transaction end marker private int sourcePartition = -1; // Source partition ID (for keyBy ordering) // Constructors public DefaultStarRocksRowData(); public DefaultStarRocksRowData(String database, String table); public DefaultStarRocksRowData(String uniqueKey, String database, String table, String row); // Setters public void setUniqueKey(String uniqueKey); public void setDatabase(String database); public void setTable(String table); public void setRow(String row); public void setTransactionEnd(boolean transactionEnd); public void setSourcePartition(int sourcePartition); // Getters (inherited from StarRocksRowData) public String getUniqueKey(); public String getDatabase(); public String getTable(); public String getRow(); public boolean isTransactionEnd(); public int getSourcePartition(); } ``` ##### User-Implemented Component[​](#user-implemented-component "Direct link to User-Implemented Component") Users need to implement a `KeyedProcessFunction` (referred to as `TransactionAssembler` in this document) that: 1. Keys by source partition and buffers data rows within a transaction 2. Emits all rows only when the source transaction is closed (for example, upon receiving `TXN_END`) 3. Sets `transactionEnd=true` on the last row 4. Sets `sourcePartition` on every row No custom `SinkFunction` is needed — the standard connector API (`SinkFunctionFactory.createSinkFunction()`) handles everything. #### Complete Example[​](#complete-example "Direct link to Complete Example") ##### StarRocks Table DDL[​](#starrocks-table-ddl "Direct link to StarRocks Table DDL") ```sql CREATE DATABASE `test`; CREATE TABLE `test`.`orders` ( `order_id` BIGINT NOT NULL, `customer_id` BIGINT NOT NULL, `total_amount` DECIMAL(10,2) DEFAULT "0", `order_status` VARCHAR(32) DEFAULT "" ) ENGINE=OLAP PRIMARY KEY(`order_id`) DISTRIBUTED BY HASH(`order_id`) PROPERTIES("replication_num" = "1"); CREATE TABLE `test`.`order_items` ( `item_id` BIGINT NOT NULL, `order_id` BIGINT NOT NULL, `product_name` VARCHAR(128) DEFAULT "", `quantity` INT DEFAULT "0", `price` DECIMAL(10,2) DEFAULT "0" ) ENGINE=OLAP PRIMARY KEY(`item_id`) DISTRIBUTED BY HASH(`item_id`) PROPERTIES("replication_num" = "1"); ``` ##### Flink Job Code[​](#flink-job-code "Direct link to Flink Job Code") ```java import com.starrocks.connector.flink.table.data.DefaultStarRocksRowData; import com.starrocks.connector.flink.table.sink.SinkFunctionFactory; import com.starrocks.connector.flink.table.sink.StarRocksSinkOptions; import com.starrocks.data.load.stream.properties.StreamLoadTableProperties; import org.apache.flink.api.common.state.ListState; import org.apache.flink.api.common.state.ListStateDescriptor; import org.apache.flink.api.common.typeinfo.Types; import org.apache.flink.api.java.utils.MultipleParameterTool; import org.apache.flink.configuration.Configuration; import org.apache.flink.streaming.api.datastream.DataStream; import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; import org.apache.flink.streaming.api.functions.KeyedProcessFunction; import org.apache.flink.streaming.api.functions.sink.SinkFunction; import org.apache.flink.streaming.api.functions.source.RichParallelSourceFunction; import org.apache.flink.util.Collector; import java.io.Serializable; import java.util.ArrayList; import java.util.List; public class WriteMultipleTablesWithTransaction { // ===================================================== // 1. Source Event Model (define per your business logic) // ===================================================== enum EventType { TXN_BEGIN, DATA, TXN_END } static class TxnEvent implements Serializable { private static final long serialVersionUID = 1L; int partition; String txnId; EventType type; String database; String table; String json; TxnEvent() {} TxnEvent(int partition, String txnId, EventType type, String database, String table, String json) { this.partition = partition; this.txnId = txnId; this.type = type; this.database = database; this.table = table; this.json = json; } static TxnEvent begin(int partition, String txnId) { return new TxnEvent(partition, txnId, EventType.TXN_BEGIN, null, null, null); } static TxnEvent data(int partition, String txnId, String db, String table, String json) { return new TxnEvent(partition, txnId, EventType.DATA, db, table, json); } static TxnEvent end(int partition, String txnId) { return new TxnEvent(partition, txnId, EventType.TXN_END, null, null, null); } } // ===================================================== // 2. TransactionAssembler — Core User Component // ===================================================== /** * Buffers DATA events per partition; on TXN_END emits the complete * transaction's rows as individual DefaultStarRocksRowData records. * * Only emits when a source transaction is fully closed (TXN_END received). * All rows are emitted synchronously within one processElement() call, * so they enter the downstream sink buffer without intervening checkpoint barriers. * * Multiple source transactions accumulate in the sink's buffer between * flush cycles — the connector handles grouping them into StarRocks transactions. */ static class TransactionAssembler extends KeyedProcessFunction { private transient ListState pendingRows; @Override public void open(Configuration parameters) throws Exception { ListStateDescriptor descriptor = new ListStateDescriptor<>( "pending-txn-rows", Types.POJO(TxnEvent.class)); pendingRows = getRuntimeContext().getListState(descriptor); } @Override public void processElement(TxnEvent event, Context ctx, Collector out) throws Exception { switch (event.type) { case TXN_BEGIN: pendingRows.clear(); break; case DATA: pendingRows.add(event); break; case TXN_END: List rows = new ArrayList<>(); for (TxnEvent row : pendingRows.get()) { rows.add(row); } int partition = ctx.getCurrentKey(); for (int i = 0; i < rows.size(); i++) { TxnEvent row = rows.get(i); DefaultStarRocksRowData rowData = new DefaultStarRocksRowData( null, row.database, row.table, row.json); rowData.setSourcePartition(partition); // Mark the last row as transaction end if (i == rows.size() - 1) { rowData.setTransactionEnd(true); } out.collect(rowData); } pendingRows.clear(); break; } } } // ===================================================== // 3. Main Program // ===================================================== public static void main(String[] args) throws Exception { MultipleParameterTool params = MultipleParameterTool.fromArgs(args); String jdbcUrl = params.get("jdbcUrl", "jdbc:mysql://127.0.0.1:9030"); String loadUrl = params.get("loadUrl", "127.0.0.1:8030"); String userName = params.get("userName", "root"); String password = params.get("password", ""); StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); env.enableCheckpointing(10_000); // Checkpoint is for recovery only; does not affect commit cycle // --- Source --- // Replace with an actual Kafka Source that deserializes into TxnEvent DataStream events = env.addSource(/* KafkaSource or MockTxnEventSource */); // --- Step 1: Assemble complete transactions per partition --- // TransactionAssembler emits rows only after TXN_END. // Multiple closed source txns accumulate in the sink buffer between flushes. DataStream rows = events .keyBy(e -> e.partition) .process(new TransactionAssembler()); // --- Step 2: Partition-affinity routing to sink --- // keyBy(sourcePartition) routes same-partition data to the same sink subtask. // The connector uses per-partition regions internally, so even when multiple // partitions land on the same sink subtask, transaction boundaries are tracked // independently per partition via PartitionCommitTracker. DataStream partitionedRows = rows .keyBy(DefaultStarRocksRowData::getSourcePartition); // --- Step 3: Configure the connector --- // sink.transaction.multi-table.enabled=true activates per-partition region // tracking inside StreamLoadManagerV2: each partition's regions are switched // independently when its txnEnd arrives, and commit triggers only when all // active partitions have been switched. StarRocksSinkOptions options = StarRocksSinkOptions.builder() .withProperty("jdbc-url", jdbcUrl) .withProperty("load-url", loadUrl) .withProperty("database-name", "*") // Wildcard for dynamic multi-table routing .withProperty("table-name", "*") // Wildcard for dynamic multi-table routing .withProperty("username", userName) .withProperty("password", password) .withProperty("sink.version", "V2") // Required: V2 .withProperty("sink.semantic", "at-least-once") .withProperty("sink.transaction.multi-table.enabled", "true") // Enable multi-table txn .withProperty("sink.buffer-flush.interval-ms", "1000") // ~1s data freshness .withProperty("sink.properties.format", "json") .withProperty("sink.properties.strip_outer_array", "true") .build(); // Optional: per-table stream load properties StreamLoadTableProperties orderItemsProps = StreamLoadTableProperties.builder() .database("test") .table("order_items") .addProperty("format", "json") .addProperty("strip_outer_array", "true") .addProperty("ignore_json_size", "true") .build(); options.addTableProperties(orderItemsProps); // --- Step 4: Create and attach the sink --- // Standard connector API — no custom SinkFunction needed. // addSink on the keyBy'd stream ensures partition affinity. SinkFunction sink = SinkFunctionFactory.createSinkFunction(options); partitionedRows.addSink(sink); env.execute("WriteMultipleTablesWithTransaction"); } } ``` ##### Data Flow Topology[​](#data-flow-topology "Direct link to Data Flow Topology") ```text Kafka (60 partitions) | v keyBy(partition) ———— Ensures same-partition events go to the same subtask | v TransactionAssembler (KeyedProcessFunction) | Buffers DATA events | On TXN_END: emits all rows (last row has transactionEnd=true) | Every row carries sourcePartition | v keyBy(sourcePartition) ———— Ensures same-partition rows go to the same sink subtask | v StarRocksDynamicSinkFunctionV2 (via SinkFunctionFactory.createSinkFunction) | | +——————————— StreamLoadManagerV2 (multi-table txn mode) —————————-——+ | | | | | Per-partition, per-table regions: | | | Region(P0, orders), Region(P0, order_items) | | | Region(P2, orders), Region(P2, order_items) | | | | | | Each region tracks: | | | - activeChunk / inactiveChunks | | | - lastSwitchTimeMs (for miniInterval batching) | | | - activeChunkCleanBoundary (true iff last task event is txnEnd)| | | | | | write(partition, db, table, row) [task thread] | | | -> routes to Region(partition, db, table) | | | -> write0 sets activeChunkCleanBoundary = false | | | -> In multi-table mode write0 does NOT switchChunk, so | | | activeChunk only freezes at a txnEnd boundary | | | | | | setCommitAllowed(partition, txnEnd=true) [task thread] | | | -> region.tryMiniIntervalSwitch(): | | | sets activeChunkCleanBoundary = true | | | if (now - lastSwitchTimeMs >= miniInterval | | | && activeChunk has data): switchChunkForCommit | | | else: data batches into activeChunk with subsequent | | | completed source transactions (N:1 mapping) | | | -> PartitionCommitTracker.onTxnEnd(partition) | | | | | | SharedTransactionCoordinator: | | | -> eagerly opens shared txn before any flush | | | -> all autonomous flushes use the shared label | | | -> recycles idle txn at 80% of server timeout | | | | | | Manager thread (every scanningFrequency): | | | -> tryForceCleanSwitch per region: | | | if cleanBoundary && has data && miniInterval elapsed | | | -> switchChunkForCommit (source-idle fallback) | | | -> tryStartTimerDrivenCommit: | | | if commitInterval elapsed && hasDataLoaded | | | -> set commitInFlight = true | | | -> autonomous flush: drain inactiveChunks via streamLoad | | | (never touches activeChunk in multi-table mode) | | | | | | Manager thread (commitInFlight=true): | | | -> triggerLoadIfNeeded per region (HTTP /api/transaction/load) | | | -> wait all loads complete | | | -> unified commit via SharedTransactionCoordinator | | | -> reset tracker; open next shared txn | | +———————————————————————————————————————————————————————————————————+ | v StarRocks (test.orders + test.order_items) ``` #### How It Works[​](#how-it-works "Direct link to How It Works") ##### Chunk Lifecycle and `miniInterval` Batching[​](#chunk-lifecycle-and-miniinterval-batching "Direct link to chunk-lifecycle-and-miniinterval-batching") Each `(partition, table)` region has one `activeChunk` (currently accepting writes) and a FIFO of `inactiveChunks` (frozen data, pending HTTP load). In multi-table transaction mode, the **only** way data moves from `activeChunk` into `inactiveChunks` is via `switchChunkForCommit`, which is called from exactly three sites: 1. **Task thread, on txnEnd** — via `region.tryMiniIntervalSwitch()` inside `setCommitAllowed(partition, true)`. This is the common path. 2. **Manager thread, source-idle fallback** — via `region.tryForceCleanSwitch()` on every scan cycle, for regions whose `activeChunk` has been sitting idle at a clean transaction boundary. 3. **Manager thread, savepoint/recycle** — force-switch all regions after verifying every region is at a clean boundary. To avoid one HTTP load per source transaction in high-throughput CDC, the task thread only performs a switch when at least `miniSwitchIntervalMs` has elapsed since the previous switch on the same region. `miniSwitchIntervalMs` is computed as `min(1000 ms, max(100 ms, commitInterval / 10))`, so a 1-second commit interval batches at 100 ms while a 30-second interval caps batching at 1 second. Within a miniInterval window, multiple completed source transactions accumulate into the same `activeChunk` (N:1 mapping) and are frozen together on the next switch. Each region carries two fields that drive these decisions: * `lastSwitchTimeMs` — epoch ms of the most recent switch. Initially 0, so the very first txnEnd after region creation always triggers a switch. * `activeChunkCleanBoundary` — `true` if the most recent task-thread event on this region was either an `onTxnEnd` or a `switchChunk`. `false` after any `write()`. The manager thread's `tryForceCleanSwitch` only runs when this flag is `true`, so it can never freeze partial source-transaction data. ##### Task Thread — Write and txnEnd[​](#task-thread--write-and-txnend "Direct link to Task Thread — Write and txnEnd") ```text invoke(record) [Flink task thread] | | if record is a data row: | write(partition, db, table, row) | region.write(row) → addRow(); cleanBoundary = false | | if record carries transactionEnd=true: | setCommitAllowed(partition, true) | for each region owned by this partition: | region.tryMiniIntervalSwitch(): | cleanBoundary = true // always (txnEnd observed) | if (now - lastSwitchTimeMs >= miniInterval | && activeChunk has data): | switchChunkForCommit() // freezes activeChunk | partitionTracker.onTxnEnd(partition) // safety bookkeeping only ``` The task thread is the sole serializer of `write()` and `setCommitAllowed()` events, which is why both the `cleanBoundary` mark and the conditional switch must run here (not deferred to the manager thread): `cleanBoundary` must reflect the most recent task-thread event at all times, or the manager's idle-fallback could race a write and freeze partial data. ##### 6.3 Manager Thread — Time-Driven Commit[​](#63-manager-thread--time-driven-commit "Direct link to 6.3 Manager Thread — Time-Driven Commit") The manager thread runs a scan loop at `scanningFrequency`. Each iteration: 1. **Ensure shared transaction**: open a new one (eager) or proactively recycle the current one if it is approaching the StarRocks server-side timeout (80% of `timeout` header, default 480 s). 2. **Source-idle fallback**: call `region.tryForceCleanSwitch()` on every region to freeze any `activeChunk` that is clean and has been idle for at least `miniInterval`. This handles the "source paused after a few txnEnds" case where the task thread stopped before issuing a fresh switch. 3. **Time-driven commit trigger**: call `tryStartTimerDrivenCommit()`, which sets `commitInFlight=true` if **both** conditions hold: * `now - lastCommitTimeMs >= commitInterval` (the configured `sink.buffer-flush.interval-ms`). * There is data to commit — either `txnCoordinator.hasDataLoaded()` is true or at least one region still has pending inactiveChunks. 4. **Autonomous flush**: drain any region whose `inactiveChunks` is non-empty via the `FlushAndCommitStrategy`. Multi-table mode's `flush()` only streams out already-frozen inactive chunks — it **never** touches `activeChunk`. This preserves the invariant that every chunk reaching StarRocks under the shared label comes from completed source transactions. When `commitInFlight=true` the main loop enters `processMultiTableCommit`, which waits for in-flight loads, triggers loads for any remaining inactive chunks, runs the unified commit via `SharedTransactionCoordinator`, updates `lastCommitTimeMs`, resets the tracker, and opens a new shared transaction for the next cycle. ##### Shared Transaction Coordination[​](#shared-transaction-coordination "Direct link to Shared Transaction Coordination") All tables within the same commit cycle share a single StarRocks transaction managed by `SharedTransactionCoordinator`: 1. **Eager transaction opening**: A shared transaction is opened eagerly before any autonomous flush, so all HTTP loads use the shared label. This eliminates the data-loss window where an independent-label flush could be orphaned when a shared transaction later overwrites the label. 2. **Unified commit**: After all regions' data is loaded, a single `commit` is executed for the shared label. Multi-table transactions skip the `prepare` step because StarRocks does not support `TXN_PREPARE` in multi-table mode. 3. **Idle transaction recycling**: If the shared transaction remains open longer than 80% of the StarRocks server-side timeout (default: 480s for a 600s timeout), it is proactively recycled (commit-or-rollback + reopen) to prevent server-side timeout errors. Recycle fails fast if any region has in-progress transaction data (clean-boundary violation) or any partition has written data without ever receiving a txnEnd. ##### 6.5 PartitionCommitTracker (Safety Bookkeeping)[​](#65-partitioncommittracker-safety-bookkeeping "Direct link to 6.5 PartitionCommitTracker (Safety Bookkeeping)") In the current design, commit timing is driven entirely by the commit interval and the per-region clean-boundary flags. `PartitionCommitTracker` is reduced to an informational/safety aid: * `onWrite(partition)` registers a partition as `ACTIVE` on first write. * `onTxnEnd(partition)` transitions the partition to `TXN_END_SEEN` (sticky — subsequent writes do **not** demote it back to `ACTIVE`). * `getPartitionsWithoutTxnEnd()` lists partitions that have written data but never received a txnEnd. Used by savepoint and recycle to fail fast on upstream contract violations (a source transaction that never closed). * `reset()` clears all partitions at the end of a commit cycle. The tracker no longer drives switch/commit decisions, does not track a `SWITCHED` state, and does not manage pending txnEnd signals. Partitions that stop producing data after a commit are simply cleared by `reset()`; if they resume later, the next `onWrite` re-registers them. ##### Autonomous Flush with Shared Labels[​](#autonomous-flush-with-shared-labels "Direct link to Autonomous Flush with Shared Labels") When a region's `inactiveChunks` becomes non-empty, the manager thread's autonomous-flush loop streams it to StarRocks via `/api/transaction/load` under the **current shared label**. Because the shared transaction is always opened before any data is loaded, there is no window where data could be loaded under an independent (orphaned) label. In multi-table mode, `flush()` never triggers a `switchChunk` — it only drains already-frozen inactive chunks — so the invariant "every chunk reaching StarRocks under the shared label comes from completed source transactions" holds unconditionally. ##### Safety Guarantees[​](#safety-guarantees "Direct link to Safety Guarantees") | Guarantee | Mechanism | | ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | switchChunk does not split source transactions | `switchChunkForCommit` is only called at a clean transaction boundary (on txnEnd or when `activeChunkCleanBoundary` is `true`). | | Commit never includes partial source transactions | Every chunk reaching StarRocks originates from `switchChunkForCommit`. Autonomous flush never switches `activeChunk` in multi-table mode. | | Per-partition isolation | Each `(partition, table)` has its own region. One partition's switch never affects another's data. | | Within-partition ordering | `keyBy(sourcePartition)` routes same-partition rows to the same sink subtask. | | Task thread is non-blocking | `tryMiniIntervalSwitch` is O(regions-in-partition). HTTP work happens asynchronously on the manager thread. | | Source-idle data visibility | Manager thread's `tryForceCleanSwitch` freezes clean `activeChunk`s after `miniInterval` of idleness, so data remains visible within `commitInterval + miniInterval` even if the source pauses. | | Autonomous flushes are transaction-safe | Every load uses the shared label. Every frozen chunk is from completed source transactions. | | Idle transactions do not timeout | Shared transactions are recycled at 80% of server timeout. Recycle fails fast on in-progress data. | | Per-partition independent commit | A partition with a completed source transaction commits on the next commit interval, independent of other partitions' in-progress transactions. | ##### N:1 Transaction Mapping[​](#n1-transaction-mapping "Direct link to N:1 Transaction Mapping") Multiple source transactions can accumulate in a single StarRocks transaction via the miniInterval batching mechanism: ```text Source txn K1 (3 rows) -> write -> activeChunk -> txnEnd (1st switch) Source txn K2 (2 rows) -> write -> activeChunk -> txnEnd (inside miniInterval: no switch) Source txn K3 (4 rows) -> write -> activeChunk -> txnEnd (inside miniInterval: no switch) (miniInterval elapsed → next switch batches K2+K3) -> ... commitInterval elapsed -> commit(label=A) -> K1 + K2 + K3 atomically committed to StarRocks ``` Because the commit decision is time-driven (not tied to a specific txnEnd), the connector amortizes HTTP-load and begin/commit overhead across many small source transactions without any configuration changes. For a CDC source emitting 100 txnEnds per second with `commitInterval=1 s`, `miniInterval=100 ms`, the connector issues at most ~10 load calls per second instead of ~100. #### Limitations[​](#limitations "Direct link to Limitations") * **Requires `sink.version=V2`**: V1 does not support transaction stream load. * **At-least-once only**: Failed retries may produce duplicate writes. Multi-table mode guarantees all tables within the same batch succeed or fail together, but does not provide global exactly-once. For PRIMARY KEY tables, duplicate writes are idempotent (upsert). * **All tables must be in the same database**: StarRocks multi-table transactions are database-scoped; cross-database transactions are not supported. * **Transaction scope is per sink subtask + per partition**: Each sink subtask maintains its own StarRocks transaction independently. Atomicity is guaranteed **within a single source transaction** (all rows for one txnEnd on one partition, across all tables that partition writes to). Data visibility across **different** source partitions can interleave: once partition P0's source transaction has fully arrived and the commit interval has elapsed, P0's data is committed even if partition P1's source transaction is still in progress. Applications that require cross-partition atomicity at the StarRocks level must either use a single source partition or coordinate commits upstream. * **Data visibility latency**: Governed by `sink.buffer-flush.interval-ms` and the internal `miniInterval = min(1000, max(100, commitInterval/10))`. In a continuously flowing CDC stream, an individual row becomes visible in StarRocks within roughly `commitInterval + miniInterval` of its source commit. During a source pause, previously-committed data remains visible while any row between the last switch and the pause becomes visible after one more `miniInterval` (handled by the manager-thread clean-boundary fallback). * **Depends on StarRocks cluster transaction settings**: Monitor running txn limits, prepared timeout (default 600s), and label retention. Ensure `sink.buffer-flush.interval-ms` is significantly shorter than the StarRocks transaction timeout. * **`activeChunk` memory growth under long source transactions**: Because multi-table mode disables chunk-size-triggered internal switching (to preserve the clean-transaction-boundary invariant), `activeChunk` can grow until the next txnEnd arrives. Memory is bounded by `sink.transaction.multi-table.buffer-size` (soft) and `2 × buffer-size` (hard via `blockIfCacheFull`). Exceptionally large source transactions will throttle the task thread via back-pressure; if this becomes routine, either split the source transactions upstream or increase `sink.transaction.multi-table.buffer-size`. * **Cross-database writes are rejected**: Multi-table transactions validate that all regions belong to the same database. Writing to tables in different databases within the same commit cycle will throw an error. * **Incompatible with merge commit**: `sink.properties.enable_merge_commit=true` cannot be combined with `sink.transaction.multi-table.enabled=true`. Merge commit routes writes through `MergeCommitManager`, which lacks the partition-aware `write(int, ...)` / `setCommitAllowed(int, ...)` hooks that multi-table mode relies on for transaction boundaries. The connector fails fast at validation time if both are enabled. #### Monitoring and Troubleshooting[​](#monitoring-and-troubleshooting "Direct link to Monitoring and Troubleshooting") Recommended metrics: * **On the Flink Side** * Checkpoint success rate * Checkpoint duration * Sink flush/commit latency * **On the StarRocks Side** * Running/prepared txn count * Txn timeout occurrences * Label conflicts ##### Common issues[​](#common-issues "Direct link to Common issues") ###### `transaction not existed`[​](#transaction-not-existed "Direct link to transaction-not-existed") * Cause: StarRocks transaction timeout * Solution: The connector automatically recycles idle transactions at 80% of server timeout. If this still occurs, check if prepared timeout is too short or flush interval is too large. ###### `too many running txns`[​](#too-many-running-txns "Direct link to too-many-running-txns") * Cause: There are excessive concurrent transactions. * Solution: Reduce sink parallelism or increase the value of StarRocks FE configuration `max_running_txn_num_per_db`. ###### `Transaction start failed`[​](#transaction-start-failed "Direct link to transaction-start-failed") * Cause: The `beginTransaction` HTTP call failed * Solution: Verify load-url connectivity and the StarRocks version (v4.0 or later is required). ###### High data visibility latency[​](#high-data-visibility-latency "Direct link to High data visibility latency") * Cause: Commit conditions are not met. * Solution: Verify upstream data has correct `transactionEnd=true` markers; expect up to `commitInterval + miniInterval` latency per row. If latency exceeds this budget, check the manager thread is not stuck in a recycle or in-flight load (see `StarRocks-Sink-Manager` logs). ###### Cross-database write error[​](#cross-database-write-error "Direct link to Cross-database write error") * Cause: Tables in different databases are in same commit cycle. * Solution: Ensure all tables written in the same job belong to the same StarRocks database. #### Best Practices[​](#best-practices "Direct link to Best Practices") 1. **TransactionAssembler contract**: * Emit rows only after the source transaction is fully closed. * The last row must have `setTransactionEnd(true)`. * Every row must have `setSourcePartition(partition)`. * All rows must be emitted synchronously within a single `processElement()` call. 2. **keyBy before sink is mandatory**: `rows.keyBy(DefaultStarRocksRowData::getSourcePartition).addSink(sink)` — omitting this breaks within-partition transaction ordering. 3. **Checkpoint is decoupled from commit**: Checkpoint interval can be set to a large value (for example, 60 seconds) for fault recovery. Data visibility is governed by `sink.buffer-flush.interval-ms` (for example, 1000ms). 4. **Keep routing strategies stable**: Avoid single transactions writing to an excessive number of distinct tables, which increases transaction duration and failure probability. 5. **Fault injection testing before production**: Kill TaskManagers / introduce network jitter and verify data correctness after checkpoint recovery. --- ### Continuously load data from Apache Flink® StarRocks provides a self-developed connector named StarRocks Connector for Apache Flink® (Flink connector for short) to help you load data into a StarRocks table by using Flink. The basic principle is to accumulate the data and then load it all at a time into StarRocks through [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). The Flink connector supports DataStream API, Table API & SQL, and Python API. It has a higher and more stable performance than [flink-connector-jdbc](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/jdbc/) provided by Apache Flink®. > **NOTICE** > > Loading data into StarRocks tables with Flink connector needs SELECT and INSERT privileges on the target StarRocks table. If you do not have these privileges, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant these privileges to the user that you use to connect to your StarRocks cluster. #### Version requirements[​](#version-requirements "Direct link to Version requirements") | Connector | Flink | StarRocks | Java | Scala | | --------- | ----------------------------- | ------------- | ---- | --------- | | 1.2.15 | 1.16,1.17,1.18,1.19,1.20 | 2.1 and later | 8 | 2.11,2.12 | | 1.2.14 | 1.16,1.17,1.18,1.19,1.20 | 2.1 and later | 8 | 2.11,2.12 | | 1.2.12 | 1.16,1.17,1.18,1.19,1.20 | 2.1 and later | 8 | 2.11,2.12 | | 1.2.11 | 1.15,1.16,1.17,1.18,1.19,1.20 | 2.1 and later | 8 | 2.11,2.12 | #### Obtain Flink connector[​](#obtain-flink-connector "Direct link to Obtain Flink connector") You can obtain the Flink connector JAR file in the following ways: * Directly download the compiled Flink connector JAR file. * Add the Flink connector as a dependency in your Maven project and then download the JAR file. * Compile the source code of the Flink connector into a JAR file by yourself. The naming format of the Flink connector JAR file is as follows: * Since Flink 1.15, it's `flink-connector-starrocks-${connector_version}_flink-${flink_version}.jar`. For example, if you install Flink 1.15 and you want to use Flink connector 1.2.7, you can use `flink-connector-starrocks-1.2.7_flink-1.15.jar`. * Prior to Flink 1.15, it's `flink-connector-starrocks-${connector_version}_flink-${flink_version}_${scala_version}.jar`. For example, if you install Flink 1.14 and Scala 2.12 in your environment, and you want to use Flink connector 1.2.7, you can use `flink-connector-starrocks-1.2.7_flink-1.14_2.12.jar`. > **NOTICE** > > In general, the latest version of the Flink connector only maintains compatibility with the three most recent versions of Flink. ##### Download the compiled Jar file[​](#download-the-compiled-jar-file "Direct link to Download the compiled Jar file") Directly download the corresponding version of the Flink connector Jar file from the [Maven Central Repository](https://repo1.maven.org/maven2/com/starrocks). ##### Maven Dependency[​](#maven-dependency "Direct link to Maven Dependency") In your Maven project's `pom.xml` file, add the Flink connector as a dependency according to the following format. Replace `flink_version`, `scala_version`, and `connector_version` with the respective versions. * In Flink 1.15 and later ```xml com.starrocks flink-connector-starrocks ${connector_version}_flink-${flink_version} ``` * In versions earlier than Flink 1.15 ```xml com.starrocks flink-connector-starrocks ${connector_version}_flink-${flink_version}_${scala_version} ``` ##### Compile by yourself[​](#compile-by-yourself "Direct link to Compile by yourself") 1. Download the [Flink connector source code](https://github.com/StarRocks/starrocks-connector-for-apache-flink). 2. Execute the following command to compile the source code of Flink connector into a JAR file. Note that `flink_version` is replaced with the corresponding Flink version. ```bash sh build.sh ``` For example, if the Flink version in your environment is 1.16, you need to execute the following command: ```bash sh build.sh 1.16 ``` 3. Go to the `target/` directory to find the Flink connector JAR file, such as `flink-connector-starrocks-1.2.7_flink-1.16-SNAPSHOT.jar`, generated upon compilation. > **NOTE** > > The name of Flink connector which is not formally released contains the `SNAPSHOT` suffix. #### Options[​](#options "Direct link to Options") ##### General Options[​](#general-options "Direct link to General Options") ###### connector[​](#connector "Direct link to connector") * **Required**: Yes * **Default value**: NONE * **Description**: The connector that you want to use. The value must be "starrocks". ###### jdbc-url[​](#jdbc-url "Direct link to jdbc-url") * **Required**: Yes * **Default value**: NONE * **Description**: The address that is used to connect to the MySQL server of the FE. You can specify multiple addresses, which must be separated by a comma (,). Format: `jdbc:mysql://:,:,:`. ###### load-url[​](#load-url "Direct link to load-url") * **Required**: Yes * **Default value**: NONE * **Description**: The address that is used to connect to the HTTP server of the FE. You can specify multiple addresses, which must be separated by a semicolon (;). Format: `:;:`. ###### database-name[​](#database-name "Direct link to database-name") * **Required**: Yes * **Default value**: NONE * **Description**: The name of the StarRocks database into which you want to load data. ###### table-name[​](#table-name "Direct link to table-name") * **Required**: Yes * **Default value**: NONE * **Description**: The name of the table that you want to use to load data into StarRocks. ###### username[​](#username "Direct link to username") * **Required**: Yes * **Default value**: NONE * **Description**: The username of the account that you want to use to load data into StarRocks. The account needs [SELECT and INSERT privileges](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) on the target StarRocks table. ###### password[​](#password "Direct link to password") * **Required**: Yes * **Default value**: NONE * **Description**: The password of the preceding account. ###### sink.version[​](#sinkversion "Direct link to sink.version") * **Required**: No * **Default value**: AUTO * **Description**: The interface used to load data. This parameter is supported from Flink connector version 1.2.4 onwards. Valid Values: * `V1`: Use [Stream Load](https://docs.starrocks.io/docs/loading/StreamLoad.md) interface to load data. Connectors before 1.2.4 only support this mode. * `V2`: Use [Stream Load transaction](https://docs.starrocks.io/docs/loading/Stream_Load_transaction_interface.md) interface to load data. It requires StarRocks to be at least version 2.4. Recommends `V2` because it optimizes the memory usage and provides a more stable exactly-once implementation. * `AUTO`: If the version of StarRocks supports transaction Stream Load, will choose `V2` automatically, otherwise choose `V1` ###### sink.label-prefix[​](#sinklabel-prefix "Direct link to sink.label-prefix") * **Required**: No * **Default value**: NONE * **Description**: The label prefix used by Stream Load. Recommend to configure it if you are using exactly-once with connector 1.2.8 and later. See [exactly-once usage notes](#exactly-once). ###### sink.semantic[​](#sinksemantic "Direct link to sink.semantic") * **Required**: No * **Default value**: at-least-once * **Description**: The semantic guaranteed by sink. Valid values: **at-least-once** and **exactly-once**. ###### sink.buffer-flush.max-bytes[​](#sinkbuffer-flushmax-bytes "Direct link to sink.buffer-flush.max-bytes") * **Required**: No * **Default value**: 94371840(90M) * **Description**: The maximum size of data that can be accumulated in memory before being sent to StarRocks at a time. The maximum value ranges from 64 MB to 10 GB. Setting this parameter to a larger value can improve loading performance but may increase loading latency. This parameter only takes effect when `sink.semantic` is set to `at-least-once`. If `sink.semantic` is set to `exactly-once`, the data in memory is flushed when a Flink checkpoint is triggered. In this circumstance, this parameter does not take effect. ###### sink.buffer-flush.max-rows[​](#sinkbuffer-flushmax-rows "Direct link to sink.buffer-flush.max-rows") * **Required**: No * **Default value**: 500000 * **Description**: The maximum number of rows that can be accumulated in memory before being sent to StarRocks at a time. This parameter is available only when `sink.version` is `V1` and `sink.semantic` is `at-least-once`. Valid values: 64000 to 5000000. ###### sink.buffer-flush.interval-ms[​](#sinkbuffer-flushinterval-ms "Direct link to sink.buffer-flush.interval-ms") * **Required**: No * **Default value**: 300000 * **Description**: The interval at which data is flushed. This parameter is available only when `sink.semantic` is `at-least-once`. Unit: ms. Valid value range: * For versions earlier than v1.2.14: \[1000, 3600000] * For v1.2.14 and later: (0, 3600000]. ###### sink.max-retries[​](#sinkmax-retries "Direct link to sink.max-retries") * **Required**: No * **Default value**: 3 * **Description**: The number of times that the system retries to perform the Stream Load job. This parameter is available only when you set `sink.version` to `V1`. Valid values: 0 to 10. ###### sink.connect.timeout-ms[​](#sinkconnecttimeout-ms "Direct link to sink.connect.timeout-ms") * **Required**: No * **Default value**: 30000 * **Description**: The timeout for establishing HTTP connection. Valid values: 100 to 60000. Unit: ms. Before Flink connector v1.2.9, the default value is `1000`. ###### sink.socket.timeout-ms[​](#sinksockettimeout-ms "Direct link to sink.socket.timeout-ms") * **Required**: No * **Default value**: -1 * **Description**: Supported since 1.2.10. The time duration for which the HTTP client waits for data. Unit: ms. The default value `-1` means there is no timeout. ###### sink.sanitize-error-log[​](#sinksanitize-error-log "Direct link to sink.sanitize-error-log") * **Required**: No * **Default value**: false * **Description**: Supported since 1.2.12. Whether to sanitize sensitive data in the error log for production security. When this item is set to `true`, sensitive row data and column values in Stream Load error logs are redacted in both the connector and SDK logs. The value defaults to `false` for backward compatibility. ###### sink.wait-for-continue.timeout-ms[​](#sinkwait-for-continuetimeout-ms "Direct link to sink.wait-for-continue.timeout-ms") * **Required**: No * **Default value**: 10000 * **Description**: Supported since 1.2.7. The timeout for waiting response of HTTP 100-continue from the FE. Valid values: `3000` to `60000`. Unit: ms ###### sink.ignore.update-before[​](#sinkignoreupdate-before "Direct link to sink.ignore.update-before") * **Required**: No * **Default value**: true * **Description**: Supported since version 1.2.8. Whether to ignore `UPDATE_BEFORE` records from Flink when loading data to Primary Key tables. If this parameter is set to false, the record is treated as a delete operation to StarRocks table. ###### sink.parallelism[​](#sinkparallelism "Direct link to sink.parallelism") * **Required**: No * **Default value**: NONE * **Description**: The parallelism of loading. Only available for Flink SQL. If this parameter is not specified, Flink planner decides the parallelism. **In the scenario of multi-parallelism, users need to guarantee data is written in the correct order.** ###### sink.properties.\*[​](#sinkproperties "Direct link to sink.properties.*") * **Required**: No * **Default value**: NONE * **Description**: The parameters that are used to control Stream Load behavior. For example, the parameter `sink.properties.format` specifies the format used for Stream Load, such as CSV or JSON. For a list of supported parameters and their descriptions, see [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). ###### sink.properties.format[​](#sinkpropertiesformat "Direct link to sink.properties.format") * **Required**: No * **Default value**: csv * **Description**: The format used for Stream Load. The Flink connector will transform each batch of data to the format before sending them to StarRocks. Valid values: `csv` and `json`. ###### sink.properties.column\_separator[​](#sinkpropertiescolumn_separator "Direct link to sink.properties.column_separator") * **Required**: No * **Default value**: \t * **Description**: The column separator for CSV-formatted data. ###### sink.properties.row\_delimiter[​](#sinkpropertiesrow_delimiter "Direct link to sink.properties.row_delimiter") * **Required**: No * **Default value**: \n * **Description**: The row delimiter for CSV-formatted data. ###### sink.properties.max\_filter\_ratio[​](#sinkpropertiesmax_filter_ratio "Direct link to sink.properties.max_filter_ratio") * **Required**: No * **Default value**: 0 * **Description**: The maximum error tolerance of the Stream Load. It's the maximum percentage of data records that can be filtered out due to inadequate data quality. Valid values: `0` to `1`. Default value: `0`. See [Stream Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md) for details. ###### sink.properties.partial\_update[​](#sinkpropertiespartial_update "Direct link to sink.properties.partial_update") * **Required**: NO * **Default value**: `FALSE` * **Description**: Whether to use partial updates. Valid values: `TRUE` and `FALSE`. Default value: `FALSE`, indicating to disable this feature. ###### sink.properties.partial\_update\_mode[​](#sinkpropertiespartial_update_mode "Direct link to sink.properties.partial_update_mode") * **Required**: NO * **Default value**: `row` * **Description**: Specifies the mode for partial updates. Valid values: `row` and `column`. * The value `row` (default) means partial updates in row mode, which is more suitable for real-time updates with many columns and small batches. * The value `column` means partial updates in column mode, which is more suitable for batch updates with few columns and many rows. In such scenarios, enabling the column mode offers faster update speeds. For example, in a table with 100 columns, if only 10 columns (10% of the total) are updated for all rows, the update speed of the column mode is 10 times faster. ###### sink.properties.strict\_mode[​](#sinkpropertiesstrict_mode "Direct link to sink.properties.strict_mode") * **Required**: No * **Default value**: false * **Description**: Specifies whether to enable the strict mode for Stream Load. It affects the loading behavior when there are unqualified rows, such as inconsistent column values. Valid values: `true` and `false`. Default value: `false`. See [Stream Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md) for details. ###### sink.properties.compression[​](#sinkpropertiescompression "Direct link to sink.properties.compression") * **Required**: No * **Default value**: NONE * **Description**: The compression algorithm used for Stream Load. Valid values: `lz4_frame`. Compression for the JSON format requires Flink connector 1.2.10+ and StarRocks v3.2.7+. Compression for the CSV format only requires Flink connector 1.2.11+. ###### sink.properties.prepared\_timeout[​](#sinkpropertiesprepared_timeout "Direct link to sink.properties.prepared_timeout") * **Required**: No * **Default value**: NONE * **Description**: Supported since 1.2.12 and only effective when `sink.version` is set to `V2`. Requires StarRocks 3.5.4 or later. Sets the timeout in seconds for the Transaction Stream Load phase from `PREPARED` to `COMMITTED`. Typically, only needed for exactly-once; at-least-once usually does not require setting this (the connector defaults to 300s). If not set in exactly-once, StarRocks FE configuration `prepared_transaction_default_timeout_second` (default 86400s) applies. See [StarRocks Transaction timeout management](https://docs.starrocks.io/docs/loading/Stream_Load_transaction_interface.md#transaction-timeout-management). ###### sink.publish-timeout.ms[​](#sinkpublish-timeoutms "Direct link to sink.publish-timeout.ms") * **Required**: No * **Default value**: -1 * **Description**: Supported since 1.2.14 and only effective when `sink.version` is set to `V2`. Timeout in milliseconds for the Publish phase. If a transaction stays in COMMITTED status longer than this timeout, the system will consider it as successful. The default value `-1` means using StarRocks server-side default behavior. When Merge Commit is enabled, the default timeout is 10000 ms. ##### Merge Commit options[​](#merge-commit-options "Direct link to Merge Commit options") Supported from v1.2.14 onwards. Merge Commit allows the system to merge data from multiple subtasks into a single Stream Load transaction for better performance. You can enable this feature by setting `sink.properties.enable_merge_commit` to `true`. For more details about the merge commit feature in StarRocks, see [Merge Commit parameters](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md#merge-commit-parameters). The following Stream Load properties are used to control the Merge Commit behavior: ###### sink.properties.enable\_merge\_commit[​](#sinkpropertiesenable_merge_commit "Direct link to sink.properties.enable_merge_commit") * **Required**: No * **Default value**: false * **Description**: Whether to enable Merge Commit. ###### sink.properties.merge\_commit\_interval\_ms[​](#sinkpropertiesmerge_commit_interval_ms "Direct link to sink.properties.merge_commit_interval_ms") * **Required**: Yes (when Merge Commit is enabled) * **Default value**: NONE * **Description**: The Merge Commit time window in milliseconds. The system merges loading requests received within this window into a single transaction. A larger value improves merging efficiency but increases latency. This property must be set when `enable_merge_commit` is set to `true`. ###### sink.properties.merge\_commit\_parallel[​](#sinkpropertiesmerge_commit_parallel "Direct link to sink.properties.merge_commit_parallel") * **Required**: No * **Default value**: 3 * **Description**: The degree of parallelism for the loading plan created for each Merge Commit-enabled transaction. It is different from `sink.parallelism` which controls the parallelism of the Flink sink operator. ###### sink.properties.merge\_commit\_async[​](#sinkpropertiesmerge_commit_async "Direct link to sink.properties.merge_commit_async") * **Required**: No * **Default value**: true * **Description**: The server's return mode for Merge Commit. The default value is `true` (asynchronous), overriding the system default behavior (synchronous) for better throughput. In the asynchronous mode, the server returns immediately after receiving the data. The connector leverages Flink's checkpoint mechanism to ensure no data loss under the asynchronous mode, providing at-least-once guarantee. In most cases, you do not need to change this value. ###### sink.merge-commit.max-concurrent-requests[​](#sinkmerge-commitmax-concurrent-requests "Direct link to sink.merge-commit.max-concurrent-requests") * **Required**: No * **Default value**: Integer.MAX\_VALUE * **Description**: The maximum number of concurrent Stream Load requests. Set this property to `0` to ensure in-order (serial) loading, which is useful for Primary Key tables. A negative value is treated as `Integer.MAX_VALUE` (unlimited concurrency). ###### sink.merge-commit.chunk.size[​](#sinkmerge-commitchunksize "Direct link to sink.merge-commit.chunk.size") * **Required**: No * **Default value**: 20971520 * **Description**: The maximum size of data (in bytes) accumulated in a chunk before it is flushed and sent to StarRocks via a Stream Load request. A larger value improves throughput but increases memory usage and latency; a smaller value reduces memory usage and latency but may lower throughput. When `max-concurrent-requests` is set to `0` (in-order mode), the default value of this property is changed to 500 MB because only one request runs at a time so a larger batch maximizes throughput. #### Data type mapping between Flink and StarRocks[​](#data-type-mapping-between-flink-and-starrocks "Direct link to Data type mapping between Flink and StarRocks") | Flink data type | StarRocks data type | | ------------------------------------- | ------------------- | | BOOLEAN | BOOLEAN | | TINYINT | TINYINT | | SMALLINT | SMALLINT | | INTEGER | INTEGER | | BIGINT | BIGINT | | FLOAT | FLOAT | | DOUBLE | DOUBLE | | DECIMAL | DECIMAL | | BINARY | INT | | CHAR | STRING | | VARCHAR | STRING | | STRING | STRING | | DATE | DATE | | TIMESTAMP\_WITHOUT\_TIME\_ZONE(N) | DATETIME | | TIMESTAMP\_WITH\_LOCAL\_TIME\_ZONE(N) | DATETIME | | ARRAY\ | ARRAY\ | | MAP\ | JSON STRING | | ROW\ | JSON STRING | #### Usage notes[​](#usage-notes "Direct link to Usage notes") ##### Exactly Once[​](#exactly-once "Direct link to Exactly Once") * If you want sink to guarantee exactly-once semantics, we recommend you to upgrade StarRocks to 2.5 or later, and Flink connector to 1.2.4 or later * Since Flink connector 1.2.4, the exactly-once is redesigned based on [Stream Load transaction interface](https://docs.starrocks.io/docs/loading/Stream_Load_transaction_interface.md) provided by StarRocks since 2.4. Compared to the previous implementation based on non-transactional Stream Load non-transactional interface, the new implementation reduces memory usage and checkpoint overhead, thereby enhancing real-time performance and stability of loading. * If the version of StarRocks is earlier than 2.4 or the version of Flink connector is earlier than 1.2.4, the sink will automatically choose the implementation based on Stream Load non-transactional interface. * Configurations to guarantee exactly-once * The value of `sink.semantic` needs to be `exactly-once`. * If the version of Flink connector is 1.2.8 and later, it is recommended to specify the value of `sink.label-prefix`. Note that the label prefix must be unique among all types of loading in StarRocks, such as Flink jobs, Routine Load, and Broker Load. * If the label prefix is specified, the Flink connector will use the label prefix to clean up lingering transactions that may be generated in some Flink failure scenarios, such as the Flink job fails when a checkpoint is still in progress. These lingering transactions are generally in `PREPARED` status if you use `SHOW PROC '/transactions//running';` to view them in StarRocks. When the Flink job restores from checkpoint, the Flink connector will find these lingering transactions according to the label prefix and some information in checkpoint, and abort them. The Flink connector can not abort them when the Flink job exits because of the two-phase-commit mechanism to implement the exactly-once. When the Flink job exits, the Flink connector has not received the notification from Flink checkpoint coordinator whether the transactions should be included in a successful checkpoint, and it may lead to data loss if these transactions are aborted anyway. You can have an overview about how to achieve end-to-end exactly-once in Flink in this [blogpost](https://flink.apache.org/2018/02/28/an-overview-of-end-to-end-exactly-once-processing-in-apache-flink-with-apache-kafka-too/). * If the label prefix is not specified, lingering transactions will be cleaned up by StarRocks only after they time out. However the number of running transactions can reach the limitation of StarRocks `max_running_txn_num_per_db` if Flink jobs fail frequently before transactions time out. You can set a smaller timeout for `PREPARED` transactions to make them expired faster when the label prefix is not specified. See the following about how to set the prepared timeout. * If you are certain that the Flink job will eventually recover from checkpoint or savepoint after a long downtime because of stop or continuous failover, please adjust the following StarRocks configurations accordingly, to avoid data loss. * Adjust `PREPARED` transaction timeout. See the following about how to set the timeout. The timeout needs to be larger than the downtime of the Flink job. Otherwise, the lingering transactions that are included in a successful checkpoint may be aborted because of timeout before you restart the Flink job, which leads to data loss. Note that when you set a larger value to this configuration, it is better to specify the value of `sink.label-prefix` so that the lingering transactions can be cleaned according to the label prefix and some information in checkpoint, instead of due to timeout (which may cause data loss). * `label_keep_max_second` and `label_keep_max_num`: StarRocks FE configurations, default values are `259200` and `1000` respectively. For details, see [FE configurations](https://docs.starrocks.io/docs/loading/loading_introduction/loading_considerations.md#fe-configurations). The value of `label_keep_max_second` needs to be larger than the downtime of the Flink job. Otherwise, the Flink connector can not check the state of transactions in StarRocks by using the transaction labels saved in the Flink's savepoint or checkpoint and figure out whether these transactions are committed or not, which may eventually lead to data loss. * How to set the timeout for PREPARED transactions * For Connector 1.2.12+ and StarRocks 3.5.4+, you can set the timeout by configuring the connector parameter `sink.properties.prepared_timeout`. By default, the value is not set, and it falls back to the StarRocks FE's global configuration `prepared_transaction_default_timeout_second` (default value is `86400`). * For other versions of Connector or StarRocks, you can set the timeout by configuring the StarRocks FE's global configuration `prepared_transaction_default_timeout_second` (default value is `86400`). ##### Flush Policy[​](#flush-policy "Direct link to Flush Policy") The Flink connector will buffer the data in memory, and flush them in batch to StarRocks via Stream Load. How the flush is triggered is different between at-least-once and exactly-once. For at-least-once, the flush will be triggered when any of the following conditions are met: * the bytes of buffered rows reaches the limit `sink.buffer-flush.max-bytes` * the number of buffered rows reaches the limit `sink.buffer-flush.max-rows`. (Only valid for sink version V1) * the elapsed time since the last flush reaches the limit `sink.buffer-flush.interval-ms` * a checkpoint is triggered For exactly-once, the flush only happens when a checkpoint is triggered. ##### Merge Commit[​](#merge-commit "Direct link to Merge Commit") Merge Commit helps scale throughput without proportionally increasing StarRocks transaction overhead. Without Merge Commit, each Flink sink subtask maintains its own Stream Load transaction, so increasing `sink.parallelism` leads to more concurrent transactions and higher I/O and Compaction costs on StarRocks. Conversely, keeping parallelism low limits the pipeline's overall capacity. With Merge Commit is enabled, data from multiple sink subtasks is merged into a single transaction within each Merge window. This allows you to increase `sink.parallelism` for higher throughput without increasing the number of transactions. For configuration examples, see [Load data with merge commit](#load-data-with-merge-commit). Below are some important notes when using Merge Commit: * **Single parallelism has no benefit** If the Flink sink parallelism is 1, enabling Merge Commit provides no benefit since there is only one subtask sending data. It may even introduce additional latency due to the Merge Commit time window on the server side. * **Only at-least-once semantic** Merge Commit only guarantees at-least-once semantic. It does not support exactly-once semantic. Do not set `sink.semantic` to `exactly-once` when Merge Commit is enabled. * **Ordering for Primary Key tables** By default, `sink.merge-commit.max-concurrent-requests` is `Integer.MAX_VALUE`, which means a single sink subtask may send multiple Stream Load requests concurrently. This can cause out-of-order loading, which may be problematic for Primary Key tables. To ensure in-order loading, set `sink.merge-commit.max-concurrent-requests` to `0`, which, however, will reduce throughput. Alternatively, you can use Conditional Update to prevent newer data from being overwritten by older data. For configuration examples, see [In-order loading for Primary Key tables](#in-order-loading-for-primary-key-tables). * **End-to-end loading latency** The total loading latency consists of two parts: * **Connector batching latency**: Controlled by `sink.buffer-flush.interval-ms` and `sink.merge-commit.chunk.size`. Data is flushed from the connector when either the chunk size limit is reached or the flush interval elapses, whichever comes first. The maximum connector-side latency is `sink.buffer-flush.interval-ms`. A smaller `sink.buffer-flush.interval-ms` reduces connector-side latency but sends data in smaller batches. * **StarRocks merge window**: Controlled by `sink.properties.merge_commit_interval_ms`. The system waits for this duration to merge requests from multiple subtasks into a single transaction. A larger value improves merging efficiency (more requests will be merged into one transaction) but increases server-side latency. * As a general guideline, set `sink.buffer-flush.interval-ms` to be smaller than or equal to `sink.properties.merge_commit_interval_ms`, so that each subtask can flush at least once within each Merge window. For example, if `merge_commit_interval_ms` is `10000` (10s), you could set `sink.buffer-flush.interval-ms` to `5000` (5 seconds) or less. * **Tuning `sink.parallelism` and `sink.properties.merge_commit_parallel`** These two parameters control parallelism at different layers and should be tuned independently: * `sink.parallelism` controls the number of Flink sink subtasks. Each subtask buffers and sends data to StarRocks. Increase this value when Flink sink operators are CPU- or memory-bound — you can monitor Flink's per-operator CPU and memory usage to determine whether more subtasks are needed. * `sink.properties.merge_commit_parallel` controls the degree of parallelism for the loading plan that StarRocks creates for each Merge Commit transaction. Increase this value when StarRocks becomes the bottleneck. You can monitor the StarRocks metrics [merge\_commit\_pending\_total](https://docs.starrocks.io/docs/administration/management/monitoring/metrics.md#merge_commit_pending_total) (number of pending Merge Commit tasks) and [merge\_commit\_pending\_bytes](https://docs.starrocks.io/docs/administration/management/monitoring/metrics.md#merge_commit_pending_bytes) (bytes held by pending tasks) to determine whether more parallelism is needed on the StarRocks side — sustained high values indicate that the loading plan cannot keep up with incoming data. * **Relationship between `sink.merge-commit.chunk.size` and `sink.buffer-flush.max-bytes`**: * `sink.merge-commit.chunk.size` controls the maximum data size per individual Stream Load request (per chunk). When data in a chunk reaches this size, it is flushed immediately. * `sink.buffer-flush.max-bytes` controls the total memory limit for all cached data across all tables. When the total cached data exceeds this limit, the connector will evict chunks early to free memory. * Therefore, `sink.buffer-flush.max-bytes` should be set larger than `sink.merge-commit.chunk.size` to allow at least one full chunk to be accumulated. In general, `sink.buffer-flush.max-bytes` should be several times larger than `sink.merge-commit.chunk.size`, especially when there are multiple tables or high concurrency. ##### Monitoring load metrics[​](#monitoring-load-metrics "Direct link to Monitoring load metrics") The Flink connector provides the following metrics to monitor loading. | Metric | Type | Description | | ------------------------ | ------- | ------------------------------------------------------------------ | | totalFlushBytes | counter | successfully flushed bytes. | | totalFlushRows | counter | number of rows successfully flushed. | | totalFlushSucceededTimes | counter | number of times that the data is successfully flushed. | | totalFlushFailedTimes | counter | number of times that the data fails to be flushed. | | totalFilteredRows | counter | number of rows filtered, which is also included in totalFlushRows. | #### Examples[​](#examples "Direct link to Examples") The following examples show how to use the Flink connector to load data into a StarRocks table with Flink SQL or Flink DataStream. ##### Preparations[​](#preparations "Direct link to Preparations") ###### Create a StarRocks table[​](#create-a-starrocks-table "Direct link to Create a StarRocks table") Create a database `test` and create a Primary Key table `score_board`. ```sql CREATE DATABASE `test`; CREATE TABLE `test`.`score_board` ( `id` int(11) NOT NULL COMMENT "", `name` varchar(65533) NULL DEFAULT "" COMMENT "", `score` int(11) NOT NULL DEFAULT "0" COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`id`) COMMENT "OLAP" DISTRIBUTED BY HASH(`id`); ``` ###### Set up Flink environment[​](#set-up-flink-environment "Direct link to Set up Flink environment") * Download Flink binary [Flink 1.15.2](https://archive.apache.org/dist/flink/flink-1.15.2/flink-1.15.2-bin-scala_2.12.tgz), and unzip it to directory `flink-1.15.2`. * Download [Flink connector 1.2.7](https://repo1.maven.org/maven2/com/starrocks/flink-connector-starrocks/1.2.7_flink-1.15/flink-connector-starrocks-1.2.7_flink-1.15.jar), and put it into the directory `flink-1.15.2/lib`. * Run the following commands to start a Flink cluster: ```shell cd flink-1.15.2 ./bin/start-cluster.sh ``` ###### Network configuration[​](#network-configuration "Direct link to Network configuration") Ensure that the machine where Flink is located can access the FE nodes of the StarRocks cluster via the [`http_port`](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#http_port) (default: `8030`) and [`query_port`](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#query_port) (default: `9030`), and the BE nodes via the [`be_http_port`](https://docs.starrocks.io/docs/administration/management/BE_configuration.md#be_http_port) (default: `8040`). ##### Run with Flink SQL[​](#run-with-flink-sql "Direct link to Run with Flink SQL") * Run the following command to start a Flink SQL client. ```shell ./bin/sql-client.sh ``` * Create a Flink table `score_board`, and insert values into the table via Flink SQL Client. Note you must define the primary key in the Flink DDL if you want to load data into a Primary Key table of StarRocks. It's optional for other types of StarRocks tables. ```sql CREATE TABLE `score_board` ( `id` INT, `name` STRING, `score` INT, PRIMARY KEY (id) NOT ENFORCED ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030', 'load-url' = '127.0.0.1:8030', 'database-name' = 'test', 'table-name' = 'score_board', 'username' = 'root', 'password' = '' ); INSERT INTO `score_board` VALUES (1, 'starrocks', 100), (2, 'flink', 100); ``` ##### Run with Flink DataStream[​](#run-with-flink-datastream "Direct link to Run with Flink DataStream") There are several ways to implement a Flink DataStream job according to the type of the input records, such as a CSV Java `String`, a JSON Java `String` or a custom Java object. * The input records are CSV-format `String`. See [LoadCsvRecords](https://github.com/StarRocks/starrocks-connector-for-apache-flink/tree/cd8086cfedc64d5181785bdf5e89a847dc294c1d/examples/src/main/java/com/starrocks/connector/flink/examples/datastream) for a complete example. ```java /** * Generate CSV-format records. Each record has three values separated by "\t". * These values will be loaded to the columns `id`, `name`, and `score` in the StarRocks table. */ String[] records = new String[]{ "1\tstarrocks-csv\t100", "2\tflink-csv\t100" }; DataStream source = env.fromElements(records); /** * Configure the connector with the required properties. * You also need to add properties "sink.properties.format" and "sink.properties.column_separator" * to tell the connector the input records are CSV-format, and the column separator is "\t". * You can also use other column separators in the CSV-format records, * but remember to modify the "sink.properties.column_separator" correspondingly. */ StarRocksSinkOptions options = StarRocksSinkOptions.builder() .withProperty("jdbc-url", jdbcUrl) .withProperty("load-url", loadUrl) .withProperty("database-name", "test") .withProperty("table-name", "score_board") .withProperty("username", "root") .withProperty("password", "") .withProperty("sink.properties.format", "csv") .withProperty("sink.properties.column_separator", "\t") .build(); // Create the sink with the options. SinkFunction starRockSink = StarRocksSink.sink(options); source.addSink(starRockSink); ``` * The input records are JSON-format `String`. See [LoadJsonRecords](https://github.com/StarRocks/starrocks-connector-for-apache-flink/tree/cd8086cfedc64d5181785bdf5e89a847dc294c1d/examples/src/main/java/com/starrocks/connector/flink/examples/datastream) for a complete example. ```java /** * Generate JSON-format records. * Each record has three key-value pairs corresponding to the columns `id`, `name`, and `score` in the StarRocks table. */ String[] records = new String[]{ "{\"id\":1, \"name\":\"starrocks-json\", \"score\":100}", "{\"id\":2, \"name\":\"flink-json\", \"score\":100}", }; DataStream source = env.fromElements(records); /** * Configure the connector with the required properties. * You also need to add properties "sink.properties.format" and "sink.properties.strip_outer_array" * to tell the connector the input records are JSON-format and to strip the outermost array structure. */ StarRocksSinkOptions options = StarRocksSinkOptions.builder() .withProperty("jdbc-url", jdbcUrl) .withProperty("load-url", loadUrl) .withProperty("database-name", "test") .withProperty("table-name", "score_board") .withProperty("username", "root") .withProperty("password", "") .withProperty("sink.properties.format", "json") .withProperty("sink.properties.strip_outer_array", "true") .build(); // Create the sink with the options. SinkFunction starRockSink = StarRocksSink.sink(options); source.addSink(starRockSink); ``` * The input records are custom Java objects. See [LoadCustomJavaRecords](https://github.com/StarRocks/starrocks-connector-for-apache-flink/tree/cd8086cfedc64d5181785bdf5e89a847dc294c1d/examples/src/main/java/com/starrocks/connector/flink/examples/datastream) for a complete example. * In this example, the input record is a simple POJO `RowData`. ```java public static class RowData { public int id; public String name; public int score; public RowData() {} public RowData(int id, String name, int score) { this.id = id; this.name = name; this.score = score; } } ``` * The main program is as follows: ```java // Generate records which use RowData as the container. RowData[] records = new RowData[]{ new RowData(1, "starrocks-rowdata", 100), new RowData(2, "flink-rowdata", 100), }; DataStream source = env.fromElements(records); // Configure the connector with the required properties. StarRocksSinkOptions options = StarRocksSinkOptions.builder() .withProperty("jdbc-url", jdbcUrl) .withProperty("load-url", loadUrl) .withProperty("database-name", "test") .withProperty("table-name", "score_board") .withProperty("username", "root") .withProperty("password", "") .build(); /** * The Flink connector will use a Java object array (Object[]) to represent a row to be loaded into the StarRocks table, * and each element is the value for a column. * You need to define the schema of the Object[] which matches that of the StarRocks table. */ TableSchema schema = TableSchema.builder() .field("id", DataTypes.INT().notNull()) .field("name", DataTypes.STRING()) .field("score", DataTypes.INT()) // When the StarRocks table is a Primary Key table, you must specify notNull(), for example, DataTypes.INT().notNull(), for the primary key `id`. .primaryKey("id") .build(); // Transform the RowData to the Object[] according to the schema. RowDataTransformer transformer = new RowDataTransformer(); // Create the sink with the schema, options, and transformer. SinkFunction starRockSink = StarRocksSink.sink(schema, options, transformer); source.addSink(starRockSink); ``` * The `RowDataTransformer` in the main program is defined as follows: ```java private static class RowDataTransformer implements StarRocksSinkRowBuilder { /** * Set each element of the object array according to the input RowData. * The schema of the array matches that of the StarRocks table. */ @Override public void accept(Object[] internalRow, RowData rowData) { internalRow[0] = rowData.id; internalRow[1] = rowData.name; internalRow[2] = rowData.score; // When the StarRocks table is a Primary Key table, you need to set the last element to indicate whether the data loading is an UPSERT or DELETE operation. internalRow[internalRow.length - 1] = StarRocksSinkOP.UPSERT.ordinal(); } } ``` ##### Synchronize data with Flink CDC 3.0 (with schema change supported)[​](#synchronize-data-with-flink-cdc-30-with-schema-change-supported "Direct link to Synchronize data with Flink CDC 3.0 (with schema change supported)") [Flink CDC 3.0](https://nightlies.apache.org/flink/flink-cdc-docs-stable) framework can be used to easily build a streaming ELT pipeline from CDC sources (such as MySQL and Kafka) to StarRocks. The pipeline can synchronize whole database, merged sharding tables, and schema changes from sources to StarRocks. Since v1.2.9, the Flink connector for StarRocks is integrated into this framework as [StarRocks Pipeline Connector](https://nightlies.apache.org/flink/flink-cdc-docs-release-3.1/docs/connectors/pipeline-connectors/starrocks/). The StarRocks Pipeline Connector supports: * Automatic creation of databases and tables * Schema change synchronization * Full and incremental data synchronization For quick start, see [Streaming ELT from MySQL to StarRocks using Flink CDC 3.0 with StarRocks Pipeline Connector](https://nightlies.apache.org/flink/flink-cdc-docs-release-3.4/docs/get-started/quickstart/mysql-to-starrocks/). It is advised to use StarRocks v3.2.1 and later versions to enable [fast\_schema\_evolution](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md#set-fast-schema-evolution). It will improve the speed of adding or dropping columns and reduce resource usage. #### Best practices[​](#best-practices "Direct link to Best practices") ##### Load data to a Primary Key table[​](#load-data-to-a-primary-key-table "Direct link to Load data to a Primary Key table") This section will show how to load data to a StarRocks Primary Key table to achieve partial updates and conditional updates. You can see [Change data through loading](https://docs.starrocks.io/docs/loading/Load_to_Primary_Key_tables.md) for the introduction of those features. These examples use Flink SQL. ###### Preparations[​](#preparations-1 "Direct link to Preparations") Create a database `test` and create a Primary Key table `score_board` in StarRocks. ```sql CREATE DATABASE `test`; CREATE TABLE `test`.`score_board` ( `id` int(11) NOT NULL COMMENT "", `name` varchar(65533) NULL DEFAULT "" COMMENT "", `score` int(11) NOT NULL DEFAULT "0" COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`id`) COMMENT "OLAP" DISTRIBUTED BY HASH(`id`); ``` ###### Partial update[​](#partial-update "Direct link to Partial update") This example will show how to load data only to columns `id` and `name`. 1. Insert two data rows into the StarRocks table `score_board` in MySQL client. ```sql mysql> INSERT INTO `score_board` VALUES (1, 'starrocks', 100), (2, 'flink', 100); mysql> select * from score_board; +------+-----------+-------+ | id | name | score | +------+-----------+-------+ | 1 | starrocks | 100 | | 2 | flink | 100 | +------+-----------+-------+ 2 rows in set (0.02 sec) ``` 2. Create a Flink table `score_board` in Flink SQL client. * Define the DDL which only includes the columns `id` and `name`. * Set the option `sink.properties.partial_update` to `true` which tells the Flink connector to perform partial updates. * If the Flink connector version `<=` 1.2.7, you also need to set the option `sink.properties.columns` to `id,name,__op` to tells the Flink connector which columns need to be updated. Note that you need to append the field `__op` at the end. The field `__op` indicates that the data loading is an UPSERT or DELETE operation, and its values are set by the connector automatically. ```sql CREATE TABLE `score_board` ( `id` INT, `name` STRING, PRIMARY KEY (id) NOT ENFORCED ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030', 'load-url' = '127.0.0.1:8030', 'database-name' = 'test', 'table-name' = 'score_board', 'username' = 'root', 'password' = '', 'sink.properties.partial_update' = 'true', -- only for Flink connector version <= 1.2.7 'sink.properties.columns' = 'id,name,__op' ); ``` 3. Insert two data rows into the Flink table. The primary keys of the data rows are as same as these of rows in the StarRocks table. but the values in the column `name` are modified. ```sql INSERT INTO `score_board` VALUES (1, 'starrocks-update'), (2, 'flink-update'); ``` 4. Query the StarRocks table in MySQL client. ```sql mysql> select * from score_board; +------+------------------+-------+ | id | name | score | +------+------------------+-------+ | 1 | starrocks-update | 100 | | 2 | flink-update | 100 | +------+------------------+-------+ 2 rows in set (0.02 sec) ``` You can see that only values for `name` change, and the values for `score` do not change. ###### Conditional update[​](#conditional-update "Direct link to Conditional update") This example will show how to do conditional update according to the value of column `score`. The update for an `id` takes effect only when the new value for `score` is has a greater or equal to the old value. 1. Insert two data rows into the StarRocks table in MySQL client. ```sql mysql> INSERT INTO `score_board` VALUES (1, 'starrocks', 100), (2, 'flink', 100); mysql> select * from score_board; +------+-----------+-------+ | id | name | score | +------+-----------+-------+ | 1 | starrocks | 100 | | 2 | flink | 100 | +------+-----------+-------+ 2 rows in set (0.02 sec) ``` 2. Create a Flink table `score_board` in the following ways: * Define the DDL including all of columns. * Set the option `sink.properties.merge_condition` to `score` to tell the connector to use the column `score` as the condition. * Set the option `sink.version` to `V1` or `V2`. Both support conditional update. ```sql CREATE TABLE `score_board` ( `id` INT, `name` STRING, `score` INT, PRIMARY KEY (id) NOT ENFORCED ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030', 'load-url' = '127.0.0.1:8030', 'database-name' = 'test', 'table-name' = 'score_board', 'username' = 'root', 'password' = '', 'sink.properties.merge_condition' = 'score', 'sink.version' = 'V1' ); ``` 3. Insert two data rows into the Flink table. The primary keys of the data rows are as same as these of rows in the StarRocks table. The first data row has a smaller value in the column `score`, and the second data row has a larger value in the column `score`. ```sql INSERT INTO `score_board` VALUES (1, 'starrocks-update', 99), (2, 'flink-update', 101); ``` 4. Query the StarRocks table in MySQL client. ```sql mysql> select * from score_board; +------+--------------+-------+ | id | name | score | +------+--------------+-------+ | 1 | starrocks | 100 | | 2 | flink-update | 101 | +------+--------------+-------+ 2 rows in set (0.03 sec) ``` You can see that only the values of the second data row change, and the values of the first data row do not change. ##### Load data with Merge Commit[​](#load-data-with-merge-commit "Direct link to Load data with Merge Commit") This section shows how to use Merge Commit to improve loading throughput when you have multiple Flink sink subtasks writing to the same StarRocks table. These examples use Flink SQL and StarRocks v3.4.0 or later. ###### Preparations[​](#preparations-2 "Direct link to Preparations") Create a database `test` and create a Primary Key table `score_board` in StarRocks. ```sql CREATE DATABASE `test`; CREATE TABLE `test`.`score_board` ( `id` int(11) NOT NULL COMMENT "", `name` varchar(65533) NULL DEFAULT "" COMMENT "", `score` int(11) NOT NULL DEFAULT "0" COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`id`) COMMENT "OLAP" DISTRIBUTED BY HASH(`id`); ``` ###### Basic configuration[​](#basic-configuration "Direct link to Basic configuration") This Flink SQL enables merge commit with a 10-second merge window. Data from all sink subtasks is merged into a single transaction within each window. ```sql CREATE TABLE `score_board` ( `id` INT, `name` STRING, `score` INT, PRIMARY KEY (id) NOT ENFORCED ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030', 'load-url' = '127.0.0.1:8030', 'database-name' = 'test', 'table-name' = 'score_board', 'username' = 'root', 'password' = '', 'sink.properties.enable_merge_commit' = 'true', 'sink.properties.merge_commit_interval_ms' = '10000', 'sink.buffer-flush.interval-ms' = '5000' ); ``` Insert data into the Flink table. The data will be loaded into StarRocks via merge commit. ```sql INSERT INTO `score_board` VALUES (1, 'starrocks', 100), (2, 'flink', 95), (3, 'spark', 90); ``` ###### In-order loading for Primary Key tables[​](#in-order-loading-for-primary-key-tables "Direct link to In-order loading for Primary Key tables") By default, a single sink subtask may send multiple Stream Load requests concurrently, which can cause out-of-order loading. For Primary Key tables where data ordering matters, there are two approaches to handle this issue. **Method 1: Use `sink.merge-commit.max-concurrent-requests`** Set `sink.merge-commit.max-concurrent-requests` to `0` to ensure each subtask sends requests one at a time. This guarantees in-order loading but may reduce throughput. ```sql CREATE TABLE `score_board` ( `id` INT, `name` STRING, `score` INT, PRIMARY KEY (id) NOT ENFORCED ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030', 'load-url' = '127.0.0.1:8030', 'database-name' = 'test', 'table-name' = 'score_board', 'username' = 'root', 'password' = '', 'sink.properties.enable_merge_commit' = 'true', 'sink.properties.merge_commit_interval_ms' = '10000', 'sink.buffer-flush.interval-ms' = '5000', 'sink.merge-commit.max-concurrent-requests' = '0' ); INSERT INTO `score_board` VALUES (1, 'starrocks', 100), (2, 'flink', 95), (3, 'spark', 90); ``` **Method 2: Use Conditional Update** If you want to keep concurrent requests for higher throughput but still prevent older data from overwriting newer data, you can use [Conditional Update](#conditional-update). Set `sink.properties.merge_condition` to a column (for example, a version or timestamp column) so that an update only takes effect when the incoming value is greater than or equal to the existing value. ```sql CREATE TABLE `score_board` ( `id` INT, `name` STRING, `score` INT, PRIMARY KEY (id) NOT ENFORCED ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030', 'load-url' = '127.0.0.1:8030', 'database-name' = 'test', 'table-name' = 'score_board', 'username' = 'root', 'password' = '', 'sink.properties.enable_merge_commit' = 'true', 'sink.properties.merge_commit_interval_ms' = '10000', 'sink.buffer-flush.interval-ms' = '5000', 'sink.properties.merge_condition' = 'score' ); INSERT INTO `score_board` VALUES (1, 'starrocks', 100), (2, 'flink', 95), (3, 'spark', 90); ``` With this configuration, concurrent requests are allowed (default `sink.merge-commit.max-concurrent-requests` is `Integer.MAX_VALUE`), but an update to a row only takes effect when the new `score` is greater than or equal to the existing `score`. This prevents newer data from being overwritten by older data even under out-of-order loading. ##### Load data into columns of BITMAP type[​](#load-data-into-columns-of-bitmap-type "Direct link to Load data into columns of BITMAP type") [`BITMAP`](https://docs.starrocks.io/docs/sql-reference/data-types/other-data-types/BITMAP.md) is often used to accelerate count distinct, such as counting UV, see [Use Bitmap for exact Count Distinct](https://docs.starrocks.io/docs/using_starrocks/distinct_values/Using_bitmap.md). Here we take the counting of UV as an example to show how to load data into columns of the `BITMAP` type. 1. Create a StarRocks Aggregate table in MySQL client. In the database `test`, create an Aggregate table `page_uv` where the column `visit_users` is defined as the `BITMAP` type and configured with the aggregate function `BITMAP_UNION`. ```sql CREATE TABLE `test`.`page_uv` ( `page_id` INT NOT NULL COMMENT 'page ID', `visit_date` datetime NOT NULL COMMENT 'access time', `visit_users` BITMAP BITMAP_UNION NOT NULL COMMENT 'user ID' ) ENGINE=OLAP AGGREGATE KEY(`page_id`, `visit_date`) DISTRIBUTED BY HASH(`page_id`); ``` 2. Create a Flink table in Flink SQL client. The column `visit_user_id` in the Flink table is of `BIGINT` type, and we want to load this column to the column `visit_users` of `BITMAP` type in the StarRocks table. So when defining the DDL of the Flink table, note that: * Because Flink does not support `BITMAP`, you need to define a column `visit_user_id` as `BIGINT` type to represent the column `visit_users` of `BITMAP` type in the StarRocks table. * You need to set the option `sink.properties.columns` to `page_id,visit_date,user_id,visit_users=to_bitmap(visit_user_id)`, which tells the connector the column mapping between the Flink table and StarRocks table. Also you need to use [`to_bitmap`](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/to_bitmap.md) function to tell the connector to convert the data of `BIGINT` type into `BITMAP` type. ```sql CREATE TABLE `page_uv` ( `page_id` INT, `visit_date` TIMESTAMP, `visit_user_id` BIGINT ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030', 'load-url' = '127.0.0.1:8030', 'database-name' = 'test', 'table-name' = 'page_uv', 'username' = 'root', 'password' = '', 'sink.properties.columns' = 'page_id,visit_date,visit_user_id,visit_users=to_bitmap(visit_user_id)' ); ``` 3. Load data into Flink table in Flink SQL client. ```sql INSERT INTO `page_uv` VALUES (1, CAST('2020-06-23 01:30:30' AS TIMESTAMP), 13), (1, CAST('2020-06-23 01:30:30' AS TIMESTAMP), 23), (1, CAST('2020-06-23 01:30:30' AS TIMESTAMP), 33), (1, CAST('2020-06-23 02:30:30' AS TIMESTAMP), 13), (2, CAST('2020-06-23 01:30:30' AS TIMESTAMP), 23); ``` 4. Calculate page UVs from the StarRocks table in MySQL client. ```sql MySQL [test]> SELECT `page_id`, COUNT(DISTINCT `visit_users`) FROM `page_uv` GROUP BY `page_id`; +---------+-----------------------------+ | page_id | count(DISTINCT visit_users) | +---------+-----------------------------+ | 2 | 1 | | 1 | 3 | +---------+-----------------------------+ 2 rows in set (0.05 sec) ``` ##### Load data into columns of HLL type[​](#load-data-into-columns-of-hll-type "Direct link to Load data into columns of HLL type") [`HLL`](https://docs.starrocks.io/docs/sql-reference/data-types/other-data-types/HLL.md) can be used for approximate count distinct, see [Use HLL for approximate count distinct](https://docs.starrocks.io/docs/using_starrocks/distinct_values/Using_HLL.md). Here we take the counting of UV as an example to show how to load data into columns of the `HLL` type. 1. Create a StarRocks Aggregate table In the database `test`, create an Aggregate table `hll_uv` where the column `visit_users` is defined as the `HLL` type and configured with the aggregate function `HLL_UNION`. ```sql CREATE TABLE `hll_uv` ( `page_id` INT NOT NULL COMMENT 'page ID', `visit_date` datetime NOT NULL COMMENT 'access time', `visit_users` HLL HLL_UNION NOT NULL COMMENT 'user ID' ) ENGINE=OLAP AGGREGATE KEY(`page_id`, `visit_date`) DISTRIBUTED BY HASH(`page_id`); ``` 2. Create a Flink table in Flink SQL client. The column `visit_user_id` in the Flink table is of `BIGINT` type, and we want to load this column to the column `visit_users` of `HLL` type in the StarRocks table. So when defining the DDL of the Flink table, note that: * Because Flink does not support `BITMAP`, you need to define a column `visit_user_id` as `BIGINT` type to represent the column `visit_users` of `HLL` type in the StarRocks table. * You need to set the option `sink.properties.columns` to `page_id,visit_date,user_id,visit_users=hll_hash(visit_user_id)` which tells the connector the column mapping between Flink table and StarRocks table. Also you need to use [`hll_hash`](https://docs.starrocks.io/docs/sql-reference/sql-functions/scalar-functions/hll_hash.md) function to tell the connector to convert the data of `BIGINT` type into `HLL` type. ```sql CREATE TABLE `hll_uv` ( `page_id` INT, `visit_date` TIMESTAMP, `visit_user_id` BIGINT ) WITH ( 'connector' = 'starrocks', 'jdbc-url' = 'jdbc:mysql://127.0.0.1:9030', 'load-url' = '127.0.0.1:8030', 'database-name' = 'test', 'table-name' = 'hll_uv', 'username' = 'root', 'password' = '', 'sink.properties.columns' = 'page_id,visit_date,visit_user_id,visit_users=hll_hash(visit_user_id)' ); ``` 3. Load data into Flink table in Flink SQL client. ```sql INSERT INTO `hll_uv` VALUES (3, CAST('2023-07-24 12:00:00' AS TIMESTAMP), 78), (4, CAST('2023-07-24 13:20:10' AS TIMESTAMP), 2), (3, CAST('2023-07-24 12:30:00' AS TIMESTAMP), 674); ``` 4. Calculate page UVs from the StarRocks table in MySQL client. ```sql mysql> SELECT `page_id`, COUNT(DISTINCT `visit_users`) FROM `hll_uv` GROUP BY `page_id`; **+---------+-----------------------------+ | page_id | count(DISTINCT visit_users) | +---------+-----------------------------+ | 3 | 2 | | 4 | 1 | +---------+-----------------------------+ 2 rows in set (0.04 sec) ``` --- ### Load data from GCS StarRocks provides the following options for loading data from GCS: * Synchronous loading using [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md)+[`FILES()`](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md) * Asynchronous loading using [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) Each of these options has its own advantages, which are detailed in the following sections. In most cases, we recommend that you use the INSERT+`FILES()` method, which is much easier to use. However, the INSERT+`FILES()` method currently supports only the Parquet, ORC, and CSV file formats. Therefore, if you need to load data of other file formats such as JSON, or [perform data changes such as DELETE during data loading](https://docs.starrocks.io/docs/loading/Load_to_Primary_Key_tables.md), you can resort to Broker Load. #### Before you begin[​](#before-you-begin "Direct link to Before you begin") ##### Make source data ready[​](#make-source-data-ready "Direct link to Make source data ready") Make sure the source data you want to load into StarRocks is properly stored in a GCS bucket. You may also consider where the data and the database are located, because data transfer costs are much lower when your bucket and your StarRocks cluster are located in the same region. In this topic, we provide you with a sample dataset in a GCS bucket, `gs://starrocks-samples/user_behavior_ten_million_rows.parquet`. You can access that dataset with any valid credentials as the object is readable by any GCP user. ##### Check privileges[​](#check-privileges "Direct link to Check privileges") You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. ##### Gather authentication details[​](#gather-authentication-details "Direct link to Gather authentication details") The examples in this topic use service account-based authentication. To practice IAM user-based authentication, you need to gather information about the following GCS resources: * The GCS bucket that stores your data. * The GCS object key (object name) if accessing a specific object in the bucket. Note that the object key can include a prefix if your GCS objects are stored in sub-folders. * The GCS region to which the GCS bucket belongs. * The `private_ key_id`, `private_key`, and `client_email` of your Google Cloud service account For information about all the authentication methods available, see [Authenticate to Google Cloud Storage](https://docs.starrocks.io/docs/integrations/authenticate_to_gcs.md). #### Use INSERT+FILES()[​](#use-insertfiles "Direct link to Use INSERT+FILES()") This method is available from v3.2 onwards and currently supports only the Parquet, ORC, and CSV (from v3.3.0 onwards) file formats. ##### Advantages of INSERT+FILES()[​](#advantages-of-insertfiles "Direct link to Advantages of INSERT+FILES()") `FILES()` can read the file stored in cloud storage based on the path-related properties you specify, infer the table schema of the data in the file, and then return the data from the file as data rows. With `FILES()`, you can: * Query the data directly from GCS using [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md). * Create and load a table using [CREATE TABLE AS SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE_AS_SELECT.md) (CTAS). * Load the data into an existing table using [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md). ##### Typical examples[​](#typical-examples "Direct link to Typical examples") ###### Querying directly from GCS using SELECT[​](#querying-directly-from-gcs-using-select "Direct link to Querying directly from GCS using SELECT") Querying directly from GCS using SELECT+`FILES()` can give a good preview of the content of a dataset before you create a table. For example: * Get a preview of the dataset without storing the data. * Query for the min and max values and decide what data types to use. * Check for `NULL` values. The following example queries the sample dataset `gs://starrocks-samples/user_behavior_ten_million_rows.parquet`: ```sql SELECT * FROM FILES ( "path" = "gs://starrocks-samples/user_behavior_ten_million_rows.parquet", "format" = "parquet", "gcp.gcs.service_account_email" = "sampledatareader@xxxxx-xxxxxx-000000.iam.gserviceaccount.com", "gcp.gcs.service_account_private_key_id" = "baaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "gcp.gcs.service_account_private_key" = "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----" ) LIMIT 3; ``` > **NOTE** > > Substitute the credentials in the above command with your own credentials. Any valid service account email, key, and secret can be used, as the object is readable by any GCP authenticated user. The system returns a query result similar to the following: ```plain +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 543711 | 829192 | 2355072 | pv | 2017-11-27 08:22:37 | | 543711 | 2056618 | 3645362 | pv | 2017-11-27 10:16:46 | | 543711 | 1165492 | 3645362 | pv | 2017-11-27 10:17:00 | +--------+---------+------------+--------------+---------------------+ ``` > **NOTE** > > Notice that the column names as returned above are provided by the Parquet file. ###### Creating and loading a table using CTAS[​](#creating-and-loading-a-table-using-ctas "Direct link to Creating and loading a table using CTAS") This is a continuation of the previous example. The previous query is wrapped in CREATE TABLE AS SELECT (CTAS) to automate the table creation using schema inference. This means StarRocks will infer the table schema, create the table you want, and then load the data into the table. The column names and types are not required to create a table when using the `FILES()` table function with Parquet files as the Parquet format includes the column names. > **NOTE** > > The syntax of CREATE TABLE when using schema inference does not allow setting the number of replicas. If you are using a StarRocks shared-nothing cluster, set the number of replicas before creating the table. The example below is for a system with three replicas: > > ```sql > ADMIN SET FRONTEND CONFIG ('default_replication_num' = "3"); > > ``` Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Use CTAS to create a table and load the data of the sample dataset `gs://starrocks-samples/user_behavior_ten_million_rows.parquet` into the table: ```sql CREATE TABLE user_behavior_inferred AS SELECT * FROM FILES ( "path" = "gs://starrocks-samples/user_behavior_ten_million_rows.parquet", "format" = "parquet", "gcp.gcs.service_account_email" = "sampledatareader@xxxxx-xxxxxx-000000.iam.gserviceaccount.com", "gcp.gcs.service_account_private_key_id" = "baaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "gcp.gcs.service_account_private_key" = "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----" ); ``` > **NOTE** > > Substitute the credentials in the above command with your own credentials. Any valid service account email, key, and secret can be used, as the object is readable by any GCP authenticated user. After creating the table, you can view its schema by using [DESCRIBE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DESCRIBE.md): ```sql DESCRIBE user_behavior_inferred; ``` The system returns a query result similar to the following: ```plain +--------------+-----------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------+-----------+------+-------+---------+-------+ | UserID | bigint | YES | true | NULL | | | ItemID | bigint | YES | true | NULL | | | CategoryID | bigint | YES | true | NULL | | | BehaviorType | varbinary | YES | false | NULL | | | Timestamp | varbinary | YES | false | NULL | | +--------------+-----------+------+-------+---------+-------+ ``` Query the table to verify that the data has been loaded into it. Example: ```sql SELECT * from user_behavior_inferred LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plain +--------+--------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+--------+------------+--------------+---------------------+ | 84 | 162325 | 2939262 | pv | 2017-12-02 05:41:41 | | 84 | 232622 | 4148053 | pv | 2017-11-27 04:36:10 | | 84 | 595303 | 903809 | pv | 2017-11-26 08:03:59 | +--------+--------+------------+--------------+---------------------+ ``` ###### Loading into an existing table using INSERT[​](#loading-into-an-existing-table-using-insert "Direct link to Loading into an existing table using INSERT") You may want to customize the table that you are inserting into, for example, the: * column data type, nullable setting, or default values * key types and columns * data partitioning and bucketing > **NOTE** > > Creating the most efficient table structure requires knowledge of how the data will be used and the content of the columns. This topic does not cover table design. For information about table design, see [Table types](https://docs.starrocks.io/docs/table_design/StarRocks_table_design.md). In this example, we are creating a table based on knowledge of how the table will be queried and the data in the Parquet file. The knowledge of the data in the Parquet file can be gained by querying the file directly in GCS. * Since a query of the dataset in GCS indicates that the `Timestamp` column contains data that matches a VARBINARY data type, the column type is specified in the following DDL. * By querying the data in GCS, you can find that there are no `NULL` values in the dataset, so the DDL does not set any columns as nullable. * Based on knowledge of the expected query types, the sort key and bucketing column are set to the column `UserID`. Your use case might be different for this data, so you might decide to use `ItemID` in addition to or instead of `UserID` for the sort key. Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand (we recommend that the table have the same schema as the Parquet file you want to load from GCS): ```sql CREATE TABLE user_behavior_declared ( UserID int(11), ItemID int(11), CategoryID int(11), BehaviorType varchar(65533), Timestamp varbinary ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID); ``` Display the schema so that you can compare it with the inferred schema produced by the `FILES()` table function: ```sql DESCRIBE user_behavior_declared; ``` ```plaintext +--------------+----------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------+----------------+------+-------+---------+-------+ | UserID | int | NO | true | NULL | | | ItemID | int | NO | false | NULL | | | CategoryID | int | NO | false | NULL | | | BehaviorType | varchar(65533) | NO | false | NULL | | | Timestamp | varbinary | NO | false | NULL | | +--------------+----------------+------+-------+---------+-------+ 5 rows in set (0.00 sec) ``` tip Compare the schema you just created with the schema inferred earlier using the `FILES()` table function. Look at: * data types * nullable * key fields To better control the schema of the destination table and for better query performance, we recommend that you specify the table schema by hand in production environments. After creating the table, you can load it with INSERT INTO SELECT FROM FILES(): ```sql INSERT INTO user_behavior_declared SELECT * FROM FILES ( "path" = "gs://starrocks-samples/user_behavior_ten_million_rows.parquet", "format" = "parquet", "gcp.gcs.service_account_email" = "sampledatareader@xxxxx-xxxxxx-000000.iam.gserviceaccount.com", "gcp.gcs.service_account_private_key_id" = "baaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "gcp.gcs.service_account_private_key" = "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----" ); ``` > **NOTE** > > Substitute the credentials in the above command with your own credentials. Any valid service account email, key, and secret can be used, as the object is readable by any GCP authenticated user. After the load is complete, you can query the table to verify that the data has been loaded into it. Example: ```sql SELECT * from user_behavior_declared LIMIT 3; ``` The system returns a query result similar to the following, indicating that the data has been successfully loaded: ```plain +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 142 | 2869980 | 2939262 | pv | 2017-11-25 03:43:22 | | 142 | 2522236 | 1669167 | pv | 2017-11-25 15:14:12 | | 142 | 3031639 | 3607361 | pv | 2017-11-25 15:19:25 | +--------+---------+------------+--------------+---------------------+ ``` ###### Check load progress[​](#check-load-progress "Direct link to Check load progress") You can query the progress of INSERT jobs from the [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md) view in the StarRocks Information Schema. This feature is supported from v3.1 onwards. Example: ```sql SELECT * FROM information_schema.loads ORDER BY JOB_ID DESC; ``` For information about the fields provided in the `loads` view, see [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md). If you have submitted multiple load jobs, you can filter on the `LABEL` associated with the job. Example: ```sql SELECT * FROM information_schema.loads WHERE LABEL = 'insert_f3fc2298-a553-11ee-92f4-00163e0842bd' \G *************************** 1. row *************************** JOB_ID: 10193 LABEL: insert_f3fc2298-a553-11ee-92f4-00163e0842bd DATABASE_NAME: mydatabase STATE: FINISHED PROGRESS: ETL:100%; LOAD:100% TYPE: INSERT PRIORITY: NORMAL SCAN_ROWS: 10000000 FILTERED_ROWS: 0 UNSELECTED_ROWS: 0 SINK_ROWS: 10000000 ETL_INFO: TASK_INFO: resource:N/A; timeout(s):300; max_filter_ratio:0.0 CREATE_TIME: 2023-12-28 15:37:38 ETL_START_TIME: 2023-12-28 15:37:38 ETL_FINISH_TIME: 2023-12-28 15:37:38 LOAD_START_TIME: 2023-12-28 15:37:38 LOAD_FINISH_TIME: 2023-12-28 15:39:35 JOB_DETAILS: {"All backends":{"f3fc2298-a553-11ee-92f4-00163e0842bd":[10120]},"FileNumber":0,"FileSize":0,"InternalTableLoadBytes":581730322,"InternalTableLoadRows":10000000,"ScanBytes":581574034,"ScanRows":10000000,"TaskNumber":1,"Unfinished backends":{"f3fc2298-a553-11ee-92f4-00163e0842bd":[]}} ERROR_MSG: NULL TRACKING_URL: NULL TRACKING_SQL: NULL REJECTED_RECORD_PATH: NULL ``` > **NOTE** > > INSERT is a synchronous command. If an INSERT job is still running, you need to open another session to check its execution status. #### Use Broker Load[​](#use-broker-load "Direct link to Use Broker Load") An asynchronous Broker Load process handles making the connection to GCS, pulling the data, and storing the data in StarRocks. This method supports the following file formats: * Parquet * ORC * CSV * JSON (supported from v3.2.3 onwards) ##### Advantages of Broker Load[​](#advantages-of-broker-load "Direct link to Advantages of Broker Load") * Broker Load runs in the background and clients don't need to stay connected for the job to continue. * Broker Load is preferred for long running jobs, the default timeout is 4 hours. * In addition to Parquet and ORC file format, Broker Load supports CSV file format and JSON file format (JSON file format is supported from v3.2.3 onwards). ##### Data flow[​](#data-flow "Direct link to Data flow") ![Workflow of Broker Load](/assets/images/broker_load_how-to-work_en-bb36de70866e6366b2b21808f0f77be8.png) 1. The user creates a load job. 2. The frontend (FE) creates a query plan and distributes the plan to the backend nodes (BEs) or compute nodes (CNs). 3. The BEs or CNs pull the data from the source and load the data into StarRocks. ##### Typical example[​](#typical-example "Direct link to Typical example") Create a table, start a load process that pulls the sample dataset `gs://starrocks-samples/user_behavior_ten_million_rows.parquet` from GCS, and verify the progress and success of the data loading. ###### Create a database and a table[​](#create-a-database-and-a-table "Direct link to Create a database and a table") Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand (we recommend that the table has the same schema as the Parquet file that you want to load from GCS): ```sql CREATE TABLE user_behavior ( UserID int(11), ItemID int(11), CategoryID int(11), BehaviorType varchar(65533), Timestamp varbinary ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID); ``` ###### Start a Broker Load[​](#start-a-broker-load "Direct link to Start a Broker Load") Run the following command to start a Broker Load job that loads data from the sample dataset `gs://starrocks-samples/user_behavior_ten_million_rows.parquet` to the `user_behavior` table: ```sql LOAD LABEL user_behavior ( DATA INFILE("gs://starrocks-samples/user_behavior_ten_million_rows.parquet") INTO TABLE user_behavior FORMAT AS "parquet" ) WITH BROKER ( "gcp.gcs.service_account_email" = "sampledatareader@xxxxx-xxxxxx-000000.iam.gserviceaccount.com", "gcp.gcs.service_account_private_key_id" = "baaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", "gcp.gcs.service_account_private_key" = "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----" ) PROPERTIES ( "timeout" = "72000" ); ``` > **NOTE** > > Substitute the credentials in the above command with your own credentials. Any valid service account email, key, and secret can be used, as the object is readable by any GCP authenticated user. This job has four main sections: * `LABEL`: A string used when querying the state of the load job. * `LOAD` declaration: The source URI, source data format, and destination table name. * `BROKER`: The connection details for the source. * `PROPERTIES`: The timeout value and any other properties to apply to the load job. For detailed syntax and parameter descriptions, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). ###### Check load progress[​](#check-load-progress-1 "Direct link to Check load progress") You can query the progress of INSERT jobs from the [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md) view in the StarRocks Information Schema. This feature is supported from v3.1 onwards. ```sql SELECT * FROM information_schema.loads; ``` For information about the fields provided in the `loads` view, see [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md). If you have submitted multiple load jobs, you can filter on the `LABEL` associated with the job. Example: ```sql SELECT * FROM information_schema.loads WHERE LABEL = 'user_behavior'; ``` In the output below there are two entries for the load job `user_behavior`: * The first record shows a state of `CANCELLED`. Scroll to `ERROR_MSG`, and you can see that the job has failed due to `listPath failed`. * The second record shows a state of `FINISHED`, which means that the job has succeeded. ```plain JOB_ID|LABEL |DATABASE_NAME|STATE |PROGRESS |TYPE |PRIORITY|SCAN_ROWS|FILTERED_ROWS|UNSELECTED_ROWS|SINK_ROWS|ETL_INFO|TASK_INFO |CREATE_TIME |ETL_START_TIME |ETL_FINISH_TIME |LOAD_START_TIME |LOAD_FINISH_TIME |JOB_DETAILS |ERROR_MSG |TRACKING_URL|TRACKING_SQL|REJECTED_RECORD_PATH| ------+-------------------------------------------+-------------+---------+-------------------+------+--------+---------+-------------+---------------+---------+--------+----------------------------------------------------+-------------------+-------------------+-------------------+-------------------+-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+------------+------------+--------------------+ 10121|user_behavior |mydatabase |CANCELLED|ETL:N/A; LOAD:N/A |BROKER|NORMAL | 0| 0| 0| 0| |resource:N/A; timeout(s):72000; max_filter_ratio:0.0|2023-08-10 14:59:30| | | |2023-08-10 14:59:34|{"All backends":{},"FileNumber":0,"FileSize":0,"InternalTableLoadBytes":0,"InternalTableLoadRows":0,"ScanBytes":0,"ScanRows":0,"TaskNumber":0,"Unfinished backends":{}} |type:ETL_RUN_FAIL; msg:listPath failed| | | | 10106|user_behavior |mydatabase |FINISHED |ETL:100%; LOAD:100%|BROKER|NORMAL | 86953525| 0| 0| 86953525| |resource:N/A; timeout(s):72000; max_filter_ratio:0.0|2023-08-10 14:50:15|2023-08-10 14:50:19|2023-08-10 14:50:19|2023-08-10 14:50:19|2023-08-10 14:55:10|{"All backends":{"a5fe5e1d-d7d0-4826-ba99-c7348f9a5f2f":[10004]},"FileNumber":1,"FileSize":1225637388,"InternalTableLoadBytes":2710603082,"InternalTableLoadRows":86953525,"ScanBytes":1225637388,"ScanRows":86953525,"TaskNumber":1,"Unfinished backends":{"a5| | | | | ``` After you confirm that the load job has finished, you can check a subset of the destination table to see if the data has been successfully loaded. Example: ```sql SELECT * from user_behavior LIMIT 3; ``` The system returns a query result similar to the following, indicating that the data has been successfully loaded: ```plain +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 142 | 2869980 | 2939262 | pv | 2017-11-25 03:43:22 | | 142 | 2522236 | 1669167 | pv | 2017-11-25 15:14:12 | | 142 | 3031639 | 3607361 | pv | 2017-11-25 15:19:25 | +--------+---------+------------+--------------+---------------------+ ``` --- ### Load data from HDFS StarRocks provides the following options for loading data from HDFS: * Synchronous loading using [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md)+[`FILES()`](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md) * Asynchronous loading using [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) * Continuous asynchronous loading using [Pipe](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/CREATE_PIPE.md) Each of these options has its own advantages, which are detailed in the following sections. In most cases, we recommend that you use the INSERT+`FILES()` method, which is much easier to use. However, the INSERT+`FILES()` method currently supports only the Parquet, ORC, and CSV file formats. Therefore, if you need to load data of other file formats such as JSON, or perform data changes such as DELETE during data loading, you can resort to Broker Load. If you need to load a large number of data files with a significant data volume in total (for example, more than 100 GB or even 1 TB), we recommend that you use the Pipe method. Pipe can split the files based on their number or size, breaking down the load job into smaller, sequential tasks. This approach ensures that errors in one file do not impact the entire load job and minimizes the need for retries due to data errors. #### Before you begin[​](#before-you-begin "Direct link to Before you begin") ##### Make source data ready[​](#make-source-data-ready "Direct link to Make source data ready") Make sure the source data you want to load into StarRocks is properly stored in your HDFS cluster. This topic assumes that you want to load `/user/amber/user_behavior_ten_million_rows.parquet` from HDFS into StarRocks. ##### Check privileges[​](#check-privileges "Direct link to Check privileges") You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. ##### Gather authentication details[​](#gather-authentication-details "Direct link to Gather authentication details") You can use the simple authentication method to establish connections with your HDFS cluster. To use simple authentication, you need to gather the username and password of the account that you can use to access the NameNode of the HDFS cluster. #### Use INSERT+FILES()[​](#use-insertfiles "Direct link to Use INSERT+FILES()") This method is available from v3.1 onwards and currently supports only the Parquet, ORC, and CSV (from v3.3.0 onwards) file formats. ##### Advantages of INSERT+FILES()[​](#advantages-of-insertfiles "Direct link to Advantages of INSERT+FILES()") [`FILES()`](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md) can read the file stored in cloud storage based on the path-related properties you specify, infer the table schema of the data in the file, and then return the data from the file as data rows. With `FILES()`, you can: * Query the data directly from HDFS using [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md). * Create and load a table using [CREATE TABLE AS SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE_AS_SELECT.md) (CTAS). * Load the data into an existing table using [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md). ##### Typical examples[​](#typical-examples "Direct link to Typical examples") ###### Querying directly from HDFS using SELECT[​](#querying-directly-from-hdfs-using-select "Direct link to Querying directly from HDFS using SELECT") Querying directly from HDFS using SELECT+`FILES()` can give a good preview of the content of a dataset before you create a table. For example: * Get a preview of the dataset without storing the data. * Query for the min and max values and decide what data types to use. * Check for `NULL` values. The following example queries the data file `/user/amber/user_behavior_ten_million_rows.parquet` stored in the HDFS cluster: ```sql SELECT * FROM FILES ( "path" = "hdfs://:/user/amber/user_behavior_ten_million_rows.parquet", "format" = "parquet", "hadoop.security.authentication" = "simple", "username" = "", "password" = "" ) LIMIT 3; ``` The system returns the following query result: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 543711 | 829192 | 2355072 | pv | 2017-11-27 08:22:37 | | 543711 | 2056618 | 3645362 | pv | 2017-11-27 10:16:46 | | 543711 | 1165492 | 3645362 | pv | 2017-11-27 10:17:00 | +--------+---------+------------+--------------+---------------------+ ``` > **NOTE** > > Notice that the column names as returned above are provided by the Parquet file. ###### Creating and loading a table using CTAS[​](#creating-and-loading-a-table-using-ctas "Direct link to Creating and loading a table using CTAS") This is a continuation of the previous example. The previous query is wrapped in CREATE TABLE AS SELECT (CTAS) to automate the table creation using schema inference. This means StarRocks will infer the table schema, create the table you want, and then load the data into the table. The column names and types are not required to create a table when using the `FILES()` table function with Parquet files as the Parquet format includes the column names. > **NOTE** > > The syntax of CREATE TABLE when using schema inference does not allow setting the number of replicas, so set it before creating the table. The example below is for a system with three replicas: > > ```sql > ADMIN SET FRONTEND CONFIG ('default_replication_num' = "3"); > > ``` Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Use CTAS to create a table and load the data of the data file `/user/amber/user_behavior_ten_million_rows.parquet` into the table: ```sql CREATE TABLE user_behavior_inferred AS SELECT * FROM FILES ( "path" = "hdfs://:/user/amber/user_behavior_ten_million_rows.parquet", "format" = "parquet", "hadoop.security.authentication" = "simple", "username" = "", "password" = "" ); ``` After creating the table, you can view its schema by using [DESCRIBE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DESCRIBE.md): ```sql DESCRIBE user_behavior_inferred; ``` The system returns the following query result: ```plain +--------------+-----------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------+-----------+------+-------+---------+-------+ | UserID | bigint | YES | true | NULL | | | ItemID | bigint | YES | true | NULL | | | CategoryID | bigint | YES | true | NULL | | | BehaviorType | varbinary | YES | false | NULL | | | Timestamp | varbinary | YES | false | NULL | | +--------------+-----------+------+-------+---------+-------+ ``` Query the table to verify that the data has been loaded into it. Example: ```sql SELECT * from user_behavior_inferred LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plaintext +--------+--------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+--------+------------+--------------+---------------------+ | 84 | 56257 | 1879194 | pv | 2017-11-26 05:56:23 | | 84 | 108021 | 2982027 | pv | 2017-12-02 05:43:00 | | 84 | 390657 | 1879194 | pv | 2017-11-28 11:20:30 | +--------+--------+------------+--------------+---------------------+ ``` ###### Loading into an existing table using INSERT[​](#loading-into-an-existing-table-using-insert "Direct link to Loading into an existing table using INSERT") You may want to customize the table that you are inserting into, for example, the: * column data type, nullable setting, or default values * key types and columns * data partitioning and bucketing > **NOTE** > > Creating the most efficient table structure requires knowledge of how the data will be used and the content of the columns. This topic does not cover table design. For information about table design, see [Table types](https://docs.starrocks.io/docs/table_design/StarRocks_table_design.md). In this example, we are creating a table based on knowledge of how the table will be queried and the data in the Parquet file. The knowledge of the data in the Parquet file can be gained by querying the file directly in HDFS. * Since a query of the dataset in HDFS indicates that the `Timestamp` column contains data that matches a VARBINARY data type, the column type is specified in the following DDL. * By querying the data in HDFS, you can find that there are no `NULL` values in the dataset, so the DDL does not set any columns as nullable. * Based on knowledge of the expected query types, the sort key and bucketing column are set to the column `UserID`. Your use case might be different for this data, so you might decide to use `ItemID` in addition to or instead of `UserID` for the sort key. Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand (we recommend that the table have the same schema as the Parquet file you want to load from HDFS): ```sql CREATE TABLE user_behavior_declared ( UserID int(11), ItemID int(11), CategoryID int(11), BehaviorType varchar(65533), Timestamp varbinary ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID); ``` Display the schema so that you can compare it with the inferred schema produced by the `FILES()` table function: ```sql DESCRIBE user_behavior_declared; ``` ```plaintext +--------------+----------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------+----------------+------+-------+---------+-------+ | UserID | int | NO | true | NULL | | | ItemID | int | NO | false | NULL | | | CategoryID | int | NO | false | NULL | | | BehaviorType | varchar(65533) | NO | false | NULL | | | Timestamp | varbinary | NO | false | NULL | | +--------------+----------------+------+-------+---------+-------+ 5 rows in set (0.00 sec) ``` tip Compare the schema you just created with the schema inferred earlier using the `FILES()` table function. Look at: * data types * nullable * key fields To better control the schema of the destination table and for better query performance, we recommend that you specify the table schema by hand in production environments. After creating the table, you can load it with INSERT INTO SELECT FROM FILES(): ```sql INSERT INTO user_behavior_declared SELECT * FROM FILES ( "path" = "hdfs://:/user/amber/user_behavior_ten_million_rows.parquet", "format" = "parquet", "hadoop.security.authentication" = "simple", "username" = "", "password" = "" ); ``` After the load is complete, you can query the table to verify that the data has been loaded into it. Example: ```sql SELECT * from user_behavior_declared LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 107 | 1568743 | 4476428 | pv | 2017-11-25 14:29:53 | | 107 | 470767 | 1020087 | pv | 2017-11-25 14:32:31 | | 107 | 358238 | 1817004 | pv | 2017-11-25 14:43:23 | +--------+---------+------------+--------------+---------------------+ ``` ###### Check load progress[​](#check-load-progress "Direct link to Check load progress") You can query the progress of INSERT jobs from the [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md) view in the StarRocks Information Schema. This feature is supported from v3.1 onwards. Example: ```sql SELECT * FROM information_schema.loads ORDER BY JOB_ID DESC; ``` For information about the fields provided in the `loads` view, see [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md). If you have submitted multiple load jobs, you can filter on the `LABEL` associated with the job. Example: ```sql SELECT * FROM information_schema.loads WHERE LABEL = 'insert_0d86c3f9-851f-11ee-9c3e-00163e044958' \G *************************** 1. row *************************** JOB_ID: 10214 LABEL: insert_0d86c3f9-851f-11ee-9c3e-00163e044958 DATABASE_NAME: mydatabase STATE: FINISHED PROGRESS: ETL:100%; LOAD:100% TYPE: INSERT PRIORITY: NORMAL SCAN_ROWS: 10000000 FILTERED_ROWS: 0 UNSELECTED_ROWS: 0 SINK_ROWS: 10000000 ETL_INFO: TASK_INFO: resource:N/A; timeout(s):300; max_filter_ratio:0.0 CREATE_TIME: 2023-11-17 15:58:14 ETL_START_TIME: 2023-11-17 15:58:14 ETL_FINISH_TIME: 2023-11-17 15:58:14 LOAD_START_TIME: 2023-11-17 15:58:14 LOAD_FINISH_TIME: 2023-11-17 15:58:18 JOB_DETAILS: {"All backends":{"0d86c3f9-851f-11ee-9c3e-00163e044958":[10120]},"FileNumber":0,"FileSize":0,"InternalTableLoadBytes":311710786,"InternalTableLoadRows":10000000,"ScanBytes":581574034,"ScanRows":10000000,"TaskNumber":1,"Unfinished backends":{"0d86c3f9-851f-11ee-9c3e-00163e044958":[]}} ERROR_MSG: NULL TRACKING_URL: NULL TRACKING_SQL: NULL REJECTED_RECORD_PATH: NULL ``` > **NOTE** > > INSERT is a synchronous command. If an INSERT job is still running, you need to open another session to check its execution status. #### Use Broker Load[​](#use-broker-load "Direct link to Use Broker Load") An asynchronous Broker Load process handles making the connection to HDFS, pulling the data, and storing the data in StarRocks. This method supports the following file formats: * Parquet * ORC * CSV * JSON (supported from v3.2.3 onwards) ##### Advantages of Broker Load[​](#advantages-of-broker-load "Direct link to Advantages of Broker Load") * Broker Load runs in the background and clients do not need to stay connected for the job to continue. * Broker Load is preferred for long-running jobs, with the default timeout spanning 4 hours. * In addition to Parquet and ORC file format, Broker Load supports CSV file format and JSON file format (JSON file format is supported from v3.2.3 onwards). ##### Data flow[​](#data-flow "Direct link to Data flow") ![Workflow of Broker Load](/assets/images/broker_load_how-to-work_en-bb36de70866e6366b2b21808f0f77be8.png) 1. The user creates a load job. 2. The frontend (FE) creates a query plan and distributes the plan to the backend nodes (BEs) or compute nodes (CNs). 3. The BEs or CNs pull the data from the source and load the data into StarRocks. ##### Typical example[​](#typical-example "Direct link to Typical example") Create a table, start a load process that pulls the data file `/user/amber/user_behavior_ten_million_rows.parquet` from HDFS, and verify the progress and success of the data loading. ###### Create a database and a table[​](#create-a-database-and-a-table "Direct link to Create a database and a table") Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand (we recommend that the table has the same schema as the Parquet file that you want to load from HDFS): ```sql CREATE TABLE user_behavior ( UserID int(11), ItemID int(11), CategoryID int(11), BehaviorType varchar(65533), Timestamp varbinary ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID); ``` ###### Start a Broker Load[​](#start-a-broker-load "Direct link to Start a Broker Load") Run the following command to start a Broker Load job that loads data from the data file `/user/amber/user_behavior_ten_million_rows.parquet` to the `user_behavior` table: ```sql LOAD LABEL user_behavior ( DATA INFILE("hdfs://:/user/amber/user_behavior_ten_million_rows.parquet") INTO TABLE user_behavior FORMAT AS "parquet" ) WITH BROKER ( "hadoop.security.authentication" = "simple", "username" = "", "password" = "" ) PROPERTIES ( "timeout" = "72000" ); ``` This job has four main sections: * `LABEL`: A string used when querying the state of the load job. * `LOAD` declaration: The source URI, source data format, and destination table name. * `BROKER`: The connection details for the source. * `PROPERTIES`: The timeout value and any other properties to apply to the load job. For detailed syntax and parameter descriptions, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). ###### Check load progress[​](#check-load-progress-1 "Direct link to Check load progress") You can query the progress of Broker Load jobs from the `information_schema.loads` view. This feature is supported from v3.1 onwards. ```sql SELECT * FROM information_schema.loads; ``` For information about the fields provided in the `loads` view, see [Information Schema](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md)). If you have submitted multiple load jobs, you can filter on the `LABEL` associated with the job. Example: ```sql SELECT * FROM information_schema.loads WHERE LABEL = 'user_behavior'; ``` In the output below there are two entries for the load job `user_behavior`: * The first record shows a state of `CANCELLED`. Scroll to `ERROR_MSG`, and you can see that the job has failed due to `listPath failed`. * The second record shows a state of `FINISHED`, which means that the job has succeeded. ```plaintext JOB_ID|LABEL |DATABASE_NAME|STATE |PROGRESS |TYPE |PRIORITY|SCAN_ROWS|FILTERED_ROWS|UNSELECTED_ROWS|SINK_ROWS|ETL_INFO|TASK_INFO |CREATE_TIME |ETL_START_TIME |ETL_FINISH_TIME |LOAD_START_TIME |LOAD_FINISH_TIME |JOB_DETAILS |ERROR_MSG |TRACKING_URL|TRACKING_SQL|REJECTED_RECORD_PATH| ------+-------------------------------------------+-------------+---------+-------------------+------+--------+---------+-------------+---------------+---------+--------+----------------------------------------------------+-------------------+-------------------+-------------------+-------------------+-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+------------+------------+--------------------+ 10121|user_behavior |mydatabase |CANCELLED|ETL:N/A; LOAD:N/A |BROKER|NORMAL | 0| 0| 0| 0| |resource:N/A; timeout(s):72000; max_filter_ratio:0.0|2023-08-10 14:59:30| | | |2023-08-10 14:59:34|{"All backends":{},"FileNumber":0,"FileSize":0,"InternalTableLoadBytes":0,"InternalTableLoadRows":0,"ScanBytes":0,"ScanRows":0,"TaskNumber":0,"Unfinished backends":{}} |type:ETL_RUN_FAIL; msg:listPath failed| | | | 10106|user_behavior |mydatabase |FINISHED |ETL:100%; LOAD:100%|BROKER|NORMAL | 86953525| 0| 0| 86953525| |resource:N/A; timeout(s):72000; max_filter_ratio:0.0|2023-08-10 14:50:15|2023-08-10 14:50:19|2023-08-10 14:50:19|2023-08-10 14:50:19|2023-08-10 14:55:10|{"All backends":{"a5fe5e1d-d7d0-4826-ba99-c7348f9a5f2f":[10004]},"FileNumber":1,"FileSize":1225637388,"InternalTableLoadBytes":2710603082,"InternalTableLoadRows":86953525,"ScanBytes":1225637388,"ScanRows":86953525,"TaskNumber":1,"Unfinished backends":{"a5| | | | | ``` After you confirm that the load job has finished, you can check a subset of the destination table to see if the data has been successfully loaded. Example: ```sql SELECT * from user_behavior LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 142 | 2869980 | 2939262 | pv | 2017-11-25 03:43:22 | | 142 | 2522236 | 1669167 | pv | 2017-11-25 15:14:12 | | 142 | 3031639 | 3607361 | pv | 2017-11-25 15:19:25 | +--------+---------+------------+--------------+---------------------+ ``` #### Use Pipe[​](#use-pipe "Direct link to Use Pipe") Starting from v3.2, StarRocks provides the Pipe loading method, which currently supports only the Parquet and ORC file formats. ##### Advantages of Pipe[​](#advantages-of-pipe "Direct link to Advantages of Pipe") Pipe is ideal for continuous data loading and large-scale data loading: * **Large-scale data loading in micro-batches helps reduce the cost of retries caused by data errors.** With the help of Pipe, StarRocks enables the efficient loading of a large number of data files with a significant data volume in total. Pipe automatically splits the files based on their number or size, breaking down the load job into smaller, sequential tasks. This approach ensures that errors in one file do not impact the entire load job. The load status of each file is recorded by Pipe, allowing you to easily identify and fix files that contain errors. By minimizing the need for retries due to data errors, this approach helps to reduce costs. * **Continuous data loading helps reduce manpower.** Pipe helps you write new or updated data files to a specific location and continuously load the new data from these files into StarRocks. After you create a Pipe job with `"AUTO_INGEST" = "TRUE"` specified, it will constantly monitor changes to the data files stored in the specified path and automatically load new or updated data from the data files into the destination StarRocks table. Additionally, Pipe performs file uniqueness checks to help prevent duplicate data loading.During the loading process, Pipe checks the uniqueness of each data file based on the file name and digest. If a file with a specific file name and digest has already been processed by a Pipe job, the Pipe job will skip all subsequent files with the same file name and digest. Note that HDFS uses LastModifiedTime as file digest. The load status of each data file is recorded and saved to the `information_schema.pipe_files` view. After a Pipe job associated with the view is deleted, the records about the files loaded in that job will also be deleted. ##### Data flow[​](#data-flow "Direct link to Data flow") ![Pipe data flow](/assets/images/pipe_data_flow-2a4dc0b44a06c987d9afc0ecf632f5d9.png) Pipe is ideal for continuous data loading and large-scale data loading: * **Large-scale data loading in micro-batches helps reduce the cost of retries caused by data errors.** With the help of Pipe, StarRocks enables the efficient loading of a large number of data files with a significant data volume in total. Pipe automatically splits the files based on their number or size, breaking down the load job into smaller, sequential tasks. This approach ensures that errors in one file do not impact the entire load job. The load status of each file is recorded by Pipe, allowing you to easily identify and fix files that contain errors. By minimizing the need for retries due to data errors, this approach helps to reduce costs. * **Continuous data loading helps reduce manpower.** Pipe helps you write new or updated data files to a specific location and continuously load the new data from these files into StarRocks. After you create a Pipe job with `"AUTO_INGEST" = "TRUE"` specified, it will constantly monitor changes to the data files stored in the specified path and automatically load new or updated data from the data files into the destination StarRocks table. Additionally, Pipe performs file uniqueness checks to help prevent duplicate data loading. During the loading process, Pipe checks the uniqueness of each data file based on the file name and digest. If a file with a specific file name and digest has already been processed by a Pipe job, the Pipe job will skip all subsequent files with the same file name and digest. Note that HDFS uses `LastModifiedTime` as file digest. The load status of each data file is recorded and saved to the `information_schema.pipe_files` view. After a Pipe job associated with the view is deleted, the records about the files loaded in that job will also be deleted. ##### Data flow[​](#data-flow-1 "Direct link to Data flow") ![Pipe data flow](/assets/images/pipe_data_flow-2a4dc0b44a06c987d9afc0ecf632f5d9.png) ##### Differences between Pipe and INSERT+FILES()[​](#differences-between-pipe-and-insertfiles "Direct link to Differences between Pipe and INSERT+FILES()") A Pipe job is split into one or more transactions based on the size and number of rows in each data file. Users can query the intermediate results during the loading process. In contrast, an INSERT+`FILES()` job is processed as a single transaction, and users are unable to view the data during the loading process. ##### File loading sequence[​](#file-loading-sequence "Direct link to File loading sequence") For each Pipe job, StarRocks maintains a file queue, from which it fetches and loads data files as micro-batches. Pipe does not ensure that the data files are loaded in the same order as they are uploaded. Therefore, newer data may be loaded prior to older data. ##### Typical example[​](#typical-example-1 "Direct link to Typical example") ###### Create a database and a table[​](#create-a-database-and-a-table-1 "Direct link to Create a database and a table") Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand (we recommend that the table have the same schema as the Parquet file you want to load from HDFS): ```sql CREATE TABLE user_behavior_replica ( UserID int(11), ItemID int(11), CategoryID int(11), BehaviorType varchar(65533), Timestamp varbinary ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID); ``` ###### Start a Pipe job[​](#start-a-pipe-job "Direct link to Start a Pipe job") Run the following command to start a Pipe job that loads data from the data file `/user/amber/user_behavior_ten_million_rows.parquet` to the `user_behavior_replica` table: ```sql CREATE PIPE user_behavior_replica PROPERTIES ( "AUTO_INGEST" = "TRUE" ) AS INSERT INTO user_behavior_replica SELECT * FROM FILES ( "path" = "hdfs://:/user/amber/user_behavior_ten_million_rows.parquet", "format" = "parquet", "hadoop.security.authentication" = "simple", "username" = "", "password" = "" ); ``` This job has four main sections: * `pipe_name`: The name of the pipe. The pipe name must be unique within the database to which the pipe belongs. * `INSERT_SQL`: The INSERT INTO SELECT FROM FILES statement that is used to load data from the specified source data file to the destination table. * `PROPERTIES`: A set of optional parameters that specify how to execute the pipe. These include `AUTO_INGEST`, `POLL_INTERVAL`, `BATCH_SIZE`, and `BATCH_FILES`. Specify these properties in the `"key" = "value"` format. For detailed syntax and parameter descriptions, see [CREATE PIPE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/CREATE_PIPE.md). ###### Check load progress[​](#check-load-progress-2 "Direct link to Check load progress") * Query the progress of Pipe jobs by using [SHOW PIPES](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/SHOW_PIPES.md). ```sql SHOW PIPES; ``` If you have submitted multiple load jobs, you can filter on the `NAME` associated with the job. Example: ```sql SHOW PIPES WHERE NAME = 'user_behavior_replica' \G *************************** 1. row *************************** DATABASE_NAME: mydatabase PIPE_ID: 10252 PIPE_NAME: user_behavior_replica STATE: RUNNING TABLE_NAME: mydatabase.user_behavior_replica LOAD_STATUS: {"loadedFiles":1,"loadedBytes":132251298,"loadingFiles":0,"lastLoadedTime":"2023-11-17 16:13:22"} LAST_ERROR: NULL CREATED_TIME: 2023-11-17 16:13:15 1 row in set (0.00 sec) ``` * Query the progress of Pipe jobs from the [`pipes`](https://docs.starrocks.io/docs/sql-reference/information_schema/pipes.md) view in the StarRocks Information Schema. ```sql SELECT * FROM information_schema.pipes; ``` If you have submitted multiple load jobs, you can filter on the `PIPE_NAME` associated with the job. Example: ```sql SELECT * FROM information_schema.pipes WHERE pipe_name = 'user_behavior_replica' \G *************************** 1. row *************************** DATABASE_NAME: mydatabase PIPE_ID: 10252 PIPE_NAME: user_behavior_replica STATE: RUNNING TABLE_NAME: mydatabase.user_behavior_replica LOAD_STATUS: {"loadedFiles":1,"loadedBytes":132251298,"loadingFiles":0,"lastLoadedTime":"2023-11-17 16:13:22"} LAST_ERROR: CREATED_TIME: 2023-11-17 16:13:15 1 row in set (0.00 sec) ``` ###### Check file status[​](#check-file-status "Direct link to Check file status") You can query the load status of the files loaded from the [`pipe_files`](https://docs.starrocks.io/docs/sql-reference/information_schema/pipe_files.md) view in the StarRocks Information Schema. ```sql SELECT * FROM information_schema.pipe_files; ``` If you have submitted multiple load jobs, you can filter on the `PIPE_NAME` associated with the job. Example: ```sql SELECT * FROM information_schema.pipe_files WHERE pipe_name = 'user_behavior_replica' \G *************************** 1. row *************************** DATABASE_NAME: mydatabase PIPE_ID: 10252 PIPE_NAME: user_behavior_replica FILE_NAME: hdfs://172.26.195.67:9000/user/amber/user_behavior_ten_million_rows.parquet FILE_VERSION: 1700035418838 FILE_SIZE: 132251298 LAST_MODIFIED: 2023-11-15 08:03:38 LOAD_STATE: FINISHED STAGED_TIME: 2023-11-17 16:13:16 START_LOAD_TIME: 2023-11-17 16:13:17 FINISH_LOAD_TIME: 2023-11-17 16:13:22 ERROR_MSG: 1 row in set (0.02 sec) ``` ###### Manage Pipes[​](#manage-pipes "Direct link to Manage Pipes") You can alter, suspend or resume, drop, or query the pipes you have created and retry to load specific data files. For more information, see [ALTER PIPE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/ALTER_PIPE.md), [SUSPEND or RESUME PIPE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/SUSPEND_or_RESUME_PIPE.md), [DROP PIPE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/DROP_PIPE.md), [SHOW PIPES](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/SHOW_PIPES.md), and [RETRY FILE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/RETRY_FILE.md). --- ### huawei --- ### Load data using INSERT This topic describes how to load data into StarRocks by using a SQL statement - INSERT. Similar to MySQL and many other database management systems, StarRocks supports loading data to an internal table with INSERT. You can insert one or more rows directly with the VALUES clause to test a function or a DEMO. You can also insert data defined by the results of a query into an internal table from an [external table](https://docs.starrocks.io/docs/data_source/External_table.md). From StarRocks v3.1 onwards, you can directly load data from files on cloud storage using the INSERT command and the table function [FILES()](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md). StarRocks v2.4 further supports overwriting data into a table by using INSERT OVERWRITE. The INSERT OVERWRITE statement integrates the following operations to implement the overwriting function: 1. Creates temporary partitions according to the partitions that store the original data. 2. Inserts data into the temporary partitions. 3. Swaps the original partitions with the temporary partitions. > **NOTE** > > If you need to verify the data before overwriting it, instead of using INSERT OVERWRITE, you can follow the above procedures to overwrite your data and verify it before swapping the partitions. From v3.4.0 onwards, StarRocks supports a new semantic - Dynamic Overwrite for INSERT OVERWRITE with partitioned tables. For more information, see [Dynamic Overwrite](#dynamic-overwrite). #### Precautions[​](#precautions "Direct link to Precautions") * You can cancel a synchronous INSERT transaction only by pressing the **Ctrl** and **C** keys from your MySQL client. * You can submit an asynchronous INSERT task using [SUBMIT TASK](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/ETL/SUBMIT_TASK.md). * As for the current version of StarRocks, the INSERT transaction fails by default if the data of any rows does not comply with the schema of the table. For example, the INSERT transaction fails if the length of a field in any row exceeds the length limit for the mapping field in the table. You can set the session variable `enable_insert_strict` to `false` to allow the transaction to continue by filtering out the rows that mismatch the table. * If you execute the INSERT statement frequently to load small batches of data into StarRocks, excessive data versions are generated. It severely affects query performance. We recommend that, in production, you should not load data with the INSERT command too often or use it as a routine for data loading on a daily basis. If your application or analytic scenario demand solutions to loading streaming data or small data batches separately, we recommend you use Apache Kafka® as your data source and load the data via Routine Load. * If you execute the INSERT OVERWRITE statement, StarRocks creates temporary partitions for the partitions which store the original data, inserts new data into the temporary partitions, and [swaps the original partitions with the temporary partitions](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md#use-a-temporary-partition-to-replace-the-current-partition). All these operations are executed in the FE Leader node. Hence, if the FE Leader node crashes while executing INSERT OVERWRITE command, the whole load transaction will fail, and the temporary partitions will be truncated. #### Preparation[​](#preparation "Direct link to Preparation") ##### Check privileges[​](#check-privileges "Direct link to Check privileges") You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. ##### Create objects[​](#create-objects "Direct link to Create objects") Create a database named `load_test`, and create a table `insert_wiki_edit` as the destination table and a table `source_wiki_edit` as the source table. > **NOTE** > > Examples demonstrated in this topic are based on the table `insert_wiki_edit` and the table `source_wiki_edit`. If you prefer working with your own tables and data, you can skip the preparation and move on to the next step. ```sql CREATE DATABASE IF NOT EXISTS load_test; USE load_test; CREATE TABLE insert_wiki_edit ( event_time DATETIME, channel VARCHAR(32) DEFAULT '', user VARCHAR(128) DEFAULT '', is_anonymous TINYINT DEFAULT '0', is_minor TINYINT DEFAULT '0', is_new TINYINT DEFAULT '0', is_robot TINYINT DEFAULT '0', is_unpatrolled TINYINT DEFAULT '0', delta INT DEFAULT '0', added INT DEFAULT '0', deleted INT DEFAULT '0' ) DUPLICATE KEY( event_time, channel, user, is_anonymous, is_minor, is_new, is_robot, is_unpatrolled ) PARTITION BY RANGE(event_time)( PARTITION p06 VALUES LESS THAN ('2015-09-12 06:00:00'), PARTITION p12 VALUES LESS THAN ('2015-09-12 12:00:00'), PARTITION p18 VALUES LESS THAN ('2015-09-12 18:00:00'), PARTITION p24 VALUES LESS THAN ('2015-09-13 00:00:00') ) DISTRIBUTED BY HASH(user); CREATE TABLE source_wiki_edit ( event_time DATETIME, channel VARCHAR(32) DEFAULT '', user VARCHAR(128) DEFAULT '', is_anonymous TINYINT DEFAULT '0', is_minor TINYINT DEFAULT '0', is_new TINYINT DEFAULT '0', is_robot TINYINT DEFAULT '0', is_unpatrolled TINYINT DEFAULT '0', delta INT DEFAULT '0', added INT DEFAULT '0', deleted INT DEFAULT '0' ) DUPLICATE KEY( event_time, channel,user, is_anonymous, is_minor, is_new, is_robot, is_unpatrolled ) PARTITION BY RANGE(event_time)( PARTITION p06 VALUES LESS THAN ('2015-09-12 06:00:00'), PARTITION p12 VALUES LESS THAN ('2015-09-12 12:00:00'), PARTITION p18 VALUES LESS THAN ('2015-09-12 18:00:00'), PARTITION p24 VALUES LESS THAN ('2015-09-13 00:00:00') ) DISTRIBUTED BY HASH(user); ``` > **NOTICE** > > Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). #### Insert data via INSERT INTO VALUES[​](#insert-data-via-insert-into-values "Direct link to Insert data via INSERT INTO VALUES") You can append one or more rows to a specific table by using INSERT INTO VALUES command. Multiple rows are separated by comma (,). For detailed instructions and parameter references, see [SQL Reference - INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md). > **CAUTION** > > Inserting data via INSERT INTO VALUES merely applies to the situation when you need to verify a DEMO with a small dataset. It is not recommended for a massive testing or production environment. To load mass data into StarRocks, see [Loading options](https://docs.starrocks.io/docs/loading/Loading_intro.md) for other options that suit your scenarios. The following example inserts two rows into the data source table `source_wiki_edit` with the label `insert_load_wikipedia`. Label is the unique identification label for each data load transaction within the database. ```sql INSERT INTO source_wiki_edit WITH LABEL insert_load_wikipedia VALUES ("2015-09-12 00:00:00","#en.wikipedia","AustinFF",0,0,0,0,0,21,5,0), ("2015-09-12 00:00:00","#ca.wikipedia","helloSR",0,1,0,1,0,3,23,0); ``` #### Insert data via INSERT INTO SELECT[​](#insert-data-via-insert-into-select "Direct link to Insert data via INSERT INTO SELECT") You can load the result of a query on a data source table into the target table via INSERT INTO SELECT command. INSERT INTO SELECT command performs ETL operations on the data from the data source table, and loads the data into an internal table in StarRocks. The data source can be one or more internal or external tables, or even data files on cloud storage. The target table MUST be an internal table in StarRocks. For detailed instructions and parameter references, see [SQL Reference - INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md). ##### Insert data from an internal or external table into an internal table[​](#insert-data-from-an-internal-or-external-table-into-an-internal-table "Direct link to Insert data from an internal or external table into an internal table") > **NOTE** > > Inserting data from an external table is identical to inserting data from an internal table. For simplicity, we only demonstrate how to insert data from an internal table in the following examples. * The following example inserts the data from the source table to the target table `insert_wiki_edit`. ```sql INSERT INTO insert_wiki_edit WITH LABEL insert_load_wikipedia_1 SELECT * FROM source_wiki_edit; ``` * The following example inserts the data from the source table to the `p06` and `p12` partitions of the target table `insert_wiki_edit`. If no partition is specified, the data will be inserted into all partitions. Otherwise, the data will be inserted only into the specified partition(s). ```sql INSERT INTO insert_wiki_edit PARTITION(p06, p12) WITH LABEL insert_load_wikipedia_2 SELECT * FROM source_wiki_edit; ``` Query the target table to make sure there is data in them. ```plain MySQL > select * from insert_wiki_edit; +---------------------+---------------+----------+--------------+----------+--------+----------+----------------+-------+-------+---------+ | event_time | channel | user | is_anonymous | is_minor | is_new | is_robot | is_unpatrolled | delta | added | deleted | +---------------------+---------------+----------+--------------+----------+--------+----------+----------------+-------+-------+---------+ | 2015-09-12 00:00:00 | #en.wikipedia | AustinFF | 0 | 0 | 0 | 0 | 0 | 21 | 5 | 0 | | 2015-09-12 00:00:00 | #ca.wikipedia | helloSR | 0 | 1 | 0 | 1 | 0 | 3 | 23 | 0 | +---------------------+---------------+----------+--------------+----------+--------+----------+----------------+-------+-------+---------+ 2 rows in set (0.00 sec) ``` If you truncate the `p06` and `p12` partitions, the data will not be returned in a query. ```plain MySQL > TRUNCATE TABLE insert_wiki_edit PARTITION(p06, p12); Query OK, 0 rows affected (0.01 sec) MySQL > select * from insert_wiki_edit; Empty set (0.00 sec) ``` * The following example inserts the `event_time` and `channel` columns from the source table to the target table `insert_wiki_edit`. Default values are used in the columns that are not specified here. ```sql INSERT INTO insert_wiki_edit WITH LABEL insert_load_wikipedia_3 ( event_time, channel ) SELECT event_time, channel FROM source_wiki_edit; ``` note From v3.3.1, specifying a column list in the INSERT INTO statement on a Primary Key table will perform Partial Updates (instead of Full Upsert in earlier versions). If the column list is not specified, the system will perform Full Upsert. ##### Insert data directly from files in an external source using FILES()[​](#insert-data-directly-from-files-in-an-external-source-using-files "Direct link to Insert data directly from files in an external source using FILES()") From v3.1 onwards, StarRocks supports directly loading data from files on cloud storage using the INSERT command and the [FILES()](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md) function, thereby you do not need to create an external catalog or file external table first. Besides, FILES() can automatically infer the table schema of the files, greatly simplifying the process of data loading. The following example inserts data rows from the Parquet file **parquet/insert\_wiki\_edit\_append.parquet** within the AWS S3 bucket `inserttest` into the table `insert_wiki_edit`: ```plain INSERT INTO insert_wiki_edit SELECT * FROM FILES( "path" = "s3://inserttest/parquet/insert_wiki_edit_append.parquet", "format" = "parquet", "aws.s3.access_key" = "XXXXXXXXXX", "aws.s3.secret_key" = "YYYYYYYYYY", "aws.s3.region" = "us-west-2" ); ``` #### Overwrite data via INSERT OVERWRITE VALUES[​](#overwrite-data-via-insert-overwrite-values "Direct link to Overwrite data via INSERT OVERWRITE VALUES") You can overwrite a specific table with one or more rows by using INSERT OVERWRITE VALUES command. Multiple rows are separated by comma (,). For detailed instructions and parameter references, see [SQL Reference - INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md). > **CAUTION** > > Overwriting data via INSERT OVERWRITE VALUES merely applies to the situation when you need to verify a DEMO with a small dataset. It is not recommended for a massive testing or production environment. To load mass data into StarRocks, see [Loading options](https://docs.starrocks.io/docs/loading/Loading_intro.md) for other options that suit your scenarios. Query the source table and the target table to make sure there is data in them. ```plain MySQL > SELECT * FROM source_wiki_edit; +---------------------+---------------+----------+--------------+----------+--------+----------+----------------+-------+-------+---------+ | event_time | channel | user | is_anonymous | is_minor | is_new | is_robot | is_unpatrolled | delta | added | deleted | +---------------------+---------------+----------+--------------+----------+--------+----------+----------------+-------+-------+---------+ | 2015-09-12 00:00:00 | #ca.wikipedia | helloSR | 0 | 1 | 0 | 1 | 0 | 3 | 23 | 0 | | 2015-09-12 00:00:00 | #en.wikipedia | AustinFF | 0 | 0 | 0 | 0 | 0 | 21 | 5 | 0 | +---------------------+---------------+----------+--------------+----------+--------+----------+----------------+-------+-------+---------+ 2 rows in set (0.02 sec) MySQL > SELECT * FROM insert_wiki_edit; +---------------------+---------------+----------+--------------+----------+--------+----------+----------------+-------+-------+---------+ | event_time | channel | user | is_anonymous | is_minor | is_new | is_robot | is_unpatrolled | delta | added | deleted | +---------------------+---------------+----------+--------------+----------+--------+----------+----------------+-------+-------+---------+ | 2015-09-12 00:00:00 | #ca.wikipedia | helloSR | 0 | 1 | 0 | 1 | 0 | 3 | 23 | 0 | | 2015-09-12 00:00:00 | #en.wikipedia | AustinFF | 0 | 0 | 0 | 0 | 0 | 21 | 5 | 0 | +---------------------+---------------+----------+--------------+----------+--------+----------+----------------+-------+-------+---------+ 2 rows in set (0.01 sec) ``` The following example overwrites the source table `source_wiki_edit` with two new rows. ```sql INSERT OVERWRITE source_wiki_edit WITH LABEL insert_load_wikipedia_ow VALUES ("2015-09-12 00:00:00","#cn.wikipedia","GELongstreet",0,0,0,0,0,36,36,0), ("2015-09-12 00:00:00","#fr.wikipedia","PereBot",0,1,0,1,0,17,17,0); ``` #### Overwrite data via INSERT OVERWRITE SELECT[​](#overwrite-data-via-insert-overwrite-select "Direct link to Overwrite data via INSERT OVERWRITE SELECT") You can overwrite a table with the result of a query on a data source table via INSERT OVERWRITE SELECT command. INSERT OVERWRITE SELECT statement performs ETL operations on the data from one or more internal or external tables, and overwrites an internal table with the data For detailed instructions and parameter references, see [SQL Reference - INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md). > **NOTE** > > Loading data from an external table is identical to loading data from an internal table. For simplicity, we only demonstrate how to overwrite the target table with the data from an internal table in the following examples. Query the source table and the target table to make sure that they hold different rows of data. ```plain MySQL > SELECT * FROM source_wiki_edit; +---------------------+---------------+--------------+--------------+----------+--------+----------+----------------+-------+-------+---------+ | event_time | channel | user | is_anonymous | is_minor | is_new | is_robot | is_unpatrolled | delta | added | deleted | +---------------------+---------------+--------------+--------------+----------+--------+----------+----------------+-------+-------+---------+ | 2015-09-12 00:00:00 | #cn.wikipedia | GELongstreet | 0 | 0 | 0 | 0 | 0 | 36 | 36 | 0 | | 2015-09-12 00:00:00 | #fr.wikipedia | PereBot | 0 | 1 | 0 | 1 | 0 | 17 | 17 | 0 | +---------------------+---------------+--------------+--------------+----------+--------+----------+----------------+-------+-------+---------+ 2 rows in set (0.02 sec) MySQL > SELECT * FROM insert_wiki_edit; +---------------------+---------------+----------+--------------+----------+--------+----------+----------------+-------+-------+---------+ | event_time | channel | user | is_anonymous | is_minor | is_new | is_robot | is_unpatrolled | delta | added | deleted | +---------------------+---------------+----------+--------------+----------+--------+----------+----------------+-------+-------+---------+ | 2015-09-12 00:00:00 | #en.wikipedia | AustinFF | 0 | 0 | 0 | 0 | 0 | 21 | 5 | 0 | | 2015-09-12 00:00:00 | #ca.wikipedia | helloSR | 0 | 1 | 0 | 1 | 0 | 3 | 23 | 0 | +---------------------+---------------+----------+--------------+----------+--------+----------+----------------+-------+-------+---------+ 2 rows in set (0.01 sec) ``` * The following example overwrites the table `insert_wiki_edit` with the data from the source table. ```sql INSERT OVERWRITE insert_wiki_edit WITH LABEL insert_load_wikipedia_ow_1 SELECT * FROM source_wiki_edit; ``` * The following example overwrites the `p06` and `p12` partitions of the table `insert_wiki_edit` with the data from the source table. ```sql INSERT OVERWRITE insert_wiki_edit PARTITION(p06, p12) WITH LABEL insert_load_wikipedia_ow_2 SELECT * FROM source_wiki_edit; ``` Query the target table to make sure there is data in them. ```plain MySQL > select * from insert_wiki_edit; +---------------------+---------------+--------------+--------------+----------+--------+----------+----------------+-------+-------+---------+ | event_time | channel | user | is_anonymous | is_minor | is_new | is_robot | is_unpatrolled | delta | added | deleted | +---------------------+---------------+--------------+--------------+----------+--------+----------+----------------+-------+-------+---------+ | 2015-09-12 00:00:00 | #fr.wikipedia | PereBot | 0 | 1 | 0 | 1 | 0 | 17 | 17 | 0 | | 2015-09-12 00:00:00 | #cn.wikipedia | GELongstreet | 0 | 0 | 0 | 0 | 0 | 36 | 36 | 0 | +---------------------+---------------+--------------+--------------+----------+--------+----------+----------------+-------+-------+---------+ 2 rows in set (0.01 sec) ``` If you truncate the `p06` and `p12` partitions, the data will not be returned in a query. ```plain MySQL > TRUNCATE TABLE insert_wiki_edit PARTITION(p06, p12); Query OK, 0 rows affected (0.01 sec) MySQL > select * from insert_wiki_edit; Empty set (0.00 sec) ``` note For tables that use the `PARTITION BY column` strategy, INSERT OVERWRITE supports creating new partitions in the destination table by specifying the value of the partition key. Existing partitions are overwritten as usual. The following example creates the partitioned table `activity`, and creates a new partition in the table while inserting data into it: ```sql CREATE TABLE activity ( id INT NOT NULL, dt VARCHAR(10) NOT NULL ) ENGINE=OLAP DUPLICATE KEY(`id`) PARTITION BY (`id`, `dt`) DISTRIBUTED BY HASH(`id`); INSERT OVERWRITE activity PARTITION(id='4', dt='2022-01-01') WITH LABEL insert_activity_auto_partition VALUES ('4', '2022-01-01'); ``` * The following example overwrites the target table `insert_wiki_edit` with the `event_time` and `channel` columns from the source table. The default value is assigned to the columns into which no data is overwritten. ```sql INSERT OVERWRITE insert_wiki_edit WITH LABEL insert_load_wikipedia_ow_3 ( event_time, channel ) SELECT event_time, channel FROM source_wiki_edit; ``` ##### Dynamic Overwrite[​](#dynamic-overwrite "Direct link to Dynamic Overwrite") From v3.4.0 onwards, StarRocks supports a new semantic - Dynamic Overwrite for INSERT OVERWRITE with partitioned tables. Currently, the default behavior of INSERT OVERWRITE is as follows: * When overwriting a partitioned table as a whole (that is, without specifying the PARTITION clause), new data records will replace the data in their corresponding partitions. If there are partitions that are not involved, they will be truncated while the others are overwritten. * When overwriting an empty partitioned table (that is, with no partitions in it) and specifying the PARTITION clause, the system returns an error `ERROR 1064 (HY000): Getting analyzing error. Detail message: Unknown partition 'xxx' in table 'yyy'`. * When overwriting a partitioned table and specifying a non-existent partition in the PARTITION clause, the system returns an error `ERROR 1064 (HY000): Getting analyzing error. Detail message: Unknown partition 'xxx' in table 'yyy'`. * When overwriting a partitioned table with data records that do not match any of the specified partitions in the PARTITION clause, the system either returns an error `ERROR 1064 (HY000): Insert has filtered data in strict mode` (if the strict mode is enabled) or filters the unqualified data records (if the strict mode is disabled). The behavior of the new Dynamic Overwrite semantic is much different: When overwriting a partitioned table as a whole, new data records will replace the data in their corresponding partitions. If there are partitions that are not involved, they will be left alone, instead of being truncated or deleted. And if there are new data records correspond to a non-existent partition, the system will create the partition. The Dynamic Overwrite semantic is disabled by default. To enable it, you need to set the system variable `dynamic_overwrite` to `true`. Enable Dynamic Overwrite in the current session: ```sql SET dynamic_overwrite = true; ``` You can also set it in the hint of the INSERT OVERWRITE statement to allow it take effect for the statement only:. Example: ```sql INSERT /*+set_var(dynamic_overwrite = true)*/ OVERWRITE insert_wiki_edit SELECT * FROM source_wiki_edit; ``` #### Insert data into a table with generated columns[​](#insert-data-into-a-table-with-generated-columns "Direct link to Insert data into a table with generated columns") A generated column is a special column whose value is derived from a pre-defined expression or evaluation based on other columns. Generated columns are especially useful when your query requests involve evaluations of expensive expressions, for example, querying a certain field from a JSON value, or calculating ARRAY data. StarRocks evaluates the expression and stores the results in the generated columns while data is being loaded into the table, thereby avoiding the expression evaluation during queries and improving the query performance. You can load data into a table with generated columns using INSERT. The following example creates a table `insert_generated_columns` and inserts a row into it. The table contains two generated columns: `avg_array` and `get_string`. `avg_array` calculates the average value of ARRAY data in `data_array`, and `get_string` extracts the strings from the JSON path `a` in `data_json`. ```sql CREATE TABLE insert_generated_columns ( id INT(11) NOT NULL COMMENT "ID", data_array ARRAY NOT NULL COMMENT "ARRAY", data_json JSON NOT NULL COMMENT "JSON", avg_array DOUBLE NULL AS array_avg(data_array) COMMENT "Get the average of ARRAY", get_string VARCHAR(65533) NULL AS get_json_string(json_string(data_json), '$.a') COMMENT "Extract JSON string" ) ENGINE=OLAP PRIMARY KEY(id) DISTRIBUTED BY HASH(id); INSERT INTO insert_generated_columns VALUES (1, [1,2], parse_json('{"a" : 1, "b" : 2}')); ``` > **NOTE** > > Directly loading data into generated columns is not supported. You can query the table to check the data within it. ```plain mysql> SELECT * FROM insert_generated_columns; +------+------------+------------------+-----------+------------+ | id | data_array | data_json | avg_array | get_string | +------+------------+------------------+-----------+------------+ | 1 | [1,2] | {"a": 1, "b": 2} | 1.5 | 1 | +------+------------+------------------+-----------+------------+ 1 row in set (0.02 sec) ``` #### INSERT data with PROPERTIES[​](#insert-data-with-properties "Direct link to INSERT data with PROPERTIES") From v3.4.0 onwards, INSERT statements support configuring PROPERTIES, which can serve a wide variety of purposes. PROPERTIES overrides their corresponding variables. ##### Enable strict mode[​](#enable-strict-mode "Direct link to Enable strict mode") From v3.4.0 onwards, you can enable strict mode and set `max_filter_ratio` for INSERT from FILES(). Strict mode for INSERT from FILES() has the same behavior as that of other loading methods. If you want to load a dataset with some unqualified rows, you either filter these unqualified rows or load them and assign NULL values to the unqualified columns. You can achieve them by using the properties `strict_mode` and `max_filter_ratio`. * To filter the unqualified rows: set `strict_mode` to `true`, and `max_filter_ratio` to a desired value. * To load all unqualified rows with NULL values: set `strict_mode` to `false`. The following example inserts data rows from the Parquet file **parquet/insert\_wiki\_edit\_append.parquet** within the AWS S3 bucket `inserttest` into the table `insert_wiki_edit`, enables strict mode to filter the unqualified data records, and tolerates at most 10% of error data: ```sql INSERT INTO insert_wiki_edit PROPERTIES( "strict_mode" = "true", "max_filter_ratio" = "0.1" ) SELECT * FROM FILES( "path" = "s3://inserttest/parquet/insert_wiki_edit_append.parquet", "format" = "parquet", "aws.s3.access_key" = "XXXXXXXXXX", "aws.s3.secret_key" = "YYYYYYYYYY", "aws.s3.region" = "us-west-2" ); ``` note `strict_mode` and `max_filter_ratio` are supported only for INSERT from FILES(). INSERT from tables does not support these properties. ##### Set timeout duration[​](#set-timeout-duration "Direct link to Set timeout duration") From v3.4.0 onwards, you can set the timeout duration for INSERT statements with properties. The following example inserts the data from the source table `source_wiki_edit` to the target table `insert_wiki_edit` with the timeout duration set to `2` seconds. ```sql INSERT INTO insert_wiki_edit PROPERTIES( "timeout" = "2" ) SELECT * FROM source_wiki_edit; ``` note From v3.4.0 onwards, you can also set the INSERT timeout duration using the system variable `insert_timeout`, which applies to operations involving INSERT (for example, UPDATE, DELETE, CTAS, materialized view refresh, statistics collection, and PIPE). In versions earlier than v3.4.0, the corresponding variable is `query_timeout`. ##### Match column by name[​](#match-column-by-name "Direct link to Match column by name") By default, INSERT matches the columns in the source and the target tables by their positions, that is, the mapping of the columns in the statement. The following example explicitly matches each column in the source and target tables by their positions: ```sql INSERT INTO insert_wiki_edit ( event_time, channel, user ) SELECT event_time, channel, user FROM source_wiki_edit; ``` The column mapping will change if you changed the order of `channel` and `user` in either the column list or the SELECT statement. ```sql INSERT INTO insert_wiki_edit ( event_time, channel, user ) SELECT event_time, user, channel FROM source_wiki_edit; ``` Here, the ingested data are probably not what you want, because `channel` in the target table `insert_wiki_edit` will be filled with data from `user` in the source table `source_wiki_edit`. By adding `BY NAME` clause in the INSERT statement, the system will detect the column names in the source and the target tables, and match the columns with the same name. note * You cannot specify the column list if `BY NAME` is specified. * If `BY NAME` is not specified, the system matches the columns by the position of the columns in the column list and the SELECT statement. The following example matches each column in the source and target tables by their names: ```sql INSERT INTO insert_wiki_edit BY NAME SELECT event_time, user, channel FROM source_wiki_edit; ``` In this case, changing the order of `channel` and `user` will not change the column mapping. #### Load data asynchronously using INSERT[​](#load-data-asynchronously-using-insert "Direct link to Load data asynchronously using INSERT") Loading data with INSERT submits a synchronous transaction, which may fail because of session interruption or timeout. You can submit an asynchronous INSERT transaction using [SUBMIT TASK](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/ETL/SUBMIT_TASK.md). This feature is supported since StarRocks v2.5. * The following example asynchronously inserts the data from the source table to the target table `insert_wiki_edit`. ```sql SUBMIT TASK AS INSERT INTO insert_wiki_edit SELECT * FROM source_wiki_edit; ``` * The following example asynchronously overwrites the table `insert_wiki_edit` with the data from the source table. ```sql SUBMIT TASK AS INSERT OVERWRITE insert_wiki_edit SELECT * FROM source_wiki_edit; ``` * The following example asynchronously overwrites the table `insert_wiki_edit` with the data from the source table, and extends the query timeout to `100000` seconds using hint. ```sql SUBMIT /*+set_var(insert_timeout=100000)*/ TASK AS INSERT OVERWRITE insert_wiki_edit SELECT * FROM source_wiki_edit; ``` * The following example asynchronously overwrites the table `insert_wiki_edit` with the data from the source table, and specifies the task name as `async`. ```sql SUBMIT TASK async AS INSERT OVERWRITE insert_wiki_edit SELECT * FROM source_wiki_edit; ``` You can check the status of an asynchronous INSERT task by querying the metadata view `task_runs` in Information Schema. The following example checks the status of the INSERT task `async`. ```sql SELECT * FROM information_schema.task_runs WHERE task_name = 'async'; ``` #### Check the INSERT job status[​](#check-the-insert-job-status "Direct link to Check the INSERT job status") ##### Check via the result[​](#check-via-the-result "Direct link to Check via the result") A synchronous INSERT transaction returns different status in accordance with the result of the transaction. * **Transaction succeeds** StarRocks returns the following if the transaction succeeds: ```plain Query OK, 2 rows affected (0.05 sec) {'label':'insert_load_wikipedia', 'status':'VISIBLE', 'txnId':'1006'} ``` * **Transaction fails** If all rows of data fail to be loaded into the target table, the INSERT transaction fails. StarRocks returns the following if the transaction fails: ```plain ERROR 1064 (HY000): Insert has filtered data in strict mode, tracking_url=http://x.x.x.x:yyyy/api/_load_error_log?file=error_log_9f0a4fd0b64e11ec_906bbede076e9d08 ``` You can locate the problem by checking the log with `tracking_url`. ##### Check via Information Schema[​](#check-via-information-schema "Direct link to Check via Information Schema") You can use the [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md) statement to query the results of one or more load jobs from the `loads` table in the `information_schema` database. This feature is supported from v3.1 onwards. Example 1: Query the results of load jobs executed on the `load_test` database, sort the results by creation time (`CREATE_TIME`) in descending order, and only return the top result. ```sql SELECT * FROM information_schema.loads WHERE database_name = 'load_test' ORDER BY create_time DESC LIMIT 1\G ``` Example 2: Query the result of the load job (whose label is `insert_load_wikipedia`) executed on the `load_test` database: ```sql SELECT * FROM information_schema.loads WHERE database_name = 'load_test' and label = 'insert_load_wikipedia'\G ``` The return is as follows: ```plain *************************** 1. row *************************** JOB_ID: 21319 LABEL: insert_load_wikipedia DATABASE_NAME: load_test STATE: FINISHED PROGRESS: ETL:100%; LOAD:100% TYPE: INSERT PRIORITY: NORMAL SCAN_ROWS: 0 FILTERED_ROWS: 0 UNSELECTED_ROWS: 0 SINK_ROWS: 2 ETL_INFO: TASK_INFO: resource:N/A; timeout(s):300; max_filter_ratio:0.0 CREATE_TIME: 2023-08-09 10:42:23 ETL_START_TIME: 2023-08-09 10:42:23 ETL_FINISH_TIME: 2023-08-09 10:42:23 LOAD_START_TIME: 2023-08-09 10:42:23 LOAD_FINISH_TIME: 2023-08-09 10:42:24 JOB_DETAILS: {"All backends":{"5ebf11b5-365e-11ee-9e4a-7a563fb695da":[10006]},"FileNumber":0,"FileSize":0,"InternalTableLoadBytes":175,"InternalTableLoadRows":2,"ScanBytes":0,"ScanRows":0,"TaskNumber":1,"Unfinished backends":{"5ebf11b5-365e-11ee-9e4a-7a563fb695da":[]}} ERROR_MSG: NULL TRACKING_URL: NULL TRACKING_SQL: NULL REJECTED_RECORD_PATH: NULL 1 row in set (0.01 sec) ``` For information about the fields in the return results, see [Information Schema > loads](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md). ##### Check via curl command[​](#check-via-curl-command "Direct link to Check via curl command") You can check the INSERT transaction status by using curl command. Launch a terminal, and execute the following command: ```bash curl --location-trusted -u : \ http://:/api//_load_info?label= ``` The following example checks the status of the transaction with label `insert_load_wikipedia`. ```bash curl --location-trusted -u : \ http://x.x.x.x:8030/api/load_test/_load_info?label=insert_load_wikipedia ``` > **NOTE** > > If you use an account for which no password is set, you need to input only `:`. The return is as follows: ```plain { "jobInfo":{ "dbName":"load_test", "tblNames":[ "source_wiki_edit" ], "label":"insert_load_wikipedia", "state":"FINISHED", "failMsg":"", "trackingUrl":"" }, "status":"OK", "msg":"Success" } ``` #### Configuration[​](#configuration "Direct link to Configuration") You can set the following configuration items for INSERT transaction: * **FE configuration** | FE configuration | Description | | -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | insert\_load\_default\_timeout\_second | Default timeout for INSERT transaction. Unit: second. If the current INSERT transaction is not completed within the time set by this parameter, it will be canceled by the system and the status will be CANCELLED. As for current version of StarRocks, you can only specify a uniform timeout for all INSERT transactions using this parameter, and you cannot set a different timeout for a specific INSERT transaction. The default is 3600 seconds (1 hour). If the INSERT transaction cannot be completed within the specified time, you can extend the timeout by adjusting this parameter. | * **Session variables** | Session variable | Description | | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | enable\_insert\_strict | Switch value to control if the INSERT transaction is tolerant of invalid data rows. When it is set to `true`, the transaction fails if any of the data rows is invalid. When it is set to `false`, the transaction succeeds when at least one row of data has been loaded correctly, and the label will be returned. The default is `true`. You can set this variable with `SET enable_insert_strict = {true or false};` command. | | insert\_timeout | Timeout for the INSERT statement. Unit: second. You can set this variable with the `SET insert_timeout = xxx;` command. | --- ### Introduction You can import semi-structured data (for example, JSON) by using stream load or routine load. #### Use Scenarios[​](#use-scenarios "Direct link to Use Scenarios") * Stream Load: For JSON data stored in text files, use stream load to import. * Routine Load: For JSON data in Kafka, use routine load to import. ##### Stream Load Import[​](#stream-load-import "Direct link to Stream Load Import") Sample data: ```json { "id": 123, "city" : "beijing"}, { "id": 456, "city" : "shanghai"}, ... ``` Example: ```shell curl -v --location-trusted -u : \ -H "format: json" -H "jsonpaths: [\"$.id\", \"$.city\"]" \ -T example.json \ http://FE_HOST:HTTP_PORT/api/DATABASE/TABLE/_stream_load ``` The `format: json` parameter allows you to execute the format of the imported data. `jsonpaths` is used to execute the corresponding data import path. Related parameters: * `jsonpaths`: Select the JSON path for each column * `json_root`: Select the column where the JSON starts to be parsed * `strip_outer_array`: Crop the outermost array field * `strict_mode`: Strictly filter for column type conversion during import When the JSON data schema and StarRocks data schema are not exactly the same, modify the `Jsonpath`. Sample data: ```json {"k1": 1, "k2": 2} ``` Import example: ```bash curl -v --location-trusted -u : \ -H "format: json" -H "jsonpaths: [\"$.k2\", \"$.k1\"]" \ -H "columns: k2, tmp_k1, k1 = tmp_k1 * 100" \ -T example.json \ http://127.0.0.1:8030/api/db1/tbl1/_stream_load ``` The ETL operation of multiplying k1 by 100 is performed during the import, and the column is matched with the original data by `Jsonpath`. The import results are as follows: ```plain +------+------+ | k1 | k2 | +------+------+ | 100 | 2 | +------+------+ ``` For missing columns, if the column definition is nullable, then `NULL` will be added, or the default value can be added by `ifnull`. Sample data: ```json [ {"k1": 1, "k2": "a"}, {"k1": 2}, {"k1": 3, "k2": "c"}, ] ``` Import Example-1: ```shell curl -v --location-trusted -u : \ -H "format: json" -H "strip_outer_array: true" \ -T example.json \ http://127.0.0.1:8030/api/db1/tbl1/_stream_load ``` The import results are as follows: ```plain +------+------+ | k1 | k2 | +------+------+ | 1 | a | +------+------+ | 2 | NULL | +------+------+ | 3 | c | +------+------+ ``` Import Example-2: ```shell curl -v --location-trusted -u : \ -H "format: json" -H "strip_outer_array: true" \ -H "jsonpaths: [\"$.k1\", \"$.k2\"]" \ -H "columns: k1, tmp_k2, k2 = ifnull(tmp_k2, 'x')" \ -T example.json \ http://127.0.0.1:8030/api/db1/tbl1/_stream_load ``` The import results are as follows: ```plain +------+------+ | k1 | k2 | +------+------+ | 1 | a | +------+------+ | 2 | x | +------+------+ | 3 | c | +------+------+ ``` ##### Routine Load Import[​](#routine-load-import "Direct link to Routine Load Import") Similar to stream load, the message content of Kafka data sources is treated as a complete JSON data. 1. If a message contains multiple rows of data in array format, all rows will be imported and Kafka's offset will only be incremented by 1. 2. If a JSON in Array format represents multiple rows of data, but the parsing of the JSON fails due to a JSON format error, the error row will only be incremented by 1 (given that the parsing fails, StarRocks cannot actually determine how many rows of data it contains, and can only record the error data as one row). ##### Use Canal to import StarRocks from MySQL with incremental sync binlogs[​](#use-canal-to-import-starrocks-from-mysql-with-incremental-sync-binlogs "Direct link to Use Canal to import StarRocks from MySQL with incremental sync binlogs") [Canal](https://github.com/alibaba/canal) is an open-source MySQL binlog synchronization tool from Alibaba, through which we can synchronize MySQL data to Kafka. The data is generated in JSON format in Kafka. Here is a demonstration of how to use routine load to synchronize data in Kafka for incremental data synchronization with MySQL. * In MySQL we have a data table with the following table creation statement. ```sql CREATE TABLE `query_record` ( `query_id` varchar(64) NOT NULL, `conn_id` int(11) DEFAULT NULL, `fe_host` varchar(32) DEFAULT NULL, `user` varchar(32) DEFAULT NULL, `start_time` datetime NOT NULL, `end_time` datetime DEFAULT NULL, `time_used` double DEFAULT NULL, `state` varchar(16) NOT NULL, `error_message` text, `sql` text NOT NULL, `database` varchar(128) NOT NULL, `profile` longtext, `plan` longtext, PRIMARY KEY (`query_id`), KEY `idx_start_time` (`start_time`) USING BTREE ) ENGINE=InnoDB DEFAULT CHARSET=utf8 ``` * Prerequisite: Make sure MySQL has binlog enabled and the format is ROW. ```bash [mysqld] log-bin=mysql-bin # Enable binlog binlog-format=ROW # Select ROW mode server_id=1 # MySQL replication need to be defined, and do not duplicate canal's slaveId ``` * Create an account and grant privileges to the secondary MySQL server: ```sql CREATE USER canal IDENTIFIED BY 'canal'; GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%'; -- GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%'; FLUSH PRIVILEGES; ``` * Then download and install Canal. ```bash wget https://github.com/alibaba/canal/releases/download/canal-1.0.17/canal.deployer-1.0.17.tar.gz mkdir /tmp/canal tar zxvf canal.deployer-$version.tar.gz -C /tmp/canal ``` * Modify the configuration (MySQL related). `$ vi conf/example/instance.properties` ```bash ## mysql serverId canal.instance.mysql.slaveId = 1234 #position info, need to change to your own database information canal.instance.master.address = 127.0.0.1:3306 canal.instance.master.journal.name = canal.instance.master.position = canal.instance.master.timestamp = #canal.instance.standby.address = #canal.instance.standby.journal.name = #canal.instance.standby.position = #canal.instance.standby.timestamp = #username/password, need to change to your own database information canal.instance.dbUsername = canal canal.instance.dbPassword = canal canal.instance.defaultDatabaseName = canal.instance.connectionCharset = UTF-8 #table regex canal.instance.filter.regex = .\*\\\\..\* # Select the name of the table to be synchronized and the partition name of the kafka target. canal.mq.dynamicTopic=databasename.query_record canal.mq.partitionHash= databasename.query_record:query_id ``` * Modify the configuration (Kafka related). `$ vi /usr/local/canal/conf/canal.properties` ```bash # Available options: tcp(by default), kafka, RocketMQ canal.serverMode = kafka # ... # kafka/rocketmq Cluster Configuration: 192.168.1.117:9092,192.168.1.118:9092,192.168.1.119:9092 canal.mq.servers = 127.0.0.1:6667 canal.mq.retries = 0 # This value can be increased in flagMessage mode, but do not exceed the maximum size of the MQ message. canal.mq.batchSize = 16384 canal.mq.maxRequestSize = 1048576 # In flatMessage mode, please change this value to a larger value, 50-200 is recommended. canal.mq.lingerMs = 1 canal.mq.bufferMemory = 33554432 # Canal's batch size with a default value of 50K. Please do not exceed 1M due to Kafka's maximum message size limit (under 900K) canal.mq.canalBatchSize = 50 # Timeout of `Canal get`, in milliseconds. Empty indicates unlimited timeout. canal.mq.canalGetTimeout = 100 # Whether the object is in flat json format canal.mq.flatMessage = false canal.mq.compressionType = none canal.mq.acks = all # Whether Kafka message delivery uses transactions canal.mq.transaction = false ``` * Initiation `bin/startup.sh` The corresponding synchronization log is shown in `logs/example/example.log` and in Kafka, with the following format: ```json { "data": [{ "query_id": "3c7ebee321e94773-b4d79cc3f08ca2ac", "conn_id": "34434", "fe_host": "172.26.34.139", "user": "zhaoheng", "start_time": "2020-10-19 20:40:10.578", "end_time": "2020-10-19 20:40:10", "time_used": "1.0", "state": "FINISHED", "error_message": "", "sql": "COMMIT", "database": "", "profile": "", "plan": "" }, { "query_id": "7ff2df7551d64f8e-804004341bfa63ad", "conn_id": "34432", "fe_host": "172.26.34.139", "user": "zhaoheng", "start_time": "2020-10-19 20:40:10.566", "end_time": "2020-10-19 20:40:10", "time_used": "0.0", "state": "FINISHED", "error_message": "", "sql": "COMMIT", "database": "", "profile": "", "plan": "" }, { "query_id": "3a4b35d1c1914748-be385f5067759134", "conn_id": "34440", "fe_host": "172.26.34.139", "user": "zhaoheng", "start_time": "2020-10-19 20:40:10.601", "end_time": "1970-01-01 08:00:00", "time_used": "-1.0", "state": "RUNNING", "error_message": "", "sql": " SELECT SUM(length(lo_custkey)), SUM(length(c_custkey)) FROM lineorder_str INNER JOIN customer_str ON lo_custkey=c_custkey;", "database": "ssb", "profile": "", "plan": "" }], "database": "center_service_lihailei", "es": 1603111211000, "id": 122, "isDdl": false, "mysqlType": { "query_id": "varchar(64)", "conn_id": "int(11)", "fe_host": "varchar(32)", "user": "varchar(32)", "start_time": "datetime(3)", "end_time": "datetime", "time_used": "double", "state": "varchar(16)", "error_message": "text", "sql": "text", "database": "varchar(128)", "profile": "longtext", "plan": "longtext" }, "old": null, "pkNames": ["query_id"], "sql": "", "sqlType": { "query_id": 12, "conn_id": 4, "fe_host": 12, "user": 12, "start_time": 93, "end_time": 93, "time_used": 8, "state": 12, "error_message": 2005, "sql": 2005, "database": 12, "profile": 2005, "plan": 2005 }, "table": "query_record", "ts": 1603111212015, "type": "INSERT" } ``` Add `json_root` and `strip_outer_array = true` to import data from `data`. ```sql create routine load manual.query_job on query_record columns (query_id,conn_id,fe_host,user,start_time,end_time,time_used,state,error_message,`sql`,`database`,profile,plan) PROPERTIES ( "format"="json", "json_root"="$.data", "desired_concurrent_number"="1", "strip_outer_array" ="true", "max_error_number"="1000" ) FROM KAFKA ( "kafka_broker_list"= "172.26.92.141:9092", "kafka_topic" = "databasename.query_record" ); ``` This completes the near real-time synchronization of data from MySQL to StarRocks. View status and error messages of the import job by `show routine load`. --- ### Load data using Kafka connector StarRocks provides a self-developed connector named Apache Kafka® connector (StarRocks Connector for Apache Kafka®, Kafka connector for short), as a sink connector, that continuously consumes messages from Kafka and loads them into StarRocks. The Kafka connector guarantees at-least-once semantics. The Kafka connector can seamlessly integrate with Kafka Connect, which allows StarRocks better integrated with the Kafka ecosystem. It is a wise choice if you want to load real-time data into StarRocks. Compared with Routine Load, it is recommended to use the Kafka connector in the following scenarios: * Compared with Routine Load which only supports loading data in CSV, JSON, and Avro formats, Kafka connector can load data in more formats, such as Protobuf. As long as data can be converted into JSON and CSV formats using Kafka Connect's converters, data can be loaded into StarRocks via the Kafka connector. * Customize data transformation, such as Debezium-formatted CDC data. * Load data from multiple Kafka topics. * Load data from Confluent Cloud. * Need finer control over load batch sizes, parallelism, and other parameters to achieve a balance between load speed and resource utilization. #### Preparations[​](#preparations "Direct link to Preparations") ##### Version requirements[​](#version-requirements "Direct link to Version requirements") | Connector | Kafka | StarRocks | Java | | --------- | --------- | ------------- | ---- | | 1.0.6 | 3.4+/4.0+ | 2.5 and later | 8 | | 1.0.5 | 3.4 | 2.5 and later | 8 | | 1.0.4 | 3.4 | 2.5 and later | 8 | | 1.0.3 | 3.4 | 2.5 and later | 8 | ##### Set up Kafka environment[​](#set-up-kafka-environment "Direct link to Set up Kafka environment") Both self-managed Apache Kafka clusters and Confluent Cloud are supported. * For a self-managed Apache Kafka cluster, you can refer to [Apache Kafka quickstart](https://kafka.apache.org/quickstart) to quickly deploy a Kafka cluster. Kafka Connect is already integrated into Kafka. * For Confluent Cloud, make sure that you have a Confluent account and have created a cluster. ##### Download Kafka connector[​](#download-kafka-connector "Direct link to Download Kafka connector") Submit the Kafka connector into Kafka Connect: * Self-managed Kafka cluster: Download [starrocks-connector-for-kafka-x.y.z-with-dependencies.jar](https://github.com/StarRocks/starrocks-connector-for-kafka/releases). * Confluent Cloud: Currently, the Kafka connector is not uploaded to Confluent Hub. You need to download [starrocks-connector-for-kafka-x.y.z-with-dependencies.jar](https://github.com/StarRocks/starrocks-connector-for-kafka/releases), package it into a ZIP file and upload the ZIP file to Confluent Cloud. ##### Network configuration[​](#network-configuration "Direct link to Network configuration") Ensure that the machine where Kafka is located can access the FE nodes of the StarRocks cluster via the [`http_port`](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#http_port) (default: `8030`) and [`query_port`](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#query_port) (default: `9030`), and the BE nodes via the [`be_http_port`](https://docs.starrocks.io/docs/administration/management/BE_configuration.md#be_http_port) (default: `8040`). #### Usage[​](#usage "Direct link to Usage") This section uses a self-managed Kafka cluster as an example to explain how to configure the Kafka connector and the Kafka Connect, and then run the Kafka Connect to load data into StarRocks. ##### Prepare a dataset[​](#prepare-a-dataset "Direct link to Prepare a dataset") Suppose that JSON-format data exists in the topic `test` in a Kafka cluster. ```json {"id":1,"city":"New York"} {"id":2,"city":"Los Angeles"} {"id":3,"city":"Chicago"} ``` ##### Create a table[​](#create-a-table "Direct link to Create a table") Create the table `test_tbl` in the database `example_db` in the StarRocks cluster according to the keys of the JSON-format data. ```sql CREATE DATABASE example_db; USE example_db; CREATE TABLE test_tbl (id INT, city STRING); ``` ##### Configure Kafka connector and Kafka Connect, and then run Kafka Connect to load data[​](#configure-kafka-connector-and-kafka-connect-and-then-run-kafka-connect-to-load-data "Direct link to Configure Kafka connector and Kafka Connect, and then run Kafka Connect to load data") ###### Run Kafka Connect in standalone mode[​](#run-kafka-connect-in-standalone-mode "Direct link to Run Kafka Connect in standalone mode") 1. Configure the Kafka connector. In the **config** directory under the Kafka installation directory, create the configuration file **connect-StarRocks-sink.properties** for the Kafka connector, and configure the following parameters. For more parameters and descriptions, see [Parameters](#Parameters). info * In this example, the Kafka connector provided by StarRocks is a sink connector that can continuously consume data from Kafka and load data into StarRocks. * If the source data is CDC data, such as data in Debezium format, and the StarRocks table is a Primary Key table, you also need to [configure `transform`](#load-debezium-formatted-cdc-data) in the configuration file **connect-StarRocks-sink.properties** for the Kafka connector provided by StarRocks, to synchronize the source data changes to the Primary Key table. ```yaml name=starrocks-kafka-connector connector.class=com.starrocks.connector.kafka.StarRocksSinkConnector topics=test key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter key.converter.schemas.enable=true value.converter.schemas.enable=false # The HTTP URL of the FE in your StarRocks cluster. The default port is 8030. starrocks.http.url=192.168.xxx.xxx:8030 # If the Kafka topic name is different from the StarRocks table name, you need to configure the mapping relationship between them. starrocks.topic2table.map=test:test_tbl # Enter the StarRocks username. starrocks.username=user1 # Enter the StarRocks password. starrocks.password=123456 starrocks.database.name=example_db sink.properties.strip_outer_array=true ``` 2. Configure and run the Kafka Connect. 1. Configure the Kafka Connect. In the configuration file **config/connect-standalone.properties** in the **config** directory, configure the following parameters. For more parameters and descriptions, see [Running Kafka Connect](https://kafka.apache.org/documentation.html#connect_running). ```yaml # The addresses of Kafka brokers. Multiple addresses of Kafka brokers need to be separated by commas (,). # Note that this example uses PLAINTEXT as the security protocol to access the Kafka cluster. If you are using other security protocol to access the Kafka cluster, you need to configure the relevant information in this file. bootstrap.servers=:9092 offset.storage.file.filename=/tmp/connect.offsets offset.flush.interval.ms=10000 key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter key.converter.schemas.enable=true value.converter.schemas.enable=false # The absolute path of starrocks-connector-for-kafka-x.y.z-with-dependencies.jar. plugin.path=/home/kafka-connect/starrocks-kafka-connector ``` 2. Run the Kafka Connect. ```bash CLASSPATH=/home/kafka-connect/starrocks-kafka-connector/* bin/connect-standalone.sh config/connect-standalone.properties config/connect-starrocks-sink.properties ``` ###### Run Kafka Connect in distributed mode[​](#run-kafka-connect-in-distributed-mode "Direct link to Run Kafka Connect in distributed mode") 1. Configure and run the Kafka Connect. 1. Configure the Kafka Connect. In the configuration file `config/connect-distributed.properties` in the **config** directory, configure the following parameters. For more parameters and descriptions, refer to [Running Kafka Connect](https://kafka.apache.org/documentation.html#connect_running). ```yaml # The addresses of Kafka brokers. Multiple addresses of Kafka brokers need to be separated by commas (,). # Note that this example uses PLAINTEXT as the security protocol to access the Kafka cluster. If you are using other security protocol to access the Kafka cluster, you need to configure the relevant information in this file. bootstrap.servers=:9092 offset.storage.file.filename=/tmp/connect.offsets offset.flush.interval.ms=10000 key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter key.converter.schemas.enable=true value.converter.schemas.enable=false # The absolute path of starrocks-connector-for-kafka-x.y.z-with-dependencies.jar. plugin.path=/home/kafka-connect/starrocks-kafka-connector ``` 2. Run the Kafka Connect. ```bash CLASSPATH=/home/kafka-connect/starrocks-kafka-connector/* bin/connect-distributed.sh config/connect-distributed.properties ``` 2. Configure and create the Kafka connector. Note that in distributed mode, you need to configure and create the Kafka connector through the REST API. For parameters and descriptions, see [Parameters](#Parameters). info * In this example, the Kafka connector provided by StarRocks is a sink connector that can continuously consume data from Kafka and load data into StarRocks. * If the source data is CDC data, such as data in Debezium format, and the StarRocks table is a Primary Key table, you also need to [configure `transform`](#load-debezium-formatted-cdc-data) in the configuration file **connect-StarRocks-sink.properties** for the Kafka connector provided by StarRocks, to synchronize the source data changes to the Primary Key table. ```shell curl -i http://127.0.0.1:8083/connectors -H "Content-Type: application/json" -X POST -d '{ "name":"starrocks-kafka-connector", "config":{ "connector.class":"com.starrocks.connector.kafka.StarRocksSinkConnector", "topics":"test", "key.converter":"org.apache.kafka.connect.json.JsonConverter", "value.converter":"org.apache.kafka.connect.json.JsonConverter", "key.converter.schemas.enable":"true", "value.converter.schemas.enable":"false", "starrocks.http.url":"192.168.xxx.xxx:8030", "starrocks.topic2table.map":"test:test_tbl", "starrocks.username":"user1", "starrocks.password":"123456", "starrocks.database.name":"example_db", "sink.properties.strip_outer_array":"true" } }' ``` ###### Query StarRocks table[​](#query-starrocks-table "Direct link to Query StarRocks table") Query the target StarRocks table `test_tbl`. ```mysql MySQL [example_db]> select * from test_tbl; +------+-------------+ | id | city | +------+-------------+ | 1 | New York | | 2 | Los Angeles | | 3 | Chicago | +------+-------------+ 3 rows in set (0.01 sec) ``` The data is successfully loaded when the above result is returned. #### Parameters[​](#parameters "Direct link to Parameters") ##### name[​](#name "Direct link to name") **Required**: YES
**Default value**:
**Description**: Name for this Kafka connector. It must be globally unique among all Kafka connectors within this Kafka Connect cluster. For example, starrocks-kafka-connector. ##### connector.class[​](#connectorclass "Direct link to connector.class") **Required**: YES
**Default value**:
**Description**: Class used by this Kafka connector's sink. Set the value to `com.starrocks.connector.kafka.StarRocksSinkConnector`. ##### topics[​](#topics "Direct link to topics") **Required**:
**Default value**:
**Description**: One or more topics to subscribe to, where each topic corresponds to a StarRocks table. By default, StarRocks assumes that the topic name matches the name of the StarRocks table. So StarRocks determines the target StarRocks table by using the topic name. Please choose either to fill in `topics` or `topics.regex` (below), but not both. However, if the StarRocks table name is not the same as the topic name, then use the optional `starrocks.topic2table.map` parameter (below) to specify the mapping from topic name to table name. ##### topics.regex[​](#topicsregex "Direct link to topics.regex") **Required**:
**Default value**: **Description**: Regular expression to match the one or more topics to subscribe to. For more description, see `topics`. Please choose either to fill in `topics.regex` or `topics` (above), but not both.
##### starrocks.topic2table.map[​](#starrockstopic2tablemap "Direct link to starrocks.topic2table.map") **Required**: NO
**Default value**:
**Description**: The mapping of the StarRocks table name and the topic name when the topic name is different from the StarRocks table name. The format is `:,:,...`. ##### starrocks.http.url[​](#starrockshttpurl "Direct link to starrocks.http.url") **Required**: YES
**Default value**:
**Description**: The HTTP URL of the FE in your StarRocks cluster. The format is `:,:,...`. Multiple addresses are separated by commas (,). For example, `192.168.xxx.xxx:8030,192.168.xxx.xxx:8030`. ##### starrocks.database.name[​](#starrocksdatabasename "Direct link to starrocks.database.name") **Required**: YES
**Default value**:
**Description**: The name of StarRocks database. ##### starrocks.username[​](#starrocksusername "Direct link to starrocks.username") **Required**: YES
**Default value**:
**Description**: The username of your StarRocks cluster account. The user needs the [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) privilege on the StarRocks table. ##### starrocks.password[​](#starrockspassword "Direct link to starrocks.password") **Required**: YES
**Default value**:
**Description**: The password of your StarRocks cluster account. ##### key.converter[​](#keyconverter "Direct link to key.converter") **Required**: NO
**Default value**: Key converter used by Kafka Connect cluster
**Description**: This parameter specifies the key converter for the sink connector (Kafka-connector-starrocks), which is used to deserialize the keys of Kafka data. The default key converter is the one used by Kafka Connect cluster. ##### value.converter[​](#valueconverter "Direct link to value.converter") **Required**: NO
**Default value**: Value converter used by Kafka Connect cluster
**Description**: This parameter specifies the value converter for the sink connector (Kafka-connector-starrocks), which is used to deserialize the values of Kafka data. The default value converter is the one used by Kafka Connect cluster. ##### key.converter.schema.registry.url[​](#keyconverterschemaregistryurl "Direct link to key.converter.schema.registry.url") **Required**: NO
**Default value**:
**Description**: Schema registry URL for the key converter. ##### value.converter.schema.registry.url[​](#valueconverterschemaregistryurl "Direct link to value.converter.schema.registry.url") **Required**: NO
**Default value**:
**Description**: Schema registry URL for the value converter. ##### tasks.max[​](#tasksmax "Direct link to tasks.max") **Required**: NO
**Default value**: 1
**Description**: The upper limit for the number of task threads that the Kafka connector can create, which is usually the same as the number of CPU cores on the worker nodes in the Kafka Connect cluster. You can tune this parameter to control load performance. ##### bufferflush.maxbytes[​](#bufferflushmaxbytes "Direct link to bufferflush.maxbytes") **Required**: NO
**Default value**: 94371840(90M)
**Description**: The maximum size of data that can be accumulated in memory before being sent to StarRocks at a time. The maximum value ranges from 64 MB to 10 GB. Keep in mind that the Stream Load SDK buffer may create multiple Stream Load jobs to buffer data. Therefore, the threshold mentioned here refers to the total data size. ##### bufferflush.intervalms[​](#bufferflushintervalms "Direct link to bufferflush.intervalms") **Required**: NO
**Default value**: 1000
**Description**: Interval for sending a batch of data which controls the load latency. Range: \[1000, 3600000]. ##### connect.timeoutms[​](#connecttimeoutms "Direct link to connect.timeoutms") **Required**: NO
**Default value**: 1000
**Description**: Timeout for connecting to the HTTP URL. Range: \[100, 60000]. ##### sink.properties.\*[​](#sinkproperties "Direct link to sink.properties.*") **Required**:
**Default value**:
**Description**: Stream Load parameters o control load behavior. For example, the parameter `sink.properties.format` specifies the format used for Stream Load, such as CSV or JSON. For a list of supported parameters and their descriptions, see [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). ##### sink.properties.format[​](#sinkpropertiesformat "Direct link to sink.properties.format") **Required**: NO
**Default value**: json
**Description**: The format used for Stream Load. The Kafka connector will transform each batch of data to the format before sending them to StarRocks. Valid values: `csv` and `json`. For more information, see [CSV parameters](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md#csv-parameters) and [JSON parameters](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md#json-parameters). ##### sink.properties.partial\_update[​](#sinkpropertiespartial_update "Direct link to sink.properties.partial_update") **Required**: NO
**Default value**: `FALSE`
**Description**: Whether to use partial updates. Valid values: `TRUE` and `FALSE`. Default value: `FALSE`, indicating to disable this feature. ##### sink.properties.partial\_update\_mode[​](#sinkpropertiespartial_update_mode "Direct link to sink.properties.partial_update_mode") **Required**: NO
**Default value**: `row`
**Description**: Specifies the mode for partial updates. Valid values: `row` and `column`. * The value `row` (default) means partial updates in row mode, which is more suitable for real-time updates with many columns and small batches. * The value `column` means partial updates in column mode, which is more suitable for batch updates with few columns and many rows. In such scenarios, enabling the column mode offers faster update speeds. For example, in a table with 100 columns, if only 10 columns (10% of the total) are updated for all rows, the update speed of the column mode is 10 times faster. #### Usage Notes[​](#usage-notes "Direct link to Usage Notes") ##### Flush Policy[​](#flush-policy "Direct link to Flush Policy") The Kafka connector will buffer the data in memory, and flush them in batch to StarRocks via Stream Load. The flush will be triggered when any of the following conditions are met: * The bytes of buffered rows reaches the limit `bufferflush.maxbytes`. * The elapsed time since the last flush reaches the limit `bufferflush.intervalms`. * The interval at which the connector tries committing offsets for tasks is reached. The interval is controlled by the Kafka Connect configuration [`offset.flush.interval.ms`](https://docs.confluent.io/platform/current/connect/references/allconfigs.html), and the default values is `60000`. For lower data latency, adjust these configurations in the Kafka connector settings. However, more frequent flushes will increase CPU and I/O usage. ##### Limits[​](#limits "Direct link to Limits") * It is not supported to flatten a single message from a Kafka topic into multiple data rows and load into StarRocks. * The sink of the Kafka connector provided by StarRocks guarantees at-least-once semantics. #### Best practices[​](#best-practices "Direct link to Best practices") ##### Load Debezium-formatted CDC data[​](#load-debezium-formatted-cdc-data "Direct link to Load Debezium-formatted CDC data") Debezium is a popular Change Data Capture (CDC) tool that supports monitoring data changes in various database systems and streaming these changes to Kafka. The following example demonstrates how to configure and use the Kafka connector to write PostgreSQL changes to a **Primary Key table** in StarRocks. ###### Step 1: Install and start Kafka[​](#step-1-install-and-start-kafka "Direct link to Step 1: Install and start Kafka") > **NOTE** > > You can skip this step if you have your own Kafka environment. 1. [Download](https://dlcdn.apache.org/kafka/) the latest Kafka release from the official site and extract the package. ```bash tar -xzf kafka_2.13-3.7.0.tgz cd kafka_2.13-3.7.0 ``` 2. Start the Kafka environment. Generate a Kafka cluster UUID. ```bash KAFKA_CLUSTER_ID="$(bin/kafka-storage.sh random-uuid)" ``` Format the log directories. ```bash bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c config/kraft/server.properties ``` Start the Kafka server. ```bash bin/kafka-server-start.sh config/kraft/server.properties ``` ###### Step 2: Configure PostgreSQL[​](#step-2-configure-postgresql "Direct link to Step 2: Configure PostgreSQL") 1. Make sure the PostgreSQL user is granted `REPLICATION` privileges. 2. Adjust PostgreSQL configuration. Set `wal_level` to `logical` in **postgresql.conf**. ```properties wal_level = logical ``` Restart the PostgreSQL server to apply changes. ```bash pg_ctl restart ``` 3. Prepare the dataset. Create a table and insert test data. ```sql CREATE TABLE customers ( id int primary key , first_name varchar(65533) NULL, last_name varchar(65533) NULL , email varchar(65533) NULL ); INSERT INTO customers VALUES (1,'a','a','a@a.com'); ``` 4. Verify the CDC log messages in Kafka. ```json { "schema": { "type": "struct", "fields": [ { "type": "struct", "fields": [ { "type": "int32", "optional": false, "field": "id" }, { "type": "string", "optional": true, "field": "first_name" }, { "type": "string", "optional": true, "field": "last_name" }, { "type": "string", "optional": true, "field": "email" } ], "optional": true, "name": "test.public.customers.Value", "field": "before" }, { "type": "struct", "fields": [ { "type": "int32", "optional": false, "field": "id" }, { "type": "string", "optional": true, "field": "first_name" }, { "type": "string", "optional": true, "field": "last_name" }, { "type": "string", "optional": true, "field": "email" } ], "optional": true, "name": "test.public.customers.Value", "field": "after" }, { "type": "struct", "fields": [ { "type": "string", "optional": false, "field": "version" }, { "type": "string", "optional": false, "field": "connector" }, { "type": "string", "optional": false, "field": "name" }, { "type": "int64", "optional": false, "field": "ts_ms" }, { "type": "string", "optional": true, "name": "io.debezium.data.Enum", "version": 1, "parameters": { "allowed": "true,last,false,incremental" }, "default": "false", "field": "snapshot" }, { "type": "string", "optional": false, "field": "db" }, { "type": "string", "optional": true, "field": "sequence" }, { "type": "string", "optional": false, "field": "schema" }, { "type": "string", "optional": false, "field": "table" }, { "type": "int64", "optional": true, "field": "txId" }, { "type": "int64", "optional": true, "field": "lsn" }, { "type": "int64", "optional": true, "field": "xmin" } ], "optional": false, "name": "io.debezium.connector.postgresql.Source", "field": "source" }, { "type": "string", "optional": false, "field": "op" }, { "type": "int64", "optional": true, "field": "ts_ms" }, { "type": "struct", "fields": [ { "type": "string", "optional": false, "field": "id" }, { "type": "int64", "optional": false, "field": "total_order" }, { "type": "int64", "optional": false, "field": "data_collection_order" } ], "optional": true, "name": "event.block", "version": 1, "field": "transaction" } ], "optional": false, "name": "test.public.customers.Envelope", "version": 1 }, "payload": { "before": null, "after": { "id": 1, "first_name": "a", "last_name": "a", "email": "a@a.com" }, "source": { "version": "2.5.3.Final", "connector": "postgresql", "name": "test", "ts_ms": 1714283798721, "snapshot": "false", "db": "postgres", "sequence": "[\"22910216\",\"22910504\"]", "schema": "public", "table": "customers", "txId": 756, "lsn": 22910504, "xmin": null }, "op": "c", "ts_ms": 1714283798790, "transaction": null } } ``` ###### Step 3: Configure StarRocks[​](#step-3-configure-starrocks "Direct link to Step 3: Configure StarRocks") Create a Primary Key table in StarRocks with the same schema as the source table in PostgreSQL. ```sql CREATE TABLE `customers` ( `id` int(11) COMMENT "", `first_name` varchar(65533) NULL COMMENT "", `last_name` varchar(65533) NULL COMMENT "", `email` varchar(65533) NULL COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`id`) DISTRIBUTED BY hash(id) buckets 1 PROPERTIES ( "bucket_size" = "4294967296", "in_memory" = "false", "enable_persistent_index" = "true", "replicated_storage" = "true", "fast_schema_evolution" = "true" ); ``` ###### Step 4: Install connector[​](#step-4-install-connector "Direct link to Step 4: Install connector") 1. Download the connectors and extract the packages in the **plugins** directory. ```bash mkdir plugins tar -zxvf debezium-debezium-connector-postgresql-2.5.3.zip -C plugins mv starrocks-connector-for-kafka-x.y.z-with-dependencies.jar plugins ``` This directory is the value of the configuration item `plugin.path` in **config/connect-standalone.properties**. ```properties plugin.path=/path/to/kafka_2.13-3.7.0/plugins ``` 2. Configure the PostgreSQL source connector in **pg-source.properties**. ```json { "name": "inventory-connector", "config": { "connector.class": "io.debezium.connector.postgresql.PostgresConnector", "plugin.name": "pgoutput", "database.hostname": "localhost", "database.port": "5432", "database.user": "postgres", "database.password": "", "database.dbname" : "postgres", "topic.prefix": "test" } } ``` 3. Configure the StarRocks sink connector in **sr-sink.properties**. ```json { "name": "starrocks-kafka-connector", "config": { "connector.class": "com.starrocks.connector.kafka.StarRocksSinkConnector", "tasks.max": "1", "topics": "test.public.customers", "starrocks.http.url": "172.26.195.69:28030", "starrocks.database.name": "test", "starrocks.username": "root", "starrocks.password": "StarRocks@123", "sink.properties.strip_outer_array": "true", "connect.timeoutms": "3000", "starrocks.topic2table.map": "test.public.customers:customers", "transforms": "addfield,unwrap", "transforms.addfield.type": "com.starrocks.connector.kafka.transforms.AddOpFieldForDebeziumRecord", "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState", "transforms.unwrap.drop.tombstones": "true", "transforms.unwrap.delete.handling.mode": "rewrite" } } ``` > **NOTE** > > * If the StarRocks table is not a Primary Key table, you do not need to specify the `addfield` transform. > * The unwrap transform is provided by Debezium and is used to unwrap Debezium's complex data structure based on the operation type. For more information, see [New Record State Extraction](https://debezium.io/documentation/reference/stable/transformations/event-flattening.html). 4. Configure Kafka Connect. Configure the following configuration items in the Kafka Connect configuration file **config/connect-standalone.properties**. ```properties # The addresses of Kafka brokers. Multiple addresses of Kafka brokers need to be separated by commas (,). # Note that this example uses PLAINTEXT as the security protocol to access the Kafka cluster. # If you use other security protocol to access the Kafka cluster, configure the relevant information in this part. bootstrap.servers=:9092 offset.storage.file.filename=/tmp/connect.offsets key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter key.converter.schemas.enable=true value.converter.schemas.enable=false # The absolute path of starrocks-connector-for-kafka-x.y.z-with-dependencies.jar. plugin.path=/home/kafka-connect/starrocks-kafka-connector # Parameters that control the flush policy. For more information, see the Usage Note section. offset.flush.interval.ms=10000 bufferflush.maxbytes = xxx bufferflush.intervalms = xxx ``` For descriptions of more parameters, see [Running Kafka Connect](https://kafka.apache.org/documentation.html#connect_running). ###### Step 5: Start Kafka Connect in Standalone Mode[​](#step-5-start-kafka-connect-in-standalone-mode "Direct link to Step 5: Start Kafka Connect in Standalone Mode") Run Kafka Connect in standalone mode to initiate the connectors. ```bash bin/connect-standalone.sh config/connect-standalone.properties config/pg-source.properties config/sr-sink.properties ``` ###### Step 6: Verify data ingestion[​](#step-6-verify-data-ingestion "Direct link to Step 6: Verify data ingestion") Test the following operations and ensure the data is correctly ingested into StarRocks. ###### INSERT[​](#insert "Direct link to INSERT") * In PostgreSQL: ```plain postgres=# insert into customers values (2,'b','b','b@b.com'); INSERT 0 1 postgres=# select * from customers; id | first_name | last_name | email ----+------------+-----------+--------- 1 | a | a | a@a.com 2 | b | b | b@b.com (2 rows) ``` * In StarRocks: ```plain MySQL [test]> select * from customers; +------+------------+-----------+---------+ | id | first_name | last_name | email | +------+------------+-----------+---------+ | 1 | a | a | a@a.com | | 2 | b | b | b@b.com | +------+------------+-----------+---------+ 2 rows in set (0.01 sec) ``` ###### UPDATE[​](#update "Direct link to UPDATE") * In PostgreSQL: ```plain postgres=# update customers set email='c@c.com'; UPDATE 2 postgres=# select * from customers; id | first_name | last_name | email ----+------------+-----------+--------- 1 | a | a | c@c.com 2 | b | b | c@c.com (2 rows) ``` * In StarRocks: ```plain MySQL [test]> select * from customers; +------+------------+-----------+---------+ | id | first_name | last_name | email | +------+------------+-----------+---------+ | 1 | a | a | c@c.com | | 2 | b | b | c@c.com | +------+------------+-----------+---------+ 2 rows in set (0.00 sec) ``` ###### DELETE[​](#delete "Direct link to DELETE") * In PostgreSQL: ```plain postgres=# delete from customers where id=1; DELETE 1 postgres=# select * from customers; id | first_name | last_name | email ----+------------+-----------+--------- 2 | b | b | c@c.com (1 row) ``` * In StarRocks: ```plain MySQL [test]> select * from customers; +------+------------+-----------+---------+ | id | first_name | last_name | email | +------+------------+-----------+---------+ | 2 | b | b | c@c.com | +------+------------+-----------+---------+ 1 row in set (0.00 sec) ``` --- ### Strict mode Strict mode is an optional property that you can configure for data loads. It affects the loading behavior and the final loaded data. This topic introduces what strict mode is and how to set strict mode. #### Understand strict mode[​](#understand-strict-mode "Direct link to Understand strict mode") During data loading, the data types of the source columns may not be completely consistent with the data types of the destination columns. In such cases, StarRocks performs conversions on the source column values that have inconsistent data types. Data conversions may fail due to various issues such as unmatched field data types and field length overflows. Source column values that fail to be properly converted are unqualified column values, and source rows that contain unqualified column values are referred to as "unqualified rows". Strict mode is used to control whether to filter out unqualified rows during data loading. Strict mode works as follows: * If strict mode is enabled, StarRocks loads only qualified rows. It filters out unqualified rows and returns details about the unqualified rows. * If strict mode is disabled, StarRocks converts unqualified column values into `NULL` and loads unqualified rows that contain these `NULL` values together with qualified rows. Note the following points: * In actual business scenarios, both qualified and unqualified rows may contain `NULL` values. If the destination columns do not allow `NULL` values, StarRocks reports errors and filters out the rows that contain `NULL` values. * The maximum percentage of unqualified rows that can be filtered out for a load job is controlled by an optional job property `max_filter_ratio`. note The `max_filter_ratio` property for INSERT is supported from v3.4.0. For example, you want to load four rows that hold `\N` (`\N` denotes a `NULL` value), `abc`, `2000`, and `1` values respectively in a column from a CSV-formatted data file into a StarRocks table, and the data type of the destination StarRocks table column is TINYINT \[-128, 127]. * The source column value `\N` is processed into `NULL` upon conversion to TINYINT. > **NOTE** > > `\N` is always processed into `NULL` upon conversion regardless of the destination data type. * The source column value `abc` is processed into `NULL`, because its data type is not TINYINT and the conversion fails. * The source column value `2000` is processed into `NULL`, because it is beyond the range supported by TINYINT and the conversion fails. * The source column value `1` can be properly converted to a TINYINT-type value `1`. If strict mode is disabled, StarRocks loads all the four rows. If strict mode is enabled, StarRocks loads only the rows that hold `\N` or `1` and filters out the rows that hold `abc` or `2000`. The rows filtered out are counted against the maximum percentage of rows that can be filtered out due to inadequate data quality as specified by the `max_filter_ratio` parameter. ##### Final loaded data with strict mode disabled[​](#final-loaded-data-with-strict-mode-disabled "Direct link to Final loaded data with strict mode disabled") | Source column value | Column value upon conversion to TINYINT | Load result when destination column allows NULL values | Load result when destination column does not allow NULL values | | ------------------- | --------------------------------------- | ------------------------------------------------------ | -------------------------------------------------------------- | | \N | NULL | The value `NULL` is loaded. | An error is reported. | | abc | NULL | The value `NULL` is loaded. | An error is reported. | | 2000 | NULL | The value `NULL` is loaded. | An error is reported. | | 1 | 1 | The value `1` is loaded. | The value `1` is loaded. | ##### Final loaded data with strict mode enabled[​](#final-loaded-data-with-strict-mode-enabled "Direct link to Final loaded data with strict mode enabled") | Source column value | Column value upon conversion to TINYINT | Load result when destination column allows NULL values | Load result when destination column does not allow NULL values | | ------------------- | --------------------------------------- | -------------------------------------------------------------- | -------------------------------------------------------------- | | \N | NULL | The value `NULL` is loaded. | An error is reported. | | abc | NULL | The value `NULL` is not allowed and therefore is filtered out. | An error is reported. | | 2000 | NULL | The value `NULL` is not allowed and therefore is filtered out. | An error is reported. | | 1 | 1 | The value `1` is loaded. | The value `1` is loaded. | #### Set strict mode[​](#set-strict-mode "Direct link to Set strict mode") You can use the `strict_mode` parameter to set strict mode for the load job. Valid values are `true` and `false`. The default value is `false`. The value `true` enables strict mode, and the value `false` disables strict mode. Note the `strict_mode` parameter is supported for INSERT from v3.4.0, with the default value `true`. Now, except Stream Load, for all other loading methods, `strict_mode` is set the same way in PROPERTIES clause. You can also use the `enable_insert_strict` session variable to set strict mode. Valid values are `true` and `false`. The default value is `true`. The value `true` enables strict mode, and the value `false` disables strict mode. note From v3.4.0 onwards, when `enable_insert_strict` is set to `true`, the system loads only qualified rows. It filters out unqualified rows and returns details about the unqualified rows. Instead, in versions earlier than v3.4.0, when `enable_insert_strict` is set to `true`, the INSERT jobs fails when there is an unqualified rows. Examples are as follows: ##### Stream Load[​](#stream-load "Direct link to Stream Load") ```bash curl --location-trusted -u : \ -H "strict_mode: {true | false}" \ -T -XPUT \ http://:/api///_stream_load ``` For detailed syntax and parameters about Stream Load, see [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). ##### Broker Load[​](#broker-load "Direct link to Broker Load") ```sql LOAD LABEL [.] ( DATA INFILE (""[, "" ...]) INTO TABLE ) WITH BROKER ( "username" = "", "password" = "" ) PROPERTIES ( "strict_mode" = "{true | false}" ) ``` The preceding code snippet uses HDFS as an example. For detailed syntax and parameters about Broker Load, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). ##### Routine Load[​](#routine-load "Direct link to Routine Load") ```sql CREATE ROUTINE LOAD [.] ON PROPERTIES ( "strict_mode" = "{true | false}" ) FROM KAFKA ( "kafka_broker_list" =":[,:...]", "kafka_topic" = "" ) ``` The preceding code snippet uses Apache Kafka® as an example. For detailed syntax and parameters about Routine Load, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). ##### Spark Load[​](#spark-load "Direct link to Spark Load") ```sql LOAD LABEL [.] ( DATA INFILE (""[, "" ...]) INTO TABLE ) WITH RESOURCE ( "spark.executor.memory" = "3g", "broker.username" = "", "broker.password" = "" ) PROPERTIES ( "strict_mode" = "{true | false}" ) ``` The preceding code snippet uses HDFS as an example. For detailed syntax and parameters about Spark Load, see [SPARK LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SPARK_LOAD.md). ##### INSERT[​](#insert "Direct link to INSERT") ```sql INSERT INTO [.] PROPERTIES( "strict_mode" = "{true | false}" ) ``` For detailed syntax and parameters about INSERT, see [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md). --- ### Continuously load data from Apache® Pulsar™ Experimental feature [Advice on use of experimental features](https://docs.starrocks.io/docs/introduction/maturity.md) As of StarRocks version 2.5, Routine Load supports continuously loading data from Apache® Pulsar™. Pulsar is distributed, open source pub-sub messaging and streaming platform with a store-compute separation architecture. Loading data from Pulsar via Routine Load is similar to loading data from Apache Kafka. This topic uses CSV-formatted data as an example to introduce how to load data from Apache Pulsar via Routine Load. #### Supported data file formats[​](#supported-data-file-formats "Direct link to Supported data file formats") Routine Load supports consuming CSV and JSON formatted data from a Pulsar cluster. > NOTE > > As for data in CSV format, StarRocks supports UTF-8 encoded strings within 50 bytes as column separators. Commonly used column separators include comma (,), tab and pipe (|). #### Pulsar-related concepts[​](#pulsar-related-concepts "Direct link to Pulsar-related concepts") **[Topic](https://pulsar.apache.org/docs/2.10.x/concepts-messaging/#topics)** Topics in Pulsar are named channels for transmitting messages from producers to consumers. Topics in Pulsar are divided into partitioned topics and non-partitioned topics. * **[Partitioned topics](https://pulsar.apache.org/docs/2.10.x/concepts-messaging/#partitioned-topics)** are a special type of topic that are handled by multiple brokers, thus allowing for higher throughput. A partitioned topic is actually implemented as N internal topics, where N is the number of partitions. * **Non-partitioned topics** are a normal type of topic that are served only by a single broker, which limits the maximum throughput of the topic. **[Message ID](https://pulsar.apache.org/docs/2.10.x/concepts-messaging/#messages)** The message ID of a message is assigned by [BookKeeper instances](https://pulsar.apache.org/docs/2.10.x/concepts-architecture-overview/#apache-bookkeeper) as soon as the message is persistently stored. Message ID indicates a message' s specific position in a ledger and is unique within a Pulsar cluster. Pulsar supports consumers specifying the initial position through consumer.*seek*(*messageId*). But compared to the Kafka consumer offset which is a long integer value, the message ID consists of four parts: `ledgerId:entryID:partition-index:batch-index`. Therefore, you can't get the Message ID directly from the message. As a result, at present, **Routine Load does not support specifying initial position when loading data from Pulsar, but only supports consuming data from the beginning or end of a partition.** **[Subscription](https://pulsar.apache.org/docs/2.10.x/concepts-messaging/#subscriptions)** A subscription is a named configuration rule that determines how messages are delivered to consumers. Pulsar also supports consumers simultaneously subscribing to multiple topics. A topic can have multiple subscriptions. The type of a subscription is defined when a consumer connects to it, and the type can be changed by restarting all consumers with a different configuration. Four subscription types are available in Pulsar: * `exclusive` (default)*:* Only a single consumer is allowed to attach to the subscription. Only one customer is allowed to consume messages. * `shared`: Multiple consumers can attach to the same subscription. Messages are delivered in a round robin distribution across consumers, and any given message is delivered to only one consumer. * `failover`: Multiple consumers can attach to the same subscription. A master consumer is picked for a non-partitioned topic or each partition of a partitioned topic and receives messages. When the master consumer disconnects, all (non-acknowledged and subsequent) messages are delivered to the next consumer in line. * `key_shared`: Multiple consumers can attach to the same subscription. Messages are delivered in a distribution across consumers and message with same key or same ordering key are delivered to only one consumer. > Note: > > Currently Routine Load uses exclusive type. #### Create a Routine Load job[​](#create-a-routine-load-job "Direct link to Create a Routine Load job") The following examples describe how to consume CSV-formatted messages in Pulsar, and load the data into StarRocks by creating a Routine Load job. For detailed instruction and reference, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). ```sql CREATE ROUTINE LOAD load_test.routine_wiki_edit_1 ON routine_wiki_edit COLUMNS TERMINATED BY ",", ROWS TERMINATED BY "\n", COLUMNS (order_id, pay_dt, customer_name, nationality, temp_gender, price) WHERE event_time > "2022-01-01 00:00:00", PROPERTIES ( "desired_concurrent_number" = "1", "max_batch_interval" = "15000", "max_error_number" = "1000" ) FROM PULSAR ( "pulsar_service_url" = "pulsar://localhost:6650", "pulsar_topic" = "persistent://tenant/namespace/topic-name", "pulsar_subscription" = "load-test", "pulsar_partitions" = "load-partition-0,load-partition-1", "pulsar_initial_positions" = "POSITION_EARLIEST,POSITION_LATEST", "property.auth.token" = "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUJzdWIiOiJqaXV0aWFuY2hlbiJ9.lulGngOC72vE70OW54zcbyw7XdKSOxET94WT_hIqD5Y" ); ``` When Routine Load is created to consume data from Pulsar, most input parameters except for `data_source_properties` are the same as consuming data from Kafka . For descriptions about parameters except data\_source\_properties `data_source_properties` , see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). The parameters related to `data_source_properties` and their descriptions are as follows: | **Parameter** | **Required** | **Description** | | ---------------------------------------------- | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | pulsar\_service\_url | Yes | The URL that is used to connect to the Pulsar cluster. Format: `"pulsar://ip:port"` or `"pulsar://service:port"`.Example: `"pulsar_service_url" = "pulsar://``localhost:6650``"` | | pulsar\_topic | Yes | Subscribed topic. Example: "pulsar\_topic" = "persistent://tenant/namespace/topic-name" | | pulsar\_subscription | Yes | Subscription configured for the topic.Example: `"pulsar_subscription" = "my_subscription"` | | pulsar\_partitions, pulsar\_initial\_positions | No | `pulsar_partitions` : Subscribed partitions in the topic.`pulsar_initial_positions`: initial positions of partitions specified by `pulsar_partitions`. The initial positions must correspond to the partitions in `pulsar_partitions`. Valid values:`POSITION_EARLIEST` (Default value): Subscription starts from the earliest available message in the partition. `POSITION_LATEST`: Subscription starts from the latest available message in the partition.Note:If `pulsar_partitions` is not specified, the topic's all partitions are subscribed.If both `pulsar_partitions` and `property.pulsar_default_initial_position` are specified, the `pulsar_partitions` value overrides `property.pulsar_default_initial_position` value.If neither `pulsar_partitions` nor `property.pulsar_default_initial_position` is specified, subscription starts from the latest available message in the partition.Example:`"pulsar_partitions" = "my-partition-0,my-partition-1,my-partition-2,my-partition-3", "pulsar_initial_positions" = "POSITION_EARLIEST,POSITION_EARLIEST,POSITION_LATEST,POSITION_LATEST"` | Routine Load supports the following custom parameters for Pulsar. | Parameter | Required | Description | | ------------------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | property.pulsar\_default\_initial\_position | No | The default initial positions when the topic's partitions are subscribed. The parameter takes effect when `pulsar_initial_positions` is not specified. Its valid values are the same as the valid values of `pulsar_initial_positions`.Example: `"``property.pulsar_default_initial_position" = "POSITION_EARLIEST"` | | property.auth.token | No | If Pulsar enables authenticating clients using security tokens, you need the token string to verify your identity.Example: `"p``roperty.auth.token" = "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUJzdWIiOiJqaXV0aWFuY2hlbiJ9.lulGngOC72vE70OW54zcbyw7XdKSOxET94WT_hIqD"` | #### Check a load job and task[​](#check-a-load-job-and-task "Direct link to Check a load job and task") ##### Check a load job[​](#check-a-load-job "Direct link to Check a load job") Execute the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement to check the status of the load job `routine_wiki_edit_1`. StarRocks returns the execution state `State`, the statistical information (including the total rows consumed and the total rows loaded) `Statistics`, and the progress of the load job `progress`. When you check a Routine Load job that consumes data from Pulsar, most returned parameters except for `progress` are the same as consuming data from Kafka. `progress` refers to backlog, that is the number of unacked messages in a partition. ```plaintext MySQL [load_test] > SHOW ROUTINE LOAD for routine_wiki_edit_1 \G *************************** 1. row *************************** Id: 10142 Name: routine_wiki_edit_1 CreateTime: 2022-06-29 14:52:55 PauseTime: 2022-06-29 17:33:53 EndTime: NULL DbName: default_cluster:test_pulsar TableName: test1 State: PAUSED DataSourceType: PULSAR CurrentTaskNum: 0 JobProperties: {"partitions":"*","rowDelimiter":"'\n'","partial_update":"false","columnToColumnExpr":"*","maxBatchIntervalS":"10","whereExpr":"*","timezone":"Asia/Shanghai","format":"csv","columnSeparator":"','","json_root":"","strict_mode":"false","jsonpaths":"","desireTaskConcurrentNum":"3","maxErrorNum":"10","strip_outer_array":"false","currentTaskConcurrentNum":"0","maxBatchRows":"200000"} DataSourceProperties: {"serviceUrl":"pulsar://localhost:6650","currentPulsarPartitions":"my-partition-0,my-partition-1","topic":"persistent://tenant/namespace/topic-name","subscription":"load-test"} CustomProperties: {"auth.token":"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUJzdWIiOiJqaXV0aWFuY2hlbiJ9.lulGngOC72vE70OW54zcbyw7XdKSOxET94WT_hIqD"} Statistic: {"receivedBytes":5480943882,"errorRows":0,"committedTaskNum":696,"loadedRows":66243440,"loadRowsRate":29000,"abortedTaskNum":0,"totalRows":66243440,"unselectedRows":0,"receivedBytesRate":2400000,"taskExecuteTimeMs":2283166} Progress: {"my-partition-0(backlog): 100","my-partition-1(backlog): 0"} ReasonOfStateChanged: ErrorLogUrls: OtherMsg: 1 row in set (0.00 sec) ``` ##### Check a load task[​](#check-a-load-task "Direct link to Check a load task") Execute the [SHOW ROUTINE LOAD TASK](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD_TASK.md) statement to check the load tasks of the load job `routine_wiki_edit_1`, such as how many tasks are running, the Kafka topic partitions that are consumed and the consumption progress `DataSourceProperties`, and the corresponding Coordinator BE node `BeId`. ```sql MySQL [example_db]> SHOW ROUTINE LOAD TASK WHERE JobName = "routine_wiki_edit_1" \G ``` #### Alter a load job[​](#alter-a-load-job "Direct link to Alter a load job") Before altering a load job, you must pause it by using the [PAUSE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/PAUSE_ROUTINE_LOAD.md) statement. Then you can execute the [ALTER ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/ALTER_ROUTINE_LOAD.md). After altering it, you can execute the [RESUME ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/RESUME_ROUTINE_LOAD.md) statement to resume it, and check its status by using the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement. When Routine Load is used to consume data from Pulsar, most returned parameters except for `data_source_properties` are the same as consuming data from Kafka. **Take note of the following points**: * Among the `data_source_properties` related parameters, only `pulsar_partitions`, `pulsar_initial_positions`, and custom Pulsar parameters `property.pulsar_default_initial_position` and `property.auth.token` are currently supported to be modified. The parameters `pulsar_service_url`, `pulsar_topic`, and `pulsar_subscription` cannot be modified. * If you need to modify the partition to be consumed and the matched initilal position, you need to make sure that you specify the partition using `pulsar_partitions` when you create the Routine Load job, and only the intial position `pulsar_initial_positions` of the specified partition can be modified. * If you specify only Topic `pulsar_topic` when creating a Routine Load job, but not partitions `pulsar_partitions`, you can modify the starting position of all partitions under topic via `pulsar_default_initial_position`. --- ### Change data through loading [Primary Key tables](https://docs.starrocks.io/docs/table_design/table_types/primary_key_table.md) provided by StarRocks allow you to make data changes to StarRocks tables by running [Stream Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md), [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md), or [Routine Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md) jobs. These data changes include inserts, updates, and deletions. However, Primary Key tables do not support changing data by using [Spark Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SPARK_LOAD.md) or [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md). StarRocks also supports partial updates and conditional updates. You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. This topic uses CSV data as an example to describe how to make data changes to a StarRocks table through loading. The data file formats that are supported vary depending on the loading method of your choice. > **NOTE** > > For CSV data, you can use a UTF-8 string, such as a comma (,), tab, or pipe (|), whose length does not exceed 50 bytes as a text delimiter. #### Implementation[​](#implementation "Direct link to Implementation") Primary Key tables provided by StarRocks support UPSERT and DELETE operations and does not distinguish INSERT operations from UPDATE operations. When you create a load job, StarRocks supports adding a field named `__op` to the job creation statement or command. The `__op` field is used to specify the type of operation you want to perform. > **NOTE** > > When you create a table, you do not need to add a column named `__op` to that table. The method of defining the `__op` field varies depending on the loading method of your choice: * If you choose Stream Load, define the `__op` field by using the `columns` parameter. * If you choose Broker Load, define the `__op` field by using the SET clause. * If you choose Routine Load, define the `__op` field by using the `COLUMNS` column. You can decide whether to add the `__op` field based on the data changes you want to make. If you do not add the `__op` field, the operation type defaults to UPSERT. The major data change scenarios are as follows: * If the data file you want to load involves only UPSERT operations, you do not need to add the `__op` field. * If the data file you want to load involves only DELETE operations, you must add the `__op` field and specify the operation type as DELETE. * If the data file you want to load involves both UPSERT and DELETE operations, you must add the `__op` field and make sure that the data file contains a column whose values are `0` or `1`. A value of `0` indicates an UPSERT operation, and a value of `1` indicates a DELETE operation. #### Usage notes[​](#usage-notes "Direct link to Usage notes") * Make sure that each row in your data file has the same number of columns. * The columns that involve data changes must include the primary key column. #### Basic operations[​](#basic-operations "Direct link to Basic operations") This section provides examples of how to make data changes to a StarRocks table through loading. For detailed syntax and parameter descriptions, see [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md), [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md), and [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). ##### UPSERT[​](#upsert "Direct link to UPSERT") If the data file you want to load involves only UPSERT operations, you do not need to add the `__op` field. > **NOTE** > > If you add the `__op` field: > > * You can specify the operation type as UPSERT. > > * You can leave the `__op` field empty, because the operation type defaults to UPSERT. ###### Data examples[​](#data-examples "Direct link to Data examples") 1. Prepare a data file. a. Create a CSV file named `example1.csv` in your local file system. The file consists of three columns, which represent user ID, user name, and user score in sequence. ```plain 101,Lily,100 102,Rose,100 ``` b. Publish the data of `example1.csv` to `topic1` of your Kafka cluster. 2. Prepare a StarRocks table. a. Create a Primary Key table named `table1` in your StarRocks database `test_db`. The table consists of three columns: `id`, `name`, and `score`, of which `id` is the primary key. ```sql CREATE TABLE `table1` ( `id` int(11) NOT NULL COMMENT "user ID", `name` varchar(65533) NOT NULL COMMENT "user name", `score` int(11) NOT NULL COMMENT "user score" ) ENGINE=OLAP PRIMARY KEY(`id`) DISTRIBUTED BY HASH(`id`); ``` > **NOTE** > > Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). b. Insert a record into `table1`. ```sql INSERT INTO table1 VALUES (101, 'Lily',80); ``` ###### Load data[​](#load-data "Direct link to Load data") Run a load job to update the record whose `id` is `101` in `example1.csv` to `table1` and insert the record whose `id` is `102` in `example1.csv` into `table1`. * Run a Stream Load job. * If you do not want to include the `__op` field, run the following command: ```bash curl --location-trusted -u : \ -H "Expect:100-continue" \ -H "label:label1" \ -H "column_separator:," \ -T example1.csv -XPUT \ http://:/api/test_db/table1/_stream_load ``` * If you want to include the `__op` field, run the following command: ```bash curl --location-trusted -u : \ -H "Expect:100-continue" \ -H "label:label2" \ -H "column_separator:," \ -H "columns:__op ='upsert'" \ -T example1.csv -XPUT \ http://:/api/test_db/table1/_stream_load ``` * Run a Broker Load job. * If you do not want to include the `__op` field, run the following command: ```sql LOAD LABEL test_db.label1 ( data infile("hdfs://:/example1.csv") into table table1 columns terminated by "," format as "csv" ) WITH BROKER; ``` * If you want to include the `__op` field, run the following command: ```sql LOAD LABEL test_db.label2 ( data infile("hdfs://:/example1.csv") into table table1 columns terminated by "," format as "csv" set (__op = 'upsert') ) WITH BROKER; ``` * Run a Routine Load job. * If you do not want to include the `__op` field, run the following command: ```sql CREATE ROUTINE LOAD test_db.table1 ON table1 COLUMNS TERMINATED BY ",", COLUMNS (id, name, score) PROPERTIES ( "desired_concurrent_number" = "3", "max_batch_interval" = "20", "max_batch_rows"= "250000", "max_error_number" = "1000" ) FROM KAFKA ( "kafka_broker_list" =":", "kafka_topic" = "test1", "property.kafka_default_offsets" ="OFFSET_BEGINNING" ); ``` * If you want to include the `__op` field, run the following command: ```sql CREATE ROUTINE LOAD test_db.table1 ON table1 COLUMNS TERMINATED BY ",", COLUMNS (id, name, score, __op ='upsert') PROPERTIES ( "desired_concurrent_number" = "3", "max_batch_interval" = "20", "max_batch_rows"= "250000", "max_error_number" = "1000" ) FROM KAFKA ( "kafka_broker_list" =":", "kafka_topic" = "test1", "property.kafka_default_offsets" ="OFFSET_BEGINNING" ); ``` ###### Query data[​](#query-data "Direct link to Query data") After the load is complete, query the data of `table1` to verify that the load is successful: ```sql SELECT * FROM table1; +------+------+-------+ | id | name | score | +------+------+-------+ | 101 | Lily | 100 | | 102 | Rose | 100 | +------+------+-------+ 2 rows in set (0.02 sec) ``` As shown in the preceding query result, the record whose `id` is `101` in `example1.csv` has been updated to `table1`, and the record whose `id` is `102` in `example1.csv` has been inserted into `table1`. ##### DELETE[​](#delete "Direct link to DELETE") If the data file you want to load involves only DELETE operations, you must add the `__op` field and specify the operation type as DELETE. ###### Data examples[​](#data-examples-1 "Direct link to Data examples") 1. Prepare a data file. a. Create a CSV file named `example2.csv` in your local file system. The file consists of three columns, which represent user ID, user name, and user score in sequence. ```plain 101,Jack,100 ``` b. Publish the data of `example2.csv` to `topic2` of your Kafka cluster. 2. Prepare a StarRocks table. a. Create a Primary Key table named `table2` in your StarRocks table `test_db`. The table consists of three columns: `id`, `name`, and `score`, of which `id` is the primary key. ```sql CREATE TABLE `table2` ( `id` int(11) NOT NULL COMMENT "user ID", `name` varchar(65533) NOT NULL COMMENT "user name", `score` int(11) NOT NULL COMMENT "user score" ) ENGINE=OLAP PRIMARY KEY(`id`) DISTRIBUTED BY HASH(`id`); ``` > **NOTE** > > Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). b. Insert two records into `table2`. ```sql INSERT INTO table2 VALUES (101, 'Jack', 100), (102, 'Bob', 90); ``` ###### Load data[​](#load-data-1 "Direct link to Load data") Run a load job to delete the record whose `id` is `101` in `example2.csv` from `table2`. * Run a Stream Load job. ```bash curl --location-trusted -u : \ -H "Expect:100-continue" \ -H "label:label3" \ -H "column_separator:," \ -H "columns:__op='delete'" \ -T example2.csv -XPUT \ http://:/api/test_db/table2/_stream_load ``` * Run a Broker Load job. ```sql LOAD LABEL test_db.label3 ( data infile("hdfs://:/example2.csv") into table table2 columns terminated by "," format as "csv" set (__op = 'delete') ) WITH BROKER; ``` * Run a Routine Load job. ```sql CREATE ROUTINE LOAD test_db.table2 ON table2 COLUMNS(id, name, score, __op = 'delete') PROPERTIES ( "desired_concurrent_number" = "3", "max_batch_interval" = "20", "max_batch_rows"= "250000", "max_error_number" = "1000" ) FROM KAFKA ( "kafka_broker_list" =":", "kafka_topic" = "test2", "property.kafka_default_offsets" ="OFFSET_BEGINNING" ); ``` ###### Query data[​](#query-data-1 "Direct link to Query data") After the load is complete, query the data of `table2` to verify that the load is successful: ```sql SELECT * FROM table2; +------+------+-------+ | id | name | score | +------+------+-------+ | 102 | Bob | 90 | +------+------+-------+ 1 row in set (0.00 sec) ``` As shown in the preceding query result, the record whose `id` is `101` in `example2.csv` has been deleted from `table2`. ##### UPSERT and DELETE[​](#upsert-and-delete "Direct link to UPSERT and DELETE") If the data file you want to load involves both UPSERT and DELETE operations, you must add the `__op` field and make sure that the data file contains a column whose values are `0` or `1`. A value of `0` indicates an UPSERT operation, and a value of `1` indicates a DELETE operation. ###### Data examples[​](#data-examples-2 "Direct link to Data examples") 1. Prepare a data file. a. Create a CSV file named `example3.csv` in your local file system. The file consists of four columns, which represent user ID, user name, user score, and operation type in sequence. ```plain 101,Tom,100,1 102,Sam,70,0 103,Stan,80,0 ``` b. Publish the data of `example3.csv` to `topic3` of your Kafka cluster. 2. Prepare a StarRocks table. a. Create a Primary Key table named `table3` in your StarRocks database `test_db`. The table consists of three columns: `id`, `name`, and `score`, of which `id` is the primary key. ```sql CREATE TABLE `table3` ( `id` int(11) NOT NULL COMMENT "user ID", `name` varchar(65533) NOT NULL COMMENT "user name", `score` int(11) NOT NULL COMMENT "user score" ) ENGINE=OLAP PRIMARY KEY(`id`) DISTRIBUTED BY HASH(`id`); ``` > **NOTE** > > Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). b. Insert two records into `table3`. ```sql INSERT INTO table3 VALUES (101, 'Tom', 100), (102, 'Sam', 90); ``` ###### Load data[​](#load-data-2 "Direct link to Load data") Run a load job to delete the record whose `id` is `101` in `example3.csv` from `table3`, update the record whose `id` is `102` in `example3.csv` to `table3`, and insert the record whose `id` is `103` in `example3.csv` into `table3`. * Run a Stream Load job: ```bash curl --location-trusted -u : \ -H "Expect:100-continue" \ -H "label:label4" \ -H "column_separator:," \ -H "columns: id, name, score, temp, __op = temp" \ -T example3.csv -XPUT \ http://:/api/test_db/table3/_stream_load ``` > **NOTE** > > In the preceding example, the fourth column that represents the operation type in `example3.csv` is temporarily named as `temp` and the `__op` field is mapped onto the `temp` column by using the `columns` parameter. As such, StarRocks can decide whether to perform an UPSERT or DELETE operation depending on the value in the fourth column of `example3.csv` is `0` or `1`. * Run a Broker Load job: ```bash LOAD LABEL test_db.label4 ( data infile("hdfs://:/example1.csv") into table table1 columns terminated by "," format as "csv" (id, name, score, temp) set (__op=temp) ) WITH BROKER; ``` * Run a Routine Load job: ```sql CREATE ROUTINE LOAD test_db.table3 ON table3 COLUMNS(id, name, score, temp, __op = temp) PROPERTIES ( "desired_concurrent_number" = "3", "max_batch_interval" = "20", "max_batch_rows"= "250000", "max_error_number" = "1000" ) FROM KAFKA ( "kafka_broker_list" = ":", "kafka_topic" = "test3", "property.kafka_default_offsets" = "OFFSET_BEGINNING" ); ``` ###### Query data[​](#query-data-2 "Direct link to Query data") After the load is complete, query the data of `table3` to verify that the load is successful: ```sql SELECT * FROM table3; +------+------+-------+ | id | name | score | +------+------+-------+ | 102 | Sam | 70 | | 103 | Stan | 80 | +------+------+-------+ 2 rows in set (0.01 sec) ``` As shown in the preceding query result, the record whose `id` is `101` in `example3.csv` has been deleted from `table3`, the record whose `id` is `102` in `example3.csv` has been updated to `table3`, and the record whose `id` is `103` in `example3.csv` has been inserted into `table3`. #### Partial updates[​](#partial-updates "Direct link to Partial updates") Primary Key tables also support partial updates, and provide two modes of partial updates, row mode and column mode, for different data update scenarios. These two modes of partial updates can minimize the overhead of partial updates as much as possible while guaranteeing query performance, ensuring real-time updates. Row mode is more suitable for real-time update scenarios involving many columns and small batches. Column mode is suitable for batch processing update scenarios involving a few columns and a large number of rows. > **NOTICE** > > When you perform a partial update, if the row to be updated does not exist, StarRocks inserts a new row, and fills default values in fields that are empty because no data updates are inserted into them. This section uses CSV as an example to describe how to perform partial updates. ##### Data examples[​](#data-examples-3 "Direct link to Data examples") 1. Prepare a data file. a. Create a CSV file named `example4.csv` in your local file system. The file consists of two columns, which represent user ID and user name in sequence. ```plain 101,Lily 102,Rose 103,Alice ``` b. Publish the data of `example4.csv` to `topic4` of your Kafka cluster. 2. Prepare a StarRocks table. a. Create a Primary Key table named `table4` in your StarRocks database `test_db`. The table consists of three columns: `id`, `name`, and `score`, of which `id` is the primary key. ```sql CREATE TABLE `table4` ( `id` int(11) NOT NULL COMMENT "user ID", `name` varchar(65533) NOT NULL COMMENT "user name", `score` int(11) NOT NULL COMMENT "user score" ) ENGINE=OLAP PRIMARY KEY(`id`) DISTRIBUTED BY HASH(`id`); ``` > **NOTE** > > Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). b. Insert a record into `table4`. ```sql INSERT INTO table4 VALUES (101, 'Tom',80); ``` ##### Load data[​](#load-data-3 "Direct link to Load data") Run a load to update the data in the two columns of `example4.csv` to the `id` and `name` columns of `table4`. * Run a Stream Load job: ```bash curl --location-trusted -u : \ -H "Expect:100-continue" \ -H "label:label7" -H "column_separator:," \ -H "partial_update:true" \ -H "columns:id,name" \ -T example4.csv -XPUT \ http://:/api/test_db/table4/_stream_load ``` > **NOTE** > > If you choose Stream Load, you must set the `partial_update` parameter to `true` to enable the partial update feature. The default is partial updates in row mode. If you need to perform partial updates in column mode, you need to set `partial_update_mode` to `column`. Additionally, you must use the `columns` parameter to specify the columns you want to update. * Run a Broker Load job: ```sql LOAD LABEL test_db.table4 ( data infile("hdfs://:/example4.csv") into table table4 format as "csv" (id, name) ) WITH BROKER PROPERTIES ( "partial_update" = "true" ); ``` > **NOTE** > > If you choose Broker Load, you must set the `partial_update` parameter to `true` to enable the partial update feature. The default is partial updates in row mode. If you need to perform partial updates in column mode, you need to set `partial_update_mode` to `column`. Additionally, you must use the `column_list` parameter to specify the columns you want to update. * Run a Routine Load job: ```sql CREATE ROUTINE LOAD test_db.table4 on table4 COLUMNS (id, name), COLUMNS TERMINATED BY ',' PROPERTIES ( "partial_update" = "true" ) FROM KAFKA ( "kafka_broker_list" =":", "kafka_topic" = "test4", "property.kafka_default_offsets" ="OFFSET_BEGINNING" ); ``` > **NOTE** > > * If you choose Routine Load, you must set the `partial_update` parameter to `true` to enable the partial update feature. Additionally, you must use the `COLUMNS` parameter to specify the columns you want to update. > * Routine Load only supports partial updates in row modes and does not support partial updates in column mode. ##### Query data[​](#query-data-3 "Direct link to Query data") After the load is complete, query the data of `table4` to verify that the load is successful: ```sql SELECT * FROM table4; +------+-------+-------+ | id | name | score | +------+-------+-------+ | 102 | Rose | 0 | | 101 | Lily | 80 | | 103 | Alice | 0 | +------+-------+-------+ 3 rows in set (0.01 sec) ``` As shown in the preceding query result, the record whose `id` is `101` in `example4.csv` has been updated to `table4`, and the records whose `id` are `102` and `103` in `example4.csv` have been Inserted into `table4`. #### Conditional updates[​](#conditional-updates "Direct link to Conditional updates") From StarRocks v2.5 onwards, Primary Key tables support conditional updates. You can specify a non-primary key column as the condition to determine whether updates can take effect. As such, the update from a source record to a destination record takes effect only when the source data record has a greater or equal value than the destination data record in the specified column. The conditional update feature is designed to resolve data disorder. If the source data is disordered, you can use this feature to ensure that new data will not be overwritten by old data. > **NOTICE** > > * You cannot specify different columns as update conditions for the same batch of data. > * DELETE operations do not support conditional updates. > * In versions earlier than v3.1.3, partial updates and conditional updates cannot be used simultaneously. From v3.1.3 onwards, StarRocks supports using partial updates with conditional updates. ##### Data examples[​](#data-examples-4 "Direct link to Data examples") 1. Prepare a data file. a. Create a CSV file named `example5.csv` in your local file system. The file consists of three columns, which represent user ID, version, and user score in sequence. ```plain 101,1,100 102,3,100 ``` b. Publish the data of `example5.csv` to `topic5` of your Kafka cluster. 2. Prepare a StarRocks table. a. Create a Primary Key table named `table5` in your StarRocks database `test_db`. The table consists of three columns: `id`, `version`, and `score`, of which `id` is the primary key. ```sql CREATE TABLE `table5` ( `id` int(11) NOT NULL COMMENT "user ID", `version` int NOT NULL COMMENT "version", `score` int(11) NOT NULL COMMENT "user score" ) ENGINE=OLAP PRIMARY KEY(`id`) DISTRIBUTED BY HASH(`id`); ``` > **NOTE** > > Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). b. Insert a record into `table5`. ```sql INSERT INTO table5 VALUES (101, 2, 80), (102, 2, 90); ``` ##### Load data[​](#load-data-4 "Direct link to Load data") Run a load to update the records whose `id` values are `101` and `102`, respectively, from `example5.csv` into `table5`, and specify that the updates take effect only when the `version` value in each of the two records is greater or equal to their current `version` values. * Run a Stream Load job: ```bash curl --location-trusted -u : \ -H "Expect:100-continue" \ -H "label:label10" \ -H "column_separator:," \ -H "merge_condition:version" \ -T example5.csv -XPUT \ http://:/api/test_db/table5/_stream_load ``` * Run a Insert Load job: ```sql INSERT INTO test_db.table5 properties("merge_condition" = "version") VALUES (101, 2, 70), (102, 3, 100); ``` * Run a Routine Load job: ```sql CREATE ROUTINE LOAD test_db.table5 on table5 COLUMNS (id, version, score), COLUMNS TERMINATED BY ',' PROPERTIES ( "merge_condition" = "version" ) FROM KAFKA ( "kafka_broker_list" =":", "kafka_topic" = "topic5", "property.kafka_default_offsets" ="OFFSET_BEGINNING" ); ``` * Run a Broker Load job: ```sql LOAD LABEL test_db.table5 ( DATA INFILE ("s3://xxx.csv") INTO TABLE table5 COLUMNS TERMINATED BY "," FORMAT AS "CSV" ) WITH BROKER PROPERTIES ( "merge_condition" = "version" ); ``` ##### Query data[​](#query-data-4 "Direct link to Query data") After the load is complete, query the data of `table5` to verify that the load is successful: ```sql SELECT * FROM table5; +------+------+-------+ | id | version | score | +------+------+-------+ | 101 | 2 | 80 | | 102 | 3 | 100 | +------+------+-------+ 2 rows in set (0.02 sec) ``` As shown in the preceding query result, the record whose `id` is `101` in `example5.csv` is not updated to `table5`, and the record whose `id` is `102` in `example5.csv` has been Inserted into `table5`. --- ### Loading options Data loading is the process of cleansing and transforming raw data from various data sources based on your business requirements and loading the resulting data into StarRocks to facilitate analysis. StarRocks provides a variety of options for data loading: * Loading methods: Insert, Stream Load, Broker Load, Pipe, Routine Load, and Spark Load * Ecosystem tools: StarRocks Connector for Apache Kafka® (Kafka connector for short), StarRocks Connector for Apache Spark™ (Spark connector for short), StarRocks Connector for Apache Flink® (Flink connector for short), and other tools such as SMT, DataX, CloudCanal, and Kettle Connector * API: Stream Load transaction interface These options each have its own advantages and support its own set of data source systems to pull from. This topic provides an overview of these options, along with comparisons between them to help you determine the loading option of your choice based on your data source, business scenario, data volume, data file format, and loading frequency. #### Introduction to loading options[​](#introduction-to-loading-options "Direct link to Introduction to loading options") This section mainly describes the characteristics and business scenarios of the loading options available in StarRocks. ![Loading options overview](/assets/images/loading_intro_overview-01dc28ae3c1adabb8349da9fbfa0140f.png) note In the following sections, "batch" or "batch loading" refers to the loading of a large amount of data from a specified source all at a time into StarRocks, whereas "stream" or "streaming" refers to the continuous loading of data in real time. #### Loading methods[​](#loading-methods "Direct link to Loading methods") ##### [Insert](https://docs.starrocks.io/docs/loading/InsertInto.md)[​](#insert "Direct link to insert") **Business scenario:** * INSERT INTO VALUES: Append to an internal table with small amounts of data. * INSERT INTO SELECT: * INSERT INTO SELECT FROM ``: Append to a table with the result of a query on an internal or external table. * INSERT INTO SELECT FROM FILES(): Append to a table with the result of a query on data files in remote storage. note For AWS S3, this feature is supported from v3.1 onwards. For HDFS, Microsoft Azure Storage, Google GCS, and S3-compatible storage (such as MinIO), this feature is supported from v3.2 onwards. **File format:** * INSERT INTO VALUES: SQL * INSERT INTO SELECT: * INSERT INTO SELECT FROM ``: StarRocks tables * INSERT INTO SELECT FROM FILES(): Parquet and ORC **Data volume:** Not fixed (The data volume varies based on the memory size.) ##### [Stream Load](https://docs.starrocks.io/docs/loading/StreamLoad.md)[​](#stream-load "Direct link to stream-load") **Business scenario:** Batch load data from a local file system. **File format:** CSV and JSON **Data volume:** 10 GB or less ##### [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md)[​](#broker-load "Direct link to broker-load") **Business scenario:** * Batch load data from HDFS or cloud storage like AWS S3, Microsoft Azure Storage, Google GCS, and S3-compatible storage (such as MinIO). * Batch load data from a local file system or NAS. **File format:** CSV, Parquet, ORC, and JSON (supported since v3.2.3) **Data volume:** Dozens of GB to hundreds of GB ##### [Pipe](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/CREATE_PIPE.md)[​](#pipe "Direct link to pipe") **Business scenario:** Batch load or stream data from HDFS or AWS S3. note This loading method is supported from v3.2 onwards. **File format:** Parquet and ORC **Data volume:** 100 GB to 1 TB or more ##### [Routine Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md)[​](#routine-load "Direct link to routine-load") **Business scenario:** Stream data from Kafka. **File format:** CSV, JSON, and Avro (supported since v3.0.1) **Data volume:** MBs to GBs of data as mini-batches ##### [Spark Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SPARK_LOAD.md)[​](#spark-load "Direct link to spark-load") **Business scenario:** Batch load data of Apache Hive™ tables stored in HDFS by using Spark clusters. **File format:** CSV, Parquet (supported since v2.0), and ORC (supported since v2.0) **Data volume:** Dozens of GB to TBs #### Ecosystem tools[​](#ecosystem-tools "Direct link to Ecosystem tools") ##### [Kafka connector](https://docs.starrocks.io/docs/loading/Kafka-connector-starrocks.md)[​](#kafka-connector "Direct link to kafka-connector") **Business scenario:** Stream data from Kafka. ##### [Spark connector](https://docs.starrocks.io/docs/loading/Spark-connector-starrocks.md)[​](#spark-connector "Direct link to spark-connector") **Business scenario:** Batch load data from Spark. ##### [Flink connector](https://docs.starrocks.io/docs/loading/Flink-connector-starrocks.md)[​](#flink-connector "Direct link to flink-connector") **Business scenario:** Stream data from Flink. ##### [SMT](https://docs.starrocks.io/docs/integrations/loading_tools/SMT.md)[​](#smt "Direct link to smt") **Business scenario:** Load data from data sources such as MySQL, PostgreSQL, SQL Server, Oracle, Hive, ClickHouse, and TiDB through Flink. ##### [DataX](https://docs.starrocks.io/docs/integrations/loading_tools/DataX-starrocks-writer.md)[​](#datax "Direct link to datax") **Business scenario:** Synchronize data between various heterogeneous data sources, including relational databases (for example, MySQL and Oracle), HDFS, and Hive. ##### [CloudCanal](https://docs.starrocks.io/docs/integrations/loading_tools/CloudCanal.md)[​](#cloudcanal "Direct link to cloudcanal") **Business scenario:** Migrate or synchronize data from source databases (for example, MySQL, Oracle, and PostgreSQL) to StarRocks. ##### [Kettle Connector](https://github.com/StarRocks/starrocks-connector-for-kettle)[​](#kettle-connector "Direct link to kettle-connector") **Business scenario:** Integrate with Kettle. By combining Kettle's robust data processing and transformation capabilities with StarRocks's high-performance data storage and analytical abilities, more flexible and efficient data processing workflows can be achieved. #### API[​](#api "Direct link to API") ##### [Stream Load transaction interface](https://docs.starrocks.io/docs/loading/Stream_Load_transaction_interface.md)[​](#stream-load-transaction-interface "Direct link to stream-load-transaction-interface") **Business scenario:** Implement two-phase commit (2PC) for transactions that are run to load data from external systems such as Flink and Kafka, while improving the performance of highly concurrent stream loads. This feature is supported from v2.4 onwards. **File format:** CSV and JSON **Data volume:** 10 GB or less #### Choice of loading options[​](#choice-of-loading-options "Direct link to Choice of loading options") This section lists the loading options available for common data sources, helping you choose the option that best suits your situation. ##### Object storage[​](#object-storage "Direct link to Object storage") | **Data source** | **Available loading options** | | ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | AWS S3 | - (Batch) INSERT INTO SELECT FROM FILES() (supported since v3.1)
- (Batch) Broker Load
- (Batch or streaming) Pipe (supported since v3.2)See [Load data from AWS S3](https://docs.starrocks.io/docs/loading/s3.md). | | Microsoft Azure Storage | - (Batch) INSERT INTO SELECT FROM FILES() (supported since v3.2)
- (Batch) Broker LoadSee [Load data from Microsoft Azure Storage](https://docs.starrocks.io/docs/loading/azure.md). | | Google GCS | - (Batch) INSERT INTO SELECT FROM FILES() (supported since v3.2)
- (Batch) Broker LoadSee [Load data from GCS](https://docs.starrocks.io/docs/loading/gcs.md). | | S3-compatible storage (such as MinIO) | - (Batch) INSERT INTO SELECT FROM FILES() (supported since v3.2)
- (Batch) Broker LoadSee [Load data from MinIO](https://docs.starrocks.io/docs/loading/minio.md). | ##### Local file system (including NAS)[​](#local-file-system-including-nas "Direct link to Local file system (including NAS)") | **Data source** | **Available loading options** | | --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | Local file system (including NAS) | - (Batch) Stream Load
- (Batch) Broker LoadSee [Load data from a local file system](https://docs.starrocks.io/docs/loading/StreamLoad.md). | ##### HDFS[​](#hdfs "Direct link to HDFS") | **Data source** | **Available loading options** | | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | HDFS | - (Batch) INSERT INTO SELECT FROM FILES() (supported since v3.2)
- (Batch) Broker Load
- (Batch or streaming) Pipe (supported since v3.2)See [Load data from HDFS](https://docs.starrocks.io/docs/loading/hdfs_load.md). | ##### Flink, Kafka, and Spark[​](#flink-kafka-and-spark "Direct link to Flink, Kafka, and Spark") | **Data source** | **Available loading options** | | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Apache Flink® | - [Flink connector](https://docs.starrocks.io/docs/loading/Flink-connector-starrocks.md)
- [Stream Load transaction interface](https://docs.starrocks.io/docs/loading/Stream_Load_transaction_interface.md) | | Apache Kafka® | - (Streaming) [Kafka connector](https://docs.starrocks.io/docs/loading/Kafka-connector-starrocks.md)
- (Streaming) [Routine Load](https://docs.starrocks.io/docs/loading/RoutineLoad.md)
- [Stream Load transaction interface](https://docs.starrocks.io/docs/loading/Stream_Load_transaction_interface.md) **NOTE**
If the source data requires multi-table joins and extract, transform and load (ETL) operations, you can use Flink to read and pre-process the data and then use [Flink connector](https://docs.starrocks.io/docs/loading/Flink-connector-starrocks.md) to load the data into StarRocks. | | Apache Spark™ | - [Spark connector](https://docs.starrocks.io/docs/loading/Spark-connector-starrocks.md)
- [Spark Load](https://docs.starrocks.io/docs/loading/SparkLoad.md) | ##### Data lakes[​](#data-lakes "Direct link to Data lakes") | **Data source** | **Available loading options** | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Apache Hive™ | - (Batch) Create a [Hive catalog](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog.md) and then use [INSERT INTO SELECT FROM ``](https://docs.starrocks.io/docs/loading/InsertInto.md#insert-data-from-an-internal-or-external-table-into-an-internal-table).
- (Batch) [Spark Load](https://docs.starrocks.io/docs/loading/SparkLoad/). | | Apache Iceberg | (Batch) Create an [Iceberg catalog](https://docs.starrocks.io/docs/data_source/catalog/iceberg/iceberg_catalog.md) and then use [INSERT INTO SELECT FROM ``](https://docs.starrocks.io/docs/loading/InsertInto.md#insert-data-from-an-internal-or-external-table-into-an-internal-table). | | Apache Hudi | (Batch) Create a [Hudi catalog](https://docs.starrocks.io/docs/data_source/catalog/hudi_catalog.md) and then use [INSERT INTO SELECT FROM ``](https://docs.starrocks.io/docs/loading/InsertInto.md#insert-data-from-an-internal-or-external-table-into-an-internal-table). | | Delta Lake | (Batch) Create a [Delta Lake catalog](https://docs.starrocks.io/docs/data_source/catalog/deltalake_catalog.md) and then use [INSERT INTO SELECT FROM ``](https://docs.starrocks.io/docs/loading/InsertInto.md#insert-data-from-an-internal-or-external-table-into-an-internal-table). | | Elasticsearch | (Batch) Create an [Elasticsearch catalog](https://docs.starrocks.io/docs/data_source/catalog/elasticsearch_catalog.md) and then use [INSERT INTO SELECT FROM ``](https://docs.starrocks.io/docs/loading/InsertInto.md#insert-data-from-an-internal-or-external-table-into-an-internal-table). | | Apache Paimon | (Batch) Create a [Paimon catalog](https://docs.starrocks.io/docs/data_source/catalog/paimon_catalog.md) and then use [INSERT INTO SELECT FROM ``](https://docs.starrocks.io/docs/loading/InsertInto.md#insert-data-from-an-internal-or-external-table-into-an-internal-table). | Note that StarRocks provides [unified catalogs](https://docs.starrocks.io/docs/data_source/catalog/unified_catalog/) from v3.2 onwards to help you handle tables from Hive, Iceberg, Hudi, and Delta Lake data sources as a unified data source without ingestion. ##### Internal and external databases[​](#internal-and-external-databases "Direct link to Internal and external databases") | **Data source** | **Available loading options** | | ---------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | StarRocks | (Batch) Create a [StarRocks external table](https://docs.starrocks.io/docs/data_source/External_table.md#starrocks-external-table) and then use [INSERT INTO VALUES](https://docs.starrocks.io/docs/loading/InsertInto.md#insert-data-via-insert-into-values) to insert a few data records or [INSERT INTO SELECT FROM ``](https://docs.starrocks.io/docs/loading/InsertInto.md#insert-data-from-an-internal-or-external-table-into-an-internal-table) to insert the data of a table.
**NOTE**
StarRocks external tables only support data writes. They do not support data reads. | | MySQL | - (Batch) Create a [JDBC catalog](https://docs.starrocks.io/docs/data_source/catalog/jdbc_catalog.md) (recommended) or a [MySQL external table](https://docs.starrocks.io/docs/data_source/External_table.md#deprecated-mysql-external-table) and then use [INSERT INTO SELECT FROM ``](https://docs.starrocks.io/docs/loading/InsertInto.md#insert-data-from-an-internal-or-external-table-into-an-internal-table).
- (Streaming) Use [SMT, Flink CDC connector, Flink, and Flink connector](https://docs.starrocks.io/docs/loading/Flink_cdc_load.md). | | Other databases such as Oracle, PostgreSQL, SQL Server, ClickHouse, and TiDB | - (Batch) Create a [JDBC catalog](https://docs.starrocks.io/docs/data_source/catalog/jdbc_catalog.md) (recommended) or a [JDBC external table](https://docs.starrocks.io/docs/data_source/External_table.md#external-table-for-a-jdbc-compatible-database) and then use [INSERT INTO SELECT FROM ``](https://docs.starrocks.io/docs/loading/InsertInto.md#insert-data-from-an-internal-or-external-table-into-an-internal-table).
- (Streaming) Use [SMT, Flink CDC connector, Flink, and Flink connector](https://docs.starrocks.io/docs/loading/loading_tools.md). | --- ### Feature Support: Data Loading and Unloading This document outlines the features of various data loading and unloading methods supported by StarRocks. #### File format[​](#file-format "Direct link to File format") ##### Loading file formats[​](#loading-file-formats "Direct link to Loading file formats") | | Data Source | File Format | | | | | | | | -------------------- | -------------------------------------------- | ----------- | --------------- | --------------- | --------------- | ---------------- | --------------- | --------------- | | | | CSV | JSON \[3] | Parquet | ORC | Avro | ProtoBuf | Thrift | | Stream Load | Local file systems, applications, connectors | Yes | Yes | To be supported | To be supported | To be supported | | | | INSERT from FILES | HDFS, S3, OSS, Azure, GCS, NFS(NAS) \[5] | Yes (v3.3+) | To be supported | Yes (v3.1+) | Yes (v3.1+) | Yes (v3.4.4+) | To be supported | | | Broker Load | | Yes | Yes (v3.2.3+) | Yes | Yes | To be supported | | | | Routine Load | Kafka | Yes | Yes | To be supported | To be supported | Yes (v3.0+) \[1] | To be supported | To be supported | | Spark Load | | Yes | To be supported | Yes | Yes | To be supported | | | | Connectors | Flink, Spark | Yes | Yes | To be supported | To be supported | To be supported | | | | Kafka Connector \[2] | Kafka | Yes (v3.0+) | | To be supported | To be supported | Yes (v3.0+) | | To be supported | | PIPE \[4] | Consistent with INSERT from FILES | | | | | | | | note \[1], \[2]: Schema Registry is required. \[3]: JSON supports a variety of CDC formats. For details about the JSON CDC formats supported by StarRocks, see [JSON CDC format](#json-cdc-formats). \[4]: Currently, only INSERT from FILES is supported for loading with PIPE. \[5]: You need to mount a NAS device as NFS under the same directory of each BE or CN node to access the files in NFS via the `file://` protocol. ###### JSON CDC formats[​](#json-cdc-formats "Direct link to JSON CDC formats") | | Stream Load | Routine Load | Broker Load | INSERT from FILES | Kafka Connector \[1] | | -------- | --------------- | --------------- | --------------- | ----------------- | -------------------- | | Debezium | To be supported | To be supported | To be supported | To be supported | Yes (v3.0+) | | Canal | To be supported | | | | | | Maxwell | | | | | | note \[1]: You must configure the `transforms` parameter while loading Debezium CDC format data into Primary Key tables in StarRocks. ##### Unloading file formats[​](#unloading-file-formats "Direct link to Unloading file formats") | | Target | | File format | | | | | ------------------- | -------------------- | ---------------------------------------- | --------------- | --------------- | --------------- | --------------- | | | Table format | Remote storage | CSV | JSON | Parquet | ORC | | INSERT INTO FILES | N/A | HDFS, S3, OSS, Azure, GCS, NFS(NAS) \[3] | Yes (v3.3+) | To be supported | Yes (v3.2+) | Yes (v3.3+) | | INSERT INTO Catalog | Hive | HDFS, S3, OSS, Azure, GCS | Yes (v3.3+) | To be supported | Yes (v3.2+) | Yes (v3.3+) | | | Iceberg | HDFS, S3, OSS, Azure, GCS | To be supported | To be supported | Yes (v3.2+) | To be supported | | | Hudi/Delta | | To be supported | | | | | EXPORT | N/A | HDFS, S3, OSS, Azure, GCS | Yes \[1] | To be supported | To be supported | To be supported | | PIPE | To be supported \[2] | | | | | | note \[1]: Configuring Broker process is supported. \[2]: Currently, unloading data using PIPE is not supported. \[3]: You need to mount a NAS device as NFS under the same directory of each BE or CN node to access the files in NFS via the `file://` protocol. #### File format-related parameters[​](#file-format-related-parameters "Direct link to File format-related parameters") ##### Loading file format-related parameters[​](#loading-file-format-related-parameters "Direct link to Loading file format-related parameters") | File format | Parameter | Loading method | | | | | | ----------- | ------------------- | -------------- | ----------------- | ---------------- | ---------------- | --------------- | | | | Stream Load | INSERT from FILES | Broker Load | Routine Load | Spark Load | | CSV | column\_separator | Yes | Yes (v3.3+) | Yes \[1] | | | | | row\_delimiter | Yes | | Yes \[2] (v3.1+) | Yes \[3] (v2.2+) | To be supported | | | enclose | Yes (v3.0+) | | Yes (v3.0+) | Yes (v3.0+) | To be supported | | | escape | | | | | | | | skip\_header | | To be supported | | | | | | trim\_space | | Yes (v3.0+) | | | | | JSON | jsonpaths | Yes | To be supported | Yes (v3.2.3+) | Yes | To be supported | | | strip\_outer\_array | | | | | | | | json\_root | | | | | | | | ignore\_json\_size | | | | To be supported | | note \[1]: The corresponding parameter is `COLUMNS TERMINATED BY`. \[2]: The corresponding parameter is `ROWS TERMINATED BY`. \[3]: The corresponding parameter is `ROWS TERMINATED BY`. ##### Unloading file format-related parameters[​](#unloading-file-format-related-parameters "Direct link to Unloading file format-related parameters") | File format | Parameter | Unloading method | | | ----------- | -------------------- | ----------------- | ------ | | | | INSERT INTO FILES | EXPORT | | CSV | column\_separator | Yes (v3.3+) | Yes | | | line\_delimiter \[1] | | | | | enclose | Yes | No | | | escape | Yes | No | | | include\_header | Yes | No | note \[1]: The corresponding parameter in data loading is `row_delimiter`. #### Compression formats[​](#compression-formats "Direct link to Compression formats") ##### Loading compression formats[​](#loading-compression-formats "Direct link to Loading compression formats") | File format | Compression format | Loading method | | | | | | ----------- | ---------------------------------------------------------------- | ------------------ | --------------- | ----------------- | --------------- | --------------- | | | | Stream Load | Broker Load | INSERT from FILES | Routine Load | Spark Load | | CSV | - deflate
- bzip2
- gzip
- lz4\_frame
- zstd | Yes \[1] | Yes \[2] | To be supported | To be supported | To be supported | | JSON | | Yes (v3.2.7+) \[3] | To be supported | N/A | To be supported | N/A | | Parquet | - gzip
- lz4
- snappy
- zlib
- zstd | N/A | Yes \[4] | | To be supported | Yes \[4] | | ORC | | | | | | | note \[1]: Currently, only when loading CSV files with Stream Load can you specify the compression format by using `format=gzip`, indicating gzip-compressed CSV files. `deflate` and `bzip2` formats are also supported. \[2]: Broker Load does not support specifying the compression format of CSV files by using the parameter `format`. Broker Load identifies the compression format by using the suffix of the file. The suffix of gzip-compressed files is `.gz`, and that of the zstd-compressed files is `.zst`. Besides, other `format`-related parameters, such as `trim_space` and `enclose`, are also not supported. \[3]: Supports specifying the compression format by using `compression = gzip`. \[4]: Supported by Arrow Library. You do not need to configure the `compression` parameter. ##### Unloading compression formats[​](#unloading-compression-formats "Direct link to Unloading compression formats") | File format | Compression format | Unloading method | | | | | | ----------- | ---------------------------------------------------------------- | ----------------- | ------------------- | --------------- | --------------- | --------------- | | | | INSERT INTO FILES | INSERT INTO Catalog | | | EXPORT | | | | | Hive | Iceberg | Hudi/Delta | | | CSV | - deflate
- bzip2
- gzip
- lz4\_frame
- zstd | To be supported | To be supported | To be supported | To be supported | To be supported | | JSON | N/A | N/A | N/A | N/A | N/A | N/A | | Parquet | - gzip
- lz4
- snappy
- zstd | Yes (v3.2+) | Yes (v3.2+) | Yes (v3.2+) | To be supported | N/A | | ORC | | | | | | | #### Credentials[​](#credentials "Direct link to Credentials") ##### Loading - Authentication[​](#loading---authentication "Direct link to Loading - Authentication") | Authentication | Loading method | | | | | | -------------------------------------------------------- | -------------- | ------------------------------------------- | ------------------------------------- | ------------------ | ---------------------- | | | Stream Load | INSERT from FILES | Broker Load | Routine Load | External Catalog | | Single Kerberos | N/A | Yes (v3.1+) | Yes \[1] (versions earlier than v2.5) | Yes \[2] (v3.1.4+) | Yes | | Kerberos Ticket Granting Ticket (TGT) | N/A | To be supported | | | Yes (v3.1.10+/v3.2.1+) | | Single KDC Multiple Kerberos | N/A | | | | | | Basic access authentications (Access Key pair, IAM Role) | N/A | Yes (HDFS and S3-compatible object storage) | | Yes \[3] | Yes | note \[1]: For HDFS, StarRocks supports both simple authentication and Kerberos authentication. \[2]: When the security protocol is set to `sasl_plaintext` or `sasl_ssl`, both SASL and GSSAPI (Kerberos) authentications are supported. \[3]: When the security protocol is set to `sasl_plaintext` or `sasl_ssl`, both SASL and PLAIN authentications are supported. ##### Unloading - Authentication[​](#unloading---authentication "Direct link to Unloading - Authentication") | | INSERT INTO FILES | EXPORT | | --------------- | ----------------- | --------------- | | Single Kerberos | To be supported | To be supported | #### Loading - Other parameters and features[​](#loading---other-parameters-and-features "Direct link to Loading - Other parameters and features") | Parameter and feature | Loading method | | | | | | | | | -------------------------------------------- | -------------- | ----------------- | ------------------------- | ---------------------- | --------------- | --------------- | --------------- | - | | | Stream Load | INSERT from FILES | INSERT from SELECT/VALUES | Broker Load | PIPE | Routine Load | Spark Load | | | partial\_update | Yes (v3.0+) | Yes \[1] (v3.3+) | | Yes (v3.0+) | N/A | Yes (v3.0+) | To be supported | | | partial\_update\_mode | Yes (v3.1+) | To be supported | | Yes (v3.1+) | N/A | To be supported | To be supported | | | COLUMNS FROM PATH | N/A | Yes (v3.2+) | N/A | Yes | N/A | N/A | Yes | | | timezone or session variable time\_zone \[2] | Yes \[3] | Yes \[4] | Yes \[4] | Yes \[4] | To be supported | Yes \[4] | To be supported | | | Time accuracy - Microsecond | Yes | Yes | Yes | Yes (v3.1.11+/v3.2.6+) | To be supported | Yes | Yes | | note \[1]: From v3.3 onwards, StarRocks supports Partial Updates in Row mode for INSERT INTO by specifying the column list. \[2]: Setting the time zone by the parameter or the session variable will affect the results returned by functions such as strftime(), alignment\_timestamp(), and from\_unixtime(). \[3]: Only the parameter `timezone` is supported. \[4]: Only the session variable `time_zone` is supported. #### Unloading - Other parameters and features[​](#unloading---other-parameters-and-features "Direct link to Unloading - Other parameters and features") | Parameter and feature | INSERT INTO FILES | EXPORT | | --------------------------- | ----------------- | --------------- | | target\_max\_file\_size | Yes (v3.2+) | To be supported | | single | | | | Partitioned\_by | | | | Session variable time\_zone | To be supported | | | Time accuracy - Microsecond | To be supported | To be supported | --- ### Loading concepts This topic introduces common concepts and information about data loading. #### Privileges[​](#privileges "Direct link to Privileges") You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. #### Labeling[​](#labeling "Direct link to Labeling") You can load data into StarRocks by running load jobs. Each load job has a unique label that is specified by the user or automatically generated by StarRocks to identify the job. Each label can be used only for one load job. After a load job is complete, its label cannot be reused for any other load jobs. Only the labels of failed load jobs can be reused. #### Atomicity[​](#atomicity "Direct link to Atomicity") All the loading methods provided by StarRocks guarantee atomicity. Atomicity means that the qualified data within a load job must be all successfully loaded or none of the qualified data is successfully loaded. It never happens that some of the qualified data is loaded while the other data is not. Note that the qualified data does not include the data that is filtered out due to quality issues such as data type conversion errors. #### Protocols[​](#protocols "Direct link to Protocols") StarRocks supports two communication protocols that can be used to submit load jobs: MySQL and HTTP. Of all the loading methods supported by StarRocks, only Stream Load uses HTTP, whereas all the others use MySQL. #### Data types[​](#data-types "Direct link to Data types") StarRocks supports loading data of all data types. You only need to take note of the limits on the loading of a few specific data types. For more information, see [Data types](https://docs.starrocks.io/docs/sql-reference/data-types). #### Strict mode[​](#strict-mode "Direct link to Strict mode") Strict mode is an optional property that you can configure for data loads. It affects the loading behavior and the final loaded data. For details, see [Strict mode](https://docs.starrocks.io/docs/loading/load_concept/strict_mode.md). #### Loading modes[​](#loading-modes "Direct link to Loading modes") StarRocks supports two loading modes: synchronous loading mode and asynchronous loading mode. note If you load data by using external programs, you must choose a loading mode that best suits your business requirements before you decide the loading method of your choice. ##### Synchronous loading[​](#synchronous-loading "Direct link to Synchronous loading") In synchronous loading mode, after you submit a load job, StarRocks synchronously runs the job to load data, and returns the result of the job after the job finishes. You can check whether the job is successful based on the job result. StarRocks provides two loading methods that support synchronous loading: [Stream Load](https://docs.starrocks.io/docs/loading/StreamLoad.md) and [INSERT](https://docs.starrocks.io/docs/loading/InsertInto.md). The process of synchronous loading is as follows: 1. Create a load job. 2. View the job result returned by StarRocks. 3. Check whether the job is successful based on the job result. If the job result indicates a load failure, you can retry the job. ##### Asynchronous loading[​](#asynchronous-loading "Direct link to Asynchronous loading") In asynchronous loading mode, after you submit a load job, StarRocks immediately returns the job creation result. * If the result indicates a job creation success, StarRocks asynchronously runs the job. However, that does not mean that the data has been successfully loaded. You must use statements or commands to check the status of the job. Then, you can determine whether the data is successfully loaded based on the job status. * If the result indicates a job creation failure, you can determine whether you need to retry the job based on the failure information. tip You can set different write quorum for tables, that is, how many replicas are required to return loading success before StarRocks can determine the loading task is successful. You can specify write quorum by adding the property `write_quorum` when you [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md), or add this property to an existing table using [ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md). StarRocks provides four loading methods that support asynchronous loading: [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md), [Pipe](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/CREATE_PIPE.md), [Routine Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md), and [Spark Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SPARK_LOAD.md). The process of asynchronous loading is as follows: 1. Create a load job. 2. View the job creation result returned by StarRocks and determine whether the job is successfully created. * If the job creation succeeds, go to Step 3. * If the job creation fails, return to Step 1. 3. Use statements or commands to check the status of the job until the job status shows **FINISHED** or **CANCELLED**. ###### Workflow of Broker Load or Spark Load[​](#workflow-of-broker-load-or-spark-load "Direct link to Workflow of Broker Load or Spark Load") The workflow of a Broker Load or Spark Load job consists of five stages, as shown in the following figure. ![Broker Load or Spark Load overflow](/assets/images/4.1-1-5e37ffe979abb28c1b29aed2870659f7.png) The workflow is described as follows: 1. **PENDING** The job is in queue waiting to be scheduled by an FE. 2. **ETL** The FE pre-processes the data, including cleansing, partitioning, sorting, and aggregation. Only a Spark Load job has the ETL stage. A Broker Load job skips this stage. 3. **LOADING** The FE cleanses and transforms the data, and then sends the data to the BEs or CNs. After all data is loaded, the data is in queue waiting to take effect. At this time, the status of the job remains **LOADING**. 4. **FINISHED** When loading finishes and all data involved takes effect, the status of the job becomes **FINISHED**. At this time, the data can be queried. **FINISHED** is a final job state. 5. **CANCELLED** Before the status of the job becomes **FINISHED**, you can cancel the job at any time. Additionally, StarRocks can automatically cancel the job in case of load errors. After the job is canceled, the status of the job becomes **CANCELLED**, and all data updates made before the cancellation are reverted. **CANCELLED** is also a final job state. ###### Workflow of Pipe[​](#workflow-of-pipe "Direct link to Workflow of Pipe") The workflow of a Pipe job is described as follows: 1. The job is submitted to an FE from a MySQL client. 2. The FE splits the data files stored in the specified path based on their number or size, breaking down job into smaller, sequential tasks. The tasks enter a queue, waiting to be scheduled, after they are created. 3. The FE obtains the tasks from the queue, and invokes the INSERT INTO SELECT FROM FILES statement to execute each task. 4. The data loading finishes: * If `"AUTO_INGEST" = "FALSE"` is specified for the job at job creation, the job finishes after the data of all the data files stored in the specified path is loaded. * If `"AUTO_INGEST" = "TRUE"` is specified for the job at job creation, the FE will continue to monitor changes to the data files and automatically loads new or updated data from the data files into the destination StarRocks table. ###### Workflow of Routine Load[​](#workflow-of-routine-load "Direct link to Workflow of Routine Load") The workflow of a Routine Load job is described as follows: 1. The job is submitted to an FE from a MySQL client. 2. The FE splits the job into multiple tasks. Each task is engineered to load data from multiple partitions. 3. The FE distributes the tasks to specified BEs or CNs. 4. The BEs or CNs execute the tasks, and report to the FE after they finish the tasks. 5. The FE generates subsequent tasks, retries failed tasks if there are any, or suspends task scheduling based on the reports from the BEs. --- ### Considerations This topic describes some system limits and configurations that you need to consider before you run data loads. #### Memory limits[​](#memory-limits "Direct link to Memory limits") StarRocks provides parameters for you to limit the memory usage for each load job, thereby reducing memory consumption, especially in high concurrency scenarios. However, do not specify an excessively low memory usage limit. If the memory usage limit is excessively low, data may be frequently flushed from memory to disk because the memory usage for load jobs reaches the specified limit. We recommend that you specify a proper memory usage limit based on your business scenario. The parameters that are used to limit memory usage vary for each loading method. For more information, see [Stream Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md), [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md), [Routine Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md), [Spark Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SPARK_LOAD.md), and [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md). Note that a load job usually runs on multiple BEs or CNs. Therefore, the parameters limit the memory usage of each load job on each involved BE or CN rather than the total memory usage of the load job on all involved BEs or CNs. StarRocks also provides parameters for you to limit the total memory usage of all load jobs that run on each individual BE or CN. For more information, see the "[System configurations](#system-configurations)" section below. #### System configurations[​](#system-configurations "Direct link to System configurations") This section describes some parameter configurations that are applicable to all of the loading methods provided by StarRocks. ##### FE configurations[​](#fe-configurations "Direct link to FE configurations") You can configure the following parameters in the configuration file **fe.conf** of each FE: * `max_load_timeout_second` and `min_load_timeout_second` These parameters specify the maximum timeout period and minimum timeout period of each load job. The timeout periods are measured in seconds. The default maximum timeout period spans 3 days, and the default minimum timeout period spans 1 second. The maximum timeout period and minimum timeout period that you specify must fall within the range of 1 second to 3 days. These parameters are valid for both synchronous load jobs and asynchronous load jobs. * `desired_max_waiting_jobs` This parameter specifies the maximum number of load jobs that can be held waiting in queue. The default value is **1024** (100 in v2.4 and earlier, and 1024 in v2.5 and later). When the number of load jobs in the **PENDING** state on an FE reaches the maximum number that you specify, the FE rejects new load requests. This parameter is valid only for asynchronous load jobs. * `max_running_txn_num_per_db` This parameter specifies the maximum number of ongoing load transactions that are allowed in each database of your StarRocks cluster. A load job can contain one or more transactions. The default value is **100**. When the number of load transactions running in a database reaches the maximum number that you specify, the subsequent load jobs that you submit are not scheduled. In this situation, if you submit a synchronous load job, the job is rejected. If you submit an asynchronous load job, the job is held waiting in queue. note StarRocks counts all load jobs together and does not distinguish between synchronous load jobs and asynchronous load jobs. * `label_keep_max_second` This parameter specifies the retention period of the history records for load jobs that have finished and are in the **FINISHED** or **CANCELLED** state. The default retention period spans 3 days. This parameter is valid for both synchronous load jobs and asynchronous load jobs. ##### BE/CN configurations[​](#becn-configurations "Direct link to BE/CN configurations") You can configure the following parameters in the configuration file **be.conf** of each BE or the configuration file **cn.conf** of each CN: * `write_buffer_size` This parameter specifies the maximum memory block size. The default size is 100 MB. The loaded data is first written to a memory block on the BE or CN. When the amount of data that is loaded reaches the maximum memory block size that you specify, the data is flushed to disk. You must specify a proper maximum memory block size based on your business scenario. * If the maximum memory block size is exceedingly small, a large number of small files may be generated on the BE or CN. In this case, query performance degrades. You can increase the maximum memory block size to reduce the number of files generated. * If the maximum memory block size is exceedingly large, remote procedure calls (RPCs) may time out. In this case, you can adjust the value of this parameter based on your business needs. * `streaming_load_rpc_max_alive_time_sec` The waiting timeout period for each Writer process. The default value is 1200 seconds. During the data loading process, StarRocks starts a Writer process to receive data from and write data to each tablet. If a Writer process does not receive any data within the waiting timeout period that you specify, StarRocks stops the Writer process. When your StarRocks cluster processes data at low speeds, a Writer process may not receive the next batch of data within a long period of time and therefore reports a "TabletWriter add batch with unknown id" error. In this case, you can increase the value of this parameter. * `load_process_max_memory_limit_bytes` and `load_process_max_memory_limit_percent` These parameters specify the maximum amount of memory that can be consumed for all load jobs on each individual BE or CN. StarRocks identifies the smaller memory consumption among the values of the two parameters as the final memory consumption that is allowed. * `load_process_max_memory_limit_bytes`: specifies the maximum memory size. The default maximum memory size is 100 GB. * `load_process_max_memory_limit_percent`: specifies the maximum memory usage. The default value is 30%. This parameter differs from the `mem_limit` parameter. The `mem_limit` parameter specifies the total maximum memory usage of your StarRocks cluster, and the default value is 90% x 90%. If the memory capacity of the machine on which the BE or CN resides is M, the maximum amount of memory that can be consumed for load jobs is calculated as follows: `M x 90% x 90% x 30%`. ##### System variable configurations[​](#system-variable-configurations "Direct link to System variable configurations") You can configure the following [system variable](https://docs.starrocks.io/docs/sql-reference/System_variable.md): * `insert_timeout` The INSERT timeout duration. Unit: seconds. Value range: `1` to `259200`. Default value: `14400`. This variable will act on all operations involving INSERT jobs (for example, UPDATE, DELETE, CTAS, materialized view refresh, statistics collection, and PIPE) in the current connection. --- ### Troubleshooting Data Loading This guide is designed to help DBAs and operation engineers monitor the status of data load jobs through SQL interfaces—without relying on external monitoring systems. It also provides guidance on identifying performance bottlenecks and troubleshooting anomalies during load operations. #### Terminology[​](#terminology "Direct link to Terminology") **Load Job:** A continuous data load process, such as a **Routine Load Job** or **Pipe Job**. **Load Task:** A one-time data load process, usually corresponding to a single load transaction. Examples include **Broker Load**, **Stream Load**, **Spark Load**, and **INSERT INTO**. Routine Load jobs and Pipe jobs continuously generate tasks to perform data ingestion. #### Observe load jobs[​](#observe-load-jobs "Direct link to Observe load jobs") There are two ways to observe load jobs: * Using SQL statements **[SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md)** and **[SHOW PIPES](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/SHOW_PIPES.md)**. * Using system views **[information\_schema.routine\_load\_jobs](https://docs.starrocks.io/docs/sql-reference/information_schema/routine_load_jobs.md)** and **[information\_schema.pipes](https://docs.starrocks.io/docs/sql-reference/information_schema/pipes.md)**. #### Observe load tasks[​](#observe-load-tasks "Direct link to Observe load tasks") Load tasks can also be monitored in two ways: * Using SQL statements **[SHOW LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SHOW_LOAD.md)** and **[SHOW ROUTINE LOAD TASK](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD_TASK.md)**. * Using system views **[information\_schema.loads](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md)** and **statistics.loads\_history**. ##### SQL statements[​](#sql-statements "Direct link to SQL statements") The **SHOW** statements display both ongoing and recently completed load tasks for the current database, providing a quick overview of task status. The information retrieved is a subset of the **statistics.loads\_history** system view. SHOW LOAD statements return information of Broker Load, Insert Into, and Spark Load tasks, and SHOW ROUTINE LOAD TASK statements return Routine Load task information. ##### System views[​](#system-views "Direct link to System views") ###### information\_schema.loads[​](#information_schemaloads "Direct link to information_schema.loads") The **information\_schema.loads** system view stores information about recent load tasks, including active and recently completed ones. StarRocks periodically synchronizes the data to the **statistics.loads\_history** system table for persistent storage. **information\_schema.loads** provides the following fields: | Field | Description | | ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ID | Globally unique identifier. | | LABEL | Label of the load job. | | PROFILE\_ID | The ID of the Profile, which can be analyzed via `ANALYZE PROFILE`. | | DB\_NAME | The database to which the target table belongs. | | TABLE\_NAME | The target table. | | USER | The user who initiates the load job. | | WAREHOUSE | The warehouse to which the load job belongs. | | STATE | The state of the load job. Valid values:- `PENDING`/`BEGIN`: The load job is created.
- `QUEUEING`/`BEFORE_LOAD`: The load job is in the queue waiting to be scheduled.
- `LOADING`: The load job is running.
- `PREPARING`: The transaction is being pre-committed.
- `PREPARED`: The transaction has been pre-committed.
- `COMMITED`: The transaction has been committed.
- `FINISHED`: The load job succeeded.
- `CANCELLED`: The load job failed. | | PROGRESS | The progress of the ETL stage and LOADING stage of the load job. | | TYPE | The type of the load job. For Broker Load, the return value is `BROKER`. For INSERT, the return value is `INSERT`. For Stream Load, the return value is `STREAM`. For Routine Load Load, the return value is `ROUTINE`. | | PRIORITY | The priority of the load job. Valid values: `HIGHEST`, `HIGH`, `NORMAL`, `LOW`, and `LOWEST`. | | SCAN\_ROWS | The number of data rows that are scanned. | | SCAN\_BYTES | The number of bytes that are scanned. | | FILTERED\_ROWS | The number of data rows that are filtered out due to inadequate data quality. | | UNSELECTED\_ROWS | The number of data rows that are filtered out due to the conditions specified in the WHERE clause. | | SINK\_ROWS | The number of data rows that are loaded. | | RUNTIME\_DETAILS | Load runtime metadata. For details, see [RUNTIME\_DETAILS](#runtime_details). | | CREATE\_TIME | The time at which the load job was created. Format: `yyyy-MM-dd HH:mm:ss`. Example: `2023-07-24 14:58:58`. | | LOAD\_START\_TIME | The start time of the LOADING stage of the load job. Format: `yyyy-MM-dd HH:mm:ss`. Example: `2023-07-24 14:58:58`. | | LOAD\_COMMIT\_TIME | The time at which the loading transaction was committed. Format: `yyyy-MM-dd HH:mm:ss`. Example: `2023-07-24 14:58:58`. | | LOAD\_FINISH\_TIME | The end time of the LOADING stage of the load job. Format: `yyyy-MM-dd HH:mm:ss`. Example: `2023-07-24 14:58:58`. | | PROPERTIES | The static properties of the load job. For details, see [PROPERTIES](#properties). | | ERROR\_MSG | The error message of the load job. If the load job did not encounter any error, `NULL` is returned. | | TRACKING\_SQL | The SQL statement that can be used to query the tracking log of the load job. A SQL statement is returned only when the load job involves unqualified data rows. If the load job does not involve any unqualified data rows, `NULL` is returned. | | REJECTED\_RECORD\_PATH | The path from which you can access all the unqualified data rows that are filtered out in the load job. The number of unqualified data rows logged is determined by the `log_rejected_record_num` parameter configured in the load job. You can use the `wget` command to access the path. If the load job does not involve any unqualified data rows, `NULL` is returned. | ###### RUNTIME\_DETAILS[​](#runtime_details "Direct link to RUNTIME_DETAILS") * Universal metrics: | Metric | Description | | -------- | ---------------------------------------------- | | load\_id | Globally unique ID of the load execution plan. | | txn\_id | Load transaction ID. | * Specific metrics for Broker Load, INSERT INTO, and Spark Load: | Metric | Description | | -------------------- | ----------------------------------------------------------------------------------------------------------------- | | etl\_info | ETL Details. This field is only valid for Spark Load jobs. For other types of load jobs, the value will be empty. | | etl\_start\_time | The start time of the ETL stage of the load job. Format: `yyyy-MM-dd HH:mm:ss`. Example: `2023-07-24 14:58:58`. | | etl\_start\_time | The end time of the ETL stage of the load job. Format: `yyyy-MM-dd HH:mm:ss`. Example: `2023-07-24 14:58:58`. | | unfinished\_backends | List of BEs with unfinished executions. | | backends | List of BEs participating in execution. | | file\_num | Number of files read. | | file\_size | Total size of files read. | | task\_num | Number of subtasks. | * Specific metrics for Routine Load: | Metric | Description | | --------------------- | ----------------------------------------------------------------------------------- | | schedule\_interval | The interval for Routine Load to be scheduled. | | wait\_slot\_time | Time elapsed while the Routine Load task waits for execution slots. | | check\_offset\_time | Time consumed when checking offset information during Routine Load task scheduling. | | consume\_time | Time consumed by the Routine Load task to read upstream data. | | plan\_time | Time for generating the execution plan. | | commit\_publish\_time | Time consumed to execute the COMMIT RPC. | * Specific metrics for Stream Load: | Metric | Description | | ------------------------- | ---------------------------------------- | | timeout | Timeout for load tasks. | | begin\_txn\_ms | Time consumed to begin the transaction. | | plan\_time\_ms | Time for generating the execution plan. | | receive\_data\_time\_ms | Time for receiving data. | | commit\_publish\_time\_ms | Time consumed to execute the COMMIT RPC. | | client\_ip | Client IP address. | ###### PROPERTIES[​](#properties "Direct link to PROPERTIES") * Specific properties for Broker Load, INSERT INTO, and Spark Load: | Property | Description | | ------------------ | -------------------------------------------------------------------------------- | | timeout | Timeout for load tasks. | | max\_filter\_ratio | Maximum ratio of data rows that are filtered out due to inadequate data quality. | * Specific properties for Routine Load: | Property | Description | | --------- | ------------------------------------------------- | | job\_name | Routine Load job name. | | task\_num | Number of subtasks actually executed in parallel. | | timeout | Timeout for load tasks. | ###### statistics.loads\_history[​](#statisticsloads_history "Direct link to statistics.loads_history") The **statistics.loads\_history** system view stores load records for the last three months by default. DBAs can adjust the retention period by modifying the `partition_ttl` of the view. **statistics.loads\_history** has the consistent schema with **information\_schema.loads**. #### Identify loading performance issues with Load Profiles[​](#identify-loading-performance-issues-with-load-profiles "Direct link to Identify loading performance issues with Load Profiles") A **Load Profile** records execution details of all worker nodes involved in a data load. It helps you quickly pinpoint performance bottlenecks in the StarRocks cluster. ##### Enable Load Profiles[​](#enable-load-profiles "Direct link to Enable Load Profiles") StarRocks provides multiple methods to enable Load Profiles, depending on the type of load: ###### For Broker Load and INSERT INTO[​](#for-broker-load-and-insert-into "Direct link to For Broker Load and INSERT INTO") Enable Load Profiles for Broker Load and INSERT INTO at session level: ```sql SET enable_profile = true; ``` By default, profiles are automatically enabled for long-running jobs (longer than 300 seconds). You can customize this threshold by: ```sql SET big_query_profile_threshold = 60s; ``` note When `big_query_profile_threshold` is set to its default value `0`, the default behavior is to disable Query Profiling for queries. However, for load tasks, profiles are automatically recorded for tasks with execution times exceeding 300 seconds. StarRocks also supports **Runtime Profiles**, which periodically (every 30 seconds) report execution metrics of long-running load jobs. You can customize the report interval by: ```sql SET runtime_profile_report_interval = 60; ``` note `runtime_profile_report_interval` specifies only the minimum report interval for load tasks. The actual report interval is dynamically adjusted and may exceed this value. ###### For Stream Load and Routine Load[​](#for-stream-load-and-routine-load "Direct link to For Stream Load and Routine Load") Enable Load Profiles for Stream Load and Routine Load at table level: ```sql ALTER TABLE SET ("enable_load_profile" = "true"); ``` Stream Load typically has high QPS, so StarRocks allows sampling for Load Profile collection to avoid performance degradation from extensive profiling. You can adjust the collection interval by configuring the FE parameter `load_profile_collect_interval_second`. This setting only applies to Load Profiles enabled via table properties. The default value is `0`. ```sql ADMIN SET FRONTEND CONFIG ("load_profile_collect_interval_second"="30"); ``` StarRocks also allows collecting profiles only from load jobs that exceed a certain time threshold. You can adjust this threshold by configuring the FE parameter `stream_load_profile_collect_threshold_second`. The default value is `0`. ```sql ADMIN SET FRONTEND CONFIG ("stream_load_profile_collect_threshold_second"="10"); ``` ##### Analyze Load Profiles[​](#analyze-load-profiles "Direct link to Analyze Load Profiles") The structure of Load Profiles is identical to that of Query Profiles. For detailed instructions, see [Query Tuning Recipes](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_tuning_recipes.md). You can analyze Load Profiles by executing [ANALYZE PROFILE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/plan_profile/ANALYZE_PROFILE.md). For detailed instructions, see [Analyze text-based Profiles](https://docs.starrocks.io/docs/best_practices/query_tuning/query_profile_text_based_analysis.md). Profiles provide detailed operator metrics. Key components include the `OlapTableSink` operator and the `LoadChannel` operator. ###### OlapTableSink operator[​](#olaptablesink-operator "Direct link to OlapTableSink operator") | Metric | Description | | ----------------- | --------------------------------------------------------------------------------------------------------------- | | IndexNum | Number of synchronous materialized views of the target table. | | ReplicatedStorage | Whether single leader replication is enabled. | | TxnID | Load transaction ID. | | RowsRead | Number of data rows read from the upstream operator. | | RowsFiltered | The number of data rows that are filtered out due to inadequate data quality. | | RowsReturned | The number of data rows that are loaded. | | RpcClientSideTime | Total time consumed for data writing RPC from client-side statistics. | | RpcServerSideTime | Total time consumed for data writing RPC from server-side statistics. | | PrepareDataTime | Time consumed for data format conversion and data quality checks. | | SendDataTime | Local time consumed for sending data, including data serialization, compression, and writing to the send queue. | tip * The significant variance between the maximum and minimum values of `PushChunkNum` in `OLAP_TABLE_SINK` indicates data skew in the upstream operator, which may cause write performance bottlenecks. * `RpcClientSideTime` equals the sum of `RpcServerSideTime`, Network transmission time, and RPC framework processing time. If the difference between `RpcClientSideTime` and `RpcServerSideTime` is significant, consider to enable data compression to reduce transmission time. * If `RpcServerSideTime` accounts for a significant portion of the time spent, further analysis can be conducted using `LoadChannel` Profile. ###### LoadChannel operator[​](#loadchannel-operator "Direct link to LoadChannel operator") | Metric | Description | | ------------------- | -------------------------------------------------------------------------------------------------------- | | Address | IP address or FQDN of the BE node. | | LoadMemoryLimit | Memory Limit for loading. | | PeakMemoryUsage | Peak memory usage for loading. | | OpenCount | The number of times the channel is opened, reflecting the sink's total concurrency. | | OpenTime | Total time consumed for the opening channel. | | AddChunkCount | Number of loading chunks, that is, the number of calls to `TabletsChannel::add_chunk`. | | AddRowNum | The number of data rows that are loaded. | | AddChunkTime | Total time consumed by loading chunks, that is, the total execution time of `TabletsChannel::add_chunk`. | | WaitFlushTime | Total time spent by `TabletsChannel::add_chunk` waiting for MemTable flush. | | WaitWriterTime | Total time spent by `TabletsChannel::add_chunk` waiting for Async Delta Writer execution. | | WaitReplicaTime | Total time spent by `TabletsChannel::add_chunk` waiting for synchronization from replicas. | | PrimaryTabletsNum | Number of primary tablets. | | SecondaryTabletsNum | Number of secondary tablets. | tip If `WaitFlushTime` takes an extended period, it may indicate insufficient resources for the flush thread. Consider adjusting the BE configuration `flush_thread_num_per_store`. #### Best practices[​](#best-practices "Direct link to Best practices") ##### Diagnose Broker Load performance bottleneck[​](#diagnose-broker-load-performance-bottleneck "Direct link to Diagnose Broker Load performance bottleneck") 1. Load data using Broker Load: ```sql LOAD LABEL click_bench.hits_1713874468 ( DATA INFILE ("s3://test-data/benchmark_data/query_data/click_bench/hits.tbl*") INTO TABLE hits COLUMNS TERMINATED BY "\t" (WatchID,JavaEnable,Title,GoodEvent,EventTime,EventDate,CounterID,ClientIP,RegionID,UserID,CounterClass,OS,UserAgent,URL,Referer,IsRefresh,RefererCategoryID,RefererRegionID,URLCategoryID,URLRegionID,ResolutionWidth,ResolutionHeight,ResolutionDepth,FlashMajor,FlashMinor,FlashMinor2,NetMajor,NetMinor,UserAgentMajor,UserAgentMinor,CookieEnable,JavascriptEnable,IsMobile,MobilePhone,MobilePhoneModel,Params,IPNetworkID,TraficSourceID,SearchEngineID,SearchPhrase,AdvEngineID,IsArtifical,WindowClientWidth,WindowClientHeight,ClientTimeZone,ClientEventTime,SilverlightVersion1,SilverlightVersion2,SilverlightVersion3,SilverlightVersion4,PageCharset,CodeVersion,IsLink,IsDownload,IsNotBounce,FUniqID,OriginalURL,HID,IsOldCounter,IsEvent,IsParameter,DontCountHits,WithHash,HitColor,LocalEventTime,Age,Sex,Income,Interests,Robotness,RemoteIP,WindowName,OpenerName,HistoryLength,BrowserLanguage,BrowserCountry,SocialNetwork,SocialAction,HTTPError,SendTiming,DNSTiming,ConnectTiming,ResponseStartTiming,ResponseEndTiming,FetchTiming,SocialSourceNetworkID,SocialSourcePage,ParamPrice,ParamOrderID,ParamCurrency,ParamCurrencyID,OpenstatServiceName,OpenstatCampaignID,OpenstatAdID,OpenstatSourceID,UTMSource,UTMMedium,UTMCampaign,UTMContent,UTMTerm,FromTag,HasGCLID,RefererHash,URLHash,CLID) ) WITH BROKER ( "aws.s3.access_key" = "", "aws.s3.secret_key" = "", "aws.s3.region" = "" ) ``` 2. Use **SHOW PROFILELIST** to retrieve the list of runtime profiles. ```sql MySQL [click_bench]> SHOW PROFILELIST; +--------------------------------------+---------------------+----------+---------+----------------------------------------------------------------------------------------------------------------------------------+ | QueryId | StartTime | Time | State | Statement | +--------------------------------------+---------------------+----------+---------+----------------------------------------------------------------------------------------------------------------------------------+ | 3df61627-f82b-4776-b16a-6810279a79a3 | 2024-04-23 20:28:26 | 11s850ms | Running | LOAD LABEL click_bench.hits_1713875306 (DATA INFILE ("s3://test-data/benchmark_data/query_data/click_bench/hits.tbl*" ... | +--------------------------------------+---------------------+----------+---------+----------------------------------------------------------------------------------------------------------------------------------+ 1 row in set (0.00 sec) ``` 3. Use **ANALYZE PROFILE** to view the Runtime Profile. ```sql MySQL [click_bench]> ANALYZE PROFILE FROM '3df61627-f82b-4776-b16a-6810279a79a3'; +-------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Explain String | +-------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Summary | | Attention: The transaction of the statement will be aborted, and no data will be actually inserted!!! | | Attention: Profile is not identical!!! | | QueryId: 3df61627-f82b-4776-b16a-6810279a79a3 | | Version: default_profile-70fe819 | | State: Running | | Legend: ⏳ for blocked; 🚀 for running; ✅ for finished | | TotalTime: 31s832ms | | ExecutionTime: 30s1ms [Scan: 28s885ms (96.28%), Network: 0ns (0.00%), ResultDeliverTime: 7s613ms (25.38%), ScheduleTime: 145.701ms (0.49%)] | | FrontendProfileMergeTime: 3.838ms | | QueryPeakMemoryUsage: 141.367 MB, QueryAllocatedMemoryUsage: 82.422 GB | | Top Most Time-consuming Nodes: | | 1. FILE_SCAN (id=0) 🚀 : 28s902ms (85.43%) | | 2. OLAP_TABLE_SINK 🚀 : 4s930ms (14.57%) | | Top Most Memory-consuming Nodes: | | Progress (finished operator/all operator): 0.00% | | NonDefaultVariables: | | big_query_profile_threshold: 0s -> 60s | | enable_adaptive_sink_dop: false -> true | | enable_profile: false -> true | | sql_mode_v2: 32 -> 34 | | use_compute_nodes: -1 -> 0 | | Fragment 0 | | │ BackendNum: 3 | | │ InstancePeakMemoryUsage: 128.541 MB, InstanceAllocatedMemoryUsage: 82.422 GB | | │ PrepareTime: 2.304ms | | └──OLAP_TABLE_SINK | | │ TotalTime: 4s930ms (14.57%) [CPUTime: 4s930ms] | | │ OutputRows: 14.823M (14823424) | | │ PartitionType: RANDOM | | │ Table: hits | | └──FILE_SCAN (id=0) 🚀 | | Estimates: [row: ?, cpu: ?, memory: ?, network: ?, cost: ?] | | TotalTime: 28s902ms (85.43%) [CPUTime: 17.038ms, ScanTime: 28s885ms] | | OutputRows: 14.823M (14823424) | | Progress (processed rows/total rows): ? | | Detail Timers: [ScanTime = IOTaskExecTime + IOTaskWaitTime] | | IOTaskExecTime: 25s612ms [min=19s376ms, max=28s804ms] | | IOTaskWaitTime: 63.192ms [min=20.946ms, max=91.668ms] | | | +-------------------------------------------------------------------------------------------------------------------------------------------------------------+ 40 rows in set (0.04 sec) ``` The profile shows that the `FILE_SCAN` section took nearly 29 seconds, accounting for approximately 90% of the total 32-second duration. This indicates that reading data from object storage is currently the bottleneck in the loading process. ##### Diagnose Stream Load Performance[​](#diagnose-stream-load-performance "Direct link to Diagnose Stream Load Performance") 1. Enable Load Profile for the target table. ```sql mysql> ALTER TABLE duplicate_200_column_sCH SET('enable_load_profile'='true'); Query OK, 0 rows affected (0.00 sec) ``` 2. Use **SHOW PROFILELIST** to retrieve the list of profiles. ```sql mysql> SHOW PROFILELIST; +--------------------------------------+---------------------+----------+----------+-----------+ | QueryId | StartTime | Time | State | Statement | +--------------------------------------+---------------------+----------+----------+-----------+ | 90481df8-afaf-c0fd-8e91-a7889c1746b6 | 2024-09-19 10:43:38 | 9s571ms | Finished | | | 9c41a13f-4d7b-2c18-4eaf-cdeea3facba5 | 2024-09-19 10:43:37 | 10s664ms | Finished | | | 5641cf37-0af4-f116-46c6-ca7cce149886 | 2024-09-19 10:43:20 | 13s88ms | Finished | | | 4446c8b3-4dc5-9faa-dccb-e1a71ab3519e | 2024-09-19 10:43:20 | 13s64ms | Finished | | | 48469b66-3866-1cd9-9f3b-17d786bb4fa7 | 2024-09-19 10:43:20 | 13s85ms | Finished | | | bc441907-e779-bc5a-be8e-992757e4d992 | 2024-09-19 10:43:19 | 845ms | Finished | | +--------------------------------------+---------------------+----------+----------+-----------+ ``` 3. Use **ANALYZE PROFILE** to view the Profile. ```sql mysql> ANALYZE PROFILE FROM '90481df8-afaf-c0fd-8e91-a7889c1746b6'; +-----------------------------------------------------------+ | Explain String | +-----------------------------------------------------------+ | Load: | | Summary: | | - Query ID: 90481df8-afaf-c0fd-8e91-a7889c1746b6 | | - Start Time: 2024-09-19 10:43:38 | | - End Time: 2024-09-19 10:43:48 | | - Query Type: Load | | - Load Type: STREAM_LOAD | | - Query State: Finished | | - StarRocks Version: main-d49cb08 | | - Sql Statement | | - Default Db: ingestion_db | | - NumLoadBytesTotal: 799008 | | - NumRowsAbnormal: 0 | | - NumRowsNormal: 280 | | - Total: 9s571ms | | - numRowsUnselected: 0 | | Execution: | | Fragment 0: | | - Address: 172.26.93.218:59498 | | - InstanceId: 90481df8-afaf-c0fd-8e91-a7889c1746b7 | | - TxnID: 1367 | | - ReplicatedStorage: true | | - AutomaticPartition: false | | - InstanceAllocatedMemoryUsage: 12.478 MB | | - InstanceDeallocatedMemoryUsage: 10.745 MB | | - InstancePeakMemoryUsage: 9.422 MB | | - MemoryLimit: -1.000 B | | - RowsProduced: 280 | | - AllocAutoIncrementTime: 348ns | | - AutomaticBucketSize: 0 | | - BytesRead: 0.000 B | | - CloseWaitTime: 9s504ms | | - IOTaskExecTime: 0ns | | - IOTaskWaitTime: 0ns | | - IndexNum: 1 | | - NumDiskAccess: 0 | | - OpenTime: 15.639ms | | - PeakMemoryUsage: 0.000 B | | - PrepareDataTime: 583.480us | | - ConvertChunkTime: 44.670us | | - ValidateDataTime: 109.333us | | - RowsFiltered: 0 | | - RowsRead: 0 | | - RowsReturned: 280 | | - RowsReturnedRate: 12.049K (12049) /sec | | - RpcClientSideTime: 28s396ms | | - RpcServerSideTime: 28s385ms | | - RpcServerWaitFlushTime: 0ns | | - ScanTime: 9.841ms | | - ScannerQueueCounter: 1 | | - ScannerQueueTime: 3.272us | | - ScannerThreadsInvoluntaryContextSwitches: 0 | | - ScannerThreadsTotalWallClockTime: 0ns | | - MaterializeTupleTime(*): 0ns | | - ScannerThreadsSysTime: 0ns | | - ScannerThreadsUserTime: 0ns | | - ScannerThreadsVoluntaryContextSwitches: 0 | | - SendDataTime: 2.452ms | | - PackChunkTime: 1.475ms | | - SendRpcTime: 1.617ms | | - CompressTime: 0ns | | - SerializeChunkTime: 880.424us | | - WaitResponseTime: 0ns | | - TotalRawReadTime(*): 0ns | | - TotalReadThroughput: 0.000 B/sec | | DataSource: | | - DataSourceType: FileDataSource | | - FileScanner: | | - CastChunkTime: 0ns | | - CreateChunkTime: 227.100us | | - FileReadCount: 3 | | - FileReadTime: 253.765us | | - FillTime: 6.892ms | | - MaterializeTime: 133.637us | | - ReadTime: 0ns | | - ScannerTotalTime: 9.292ms | +-----------------------------------------------------------+ 76 rows in set (0.00 sec) ``` #### Appendix[​](#appendix "Direct link to Appendix") ##### Useful SQL for Operations[​](#useful-sql-for-operations "Direct link to Useful SQL for Operations") note This section only applies to shared-nothing clusters. ###### query the throughput per minute[​](#query-the-throughput-per-minute "Direct link to query the throughput per minute") ```sql -- overall select date_trunc('minute', load_finish_time) as t,count(*) as tpm,sum(SCAN_BYTES) as scan_bytes,sum(sink_rows) as sink_rows from _statistics_.loads_history group by t order by t desc limit 10; -- table select date_trunc('minute', load_finish_time) as t,count(*) as tpm,sum(SCAN_BYTES) as scan_bytes,sum(sink_rows) as sink_rows from _statistics_.loads_history where table_name = 't' group by t order by t desc limit 10; ``` ###### Query RowsetNum and SegmentNum of a table[​](#query-rowsetnum-and-segmentnum-of-a-table "Direct link to Query RowsetNum and SegmentNum of a table") ```sql -- overall select * from information_schema.be_tablets t, information_schema.tables_config c where t.table_id = c.table_id order by num_segment desc limit 5; select * from information_schema.be_tablets t, information_schema.tables_config c where t.table_id = c.table_id order by num_rowset desc limit 5; -- table select * from information_schema.be_tablets t, information_schema.tables_config c where t.table_id = c.table_id and table_name = 't' order by num_segment desc limit 5; select * from information_schema.be_tablets t, information_schema.tables_config c where t.table_id = c.table_id and table_name = 't' order by num_rowset desc limit 5; ``` * High RowsetNum (>100) indicates too frequent loads. You may consider to reduce frequency or increase Compaction threads. * High SegmentNum (>100) indicates excessive segments per load. You may consider increase Compaction threads or adopt the random distribution strategy for the table. ###### Check data skew[​](#check-data-skew "Direct link to Check data skew") ###### Data skew across nodes[​](#data-skew-across-nodes "Direct link to Data skew across nodes") ```sql -- overall SELECT tbt.be_id, sum(tbt.DATA_SIZE) FROM information_schema.tables_config tb JOIN information_schema.be_tablets tbt ON tb.TABLE_ID = tbt.TABLE_ID group by be_id; -- table SELECT tbt.be_id, sum(tbt.DATA_SIZE) FROM information_schema.tables_config tb JOIN information_schema.be_tablets tbt ON tb.TABLE_ID = tbt.TABLE_ID WHERE tb.table_name = 't' group by be_id; ``` If you detected node-level skew, you may consider to use a higher-cardinality column as the distribution key or adopt the random distribution strategy for the table. ###### Data skew across tablets[​](#data-skew-across-tablets "Direct link to Data skew across tablets") ```sql select tablet_id,t.data_size,num_row,visible_version,num_version,num_rowset,num_segment,PARTITION_NAME from information_schema.partitions_meta m, information_schema.be_tablets t where t.partition_id = m.partition_id and m.partition_name = 'att' and m.table_name='att' order by t.data_size desc; ``` ##### Common monitoring metrics for loading[​](#common-monitoring-metrics-for-loading "Direct link to Common monitoring metrics for loading") ###### BE Load[​](#be-load "Direct link to BE Load") These metrics are available under the **BE Load** category in Grafana. If you cannot find this category, verify that you are using the [latest Grafana dashboard template](https://docs.starrocks.io/docs/administration/management/monitoring/Monitor_and_Alert.md#125-configure-dashboard). ###### ThreadPool[​](#threadpool "Direct link to ThreadPool") These metrics help analyze the status of thread pools — for example, whether tasks are being backlogged, or how long they spend pending. Currently, there are four monitored thread pools: * `async_delta_writer` * `memtable_flush` * `segment_replicate_sync` * `segment_flush` Each thread pool includes the following metrics: | Name | Description | | ----------- | --------------------------------------------------------------------------------------------------------- | | **rate** | Task processing rate. | | **pending** | Time tasks spend waiting in the queue. | | **execute** | Task execution time. | | **total** | Maximum number of threads available in the pool. | | **util** | Pool utilization over a given period; due to sampling inaccuracy, it may exceed 100% when heavily loaded. | | **count** | Instantaneous number of tasks in the queue. | note * A reliable indicator for backlog is whether **pending duration** keeps increasing. **workers util** and **queue count** are necessary but not sufficient indicators. * If a backlog occurs, use **rate** and **execute duration** to determine whether it is due to increased load or slower processing. * **workers util** helps assess how busy the pool is, which can guide tuning efforts. ###### LoadChannel::add\_chunks[​](#loadchanneladd_chunks "Direct link to LoadChannel::add_chunks") These metrics help analyze the behavior of `LoadChannel::add_chunks` after receiving a `BRPC tablet_writer_add_chunks` request. | Name | Description | | ------------------ | --------------------------------------------------------------------------------------- | | **rate** | Processing rate of `add_chunks` requests. | | **execute** | Average execution time of `add_chunks`. | | **wait\_memtable** | Average wait time for the primary replica’s MemTable flush. | | **wait\_writer** | Average wait time for the primary replica’s async delta writer to perform write/commit. | | **wait\_replica** | Average wait time for secondary replicas to complete segment flush. | note * The **latency** metric equals the sum of `wait_memtable`, `wait_writer`, and `wait_replica`. * A high waiting ratio indicates downstream bottlenecks, which should be further analyzed. ###### Async Delta Writer[​](#async-delta-writer "Direct link to Async Delta Writer") These metrics help analyze the behavior of the **async delta writer**. | Name | Description | | ------------------ | ------------------------------------------------- | | **rate** | Processing rate of write/commit tasks. | | **pending** | Time spent waiting in the thread pool queue. | | **execute** | Average time to process a single task. | | **wait\_memtable** | Average time waiting for MemTable flush. | | **wait\_replica** | Average time waiting for segment synchronization. | note * The total time per task (from the upstream perspective) equals **pending** plus **execute**. * **execute** further includes **wait\_memtable** plus **wait\_replica**. * A high **pending** time may indicate that **execute** is slow or the thread pool is undersized. * If **wait** occupies a large portion of **execute**, downstream stages are the bottleneck; otherwise, the bottleneck is likely within the writer’s logic itself. ###### MemTable Flush[​](#memtable-flush "Direct link to MemTable Flush") These metrics analyze **MemTable flush** performance. | Name | Description | | --------------- | -------------------------------------------- | | **rate** | Flush rate of MemTables. | | **memory-size** | Amount of in-memory data flushed per second. | | **disk-size** | Amount of disk data written per second. | | **execute** | Task execution time. | | **io** | I/O time of the flush task. | note * By comparing **rate** and **size**, you can determine whether the workload is changing or if massive imports are occurring — for example, a small **rate** but large **size** indicates a massive import. * The compression ratio can be estimated using `memory-size / disk-size`. * You can also assess if I/O is the bottleneck by checking the proportion of **io** time in **execute**. ###### Segment Replicate Sync[​](#segment-replicate-sync "Direct link to Segment Replicate Sync") | Name | Description | | ----------- | -------------------------------------------- | | **rate** | Rate of segment synchronization. | | **execute** | Time to synchronize a single tablet replica. | ###### Segment Flush[​](#segment-flush "Direct link to Segment Flush") These metrics analyze **segment flush** performance. | Name | Description | | ----------- | --------------------------------------- | | **rate** | Segment flush rate. | | **size** | Amount of disk data flushed per second. | | **execute** | Task execution time. | | **io** | I/O time of the flush task. | note * By comparing **rate** and **size**, you can determine whether the workload is changing or if large imports are occurring — for example, a small **rate** but large **size** indicates a massive import. * You can also assess if I/O is the bottleneck by checking the proportion of **io** time in **execute**. --- ### Load data using tools StarRocks and its ecosystem partners offer the following tools to help you seamlessly integrate StarRocks with external databases. #### [SMT](https://docs.starrocks.io/docs/integrations/loading_tools/SMT.md)[​](#smt "Direct link to smt") SMT (StarRocks Migration Tool) is a data migration tool provided by StarRocks, designed to optimize complex data loading pipelines: source databases (such as MySQL, Oracle, PostgreSQL) ---> Flink ---> destination StarRocks clusters. Its main functions are as follows: * Simplifies table creation in StarRocks: Generates statements to create tables in StarRocks based on information from external databases and the target StarRocks cluster. * Simplifies the full or incremental data synchronization process in the data pipeline: Generates SQL statements that can be run in Flink's SQL client to submit Flink jobs for synchronizing data. The following flowchart illustrates the process of loading data from the source database MySQL through Flink into StarRocks. ![img](/assets/images/load_tools-8e88eab0e5c9d2a228fcd668c63bc3e1.png) #### [DataX](https://docs.starrocks.io/docs/integrations/loading_tools/DataX-starrocks-writer.md)[​](#datax "Direct link to datax") DataX is a tool for offline data synchronization, and is open-sourced by Alibaba. DataX can synchronize data between various heterogeneous data sources, including relational databases (MySQL, Oracle, etc.), HDFS, and Hive. DataX provides the StarRocks Writer plugin to synchronize data from data sources supported by DataX to StarRocks. #### [CloudCanal](https://docs.starrocks.io/docs/integrations/loading_tools/CloudCanal.md)[​](#cloudcanal "Direct link to cloudcanal") CloudCanal Community Edition is a free data migration and synchronization platform published by [ClouGence Co., Ltd](https://www.bladepipe.com/) that integrates Schema Migration, Full Data Migration, verification, Correction, and real-time Incremental Synchronization. You can directly add StarRocks as a data source in CloudCanal's visual interface and create tasks to automatically migrate or synchronize data from source databases (e.g., MySQL, Oracle, PostgreSQL) to StarRocks. #### [Kettle connector](https://github.com/StarRocks/starrocks-connector-for-kettle)[​](#kettle-connector "Direct link to kettle-connector") Kettle is an ETL (Extract, Transform, Load) tool with a visual graphical interface, which allows users to build data processing workflows by dragging components and configuring parameters. This intuitive method greatly simplifies the process of data processing and loading, enabling users to handle data more conveniently. Additionally, Kettle provides a rich library of components, allowing users to select suitable components according to their needs and perform various complex data processing tasks. StarRocks offers the Kettle Connector to integrate with Kettle. By combining Kettle's robust data processing and transformation capabilities with StarRocks's high-performance data storage and analytical abilities, more flexible and efficient data processing workflows can be achieved. --- ### Load data from MinIO StarRocks provides the following options for loading data from MinIO: * Synchronous loading using [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md)+[`FILES()`](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md) * Asynchronous loading using [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) Each of these options has its own advantages, which are detailed in the following sections. In most cases, we recommend that you use the INSERT+`FILES()` method, which is much easier to use. However, the INSERT+`FILES()` method currently supports only the Parquet, ORC, and CSV file formats. Therefore, if you need to load data of other file formats such as JSON, or [perform data changes such as DELETE during data loading](https://docs.starrocks.io/docs/loading/Load_to_Primary_Key_tables.md), you can resort to Broker Load. #### Before you begin[​](#before-you-begin "Direct link to Before you begin") ##### Make source data ready[​](#make-source-data-ready "Direct link to Make source data ready") Make sure the source data you want to load into StarRocks is properly stored in a MinIO bucket. You may also consider where the data and the database are located, because data transfer costs are much lower when your bucket and your StarRocks cluster are located in the same region. In this topic, we provide you with a sample dataset. You can download this with `curl`: ```bash curl -O https://starrocks-examples.s3.amazonaws.com/user_behavior_ten_million_rows.parquet ``` Load the Parquet file into your MinIO system and note the bucket name. The examples in this guide use a bucket name of `/starrocks`. ##### Check privileges[​](#check-privileges "Direct link to Check privileges") You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. ##### Gather connection details[​](#gather-connection-details "Direct link to Gather connection details") In a nutshell, to use MinIO Access Key authentication you need to gather the following information: * The bucket that stores your data * The object key (object name) if accessing a specific object in the bucket * The MinIO endpoint * The access key and secret key used as access credentials. ![MinIO access key](/assets/images/MinIO-create-f8a2e63a6a2e6a718cc0b46a115a019d.png) #### Use INSERT+FILES()[​](#use-insertfiles "Direct link to Use INSERT+FILES()") This method is available from v3.1 onwards and currently supports only the Parquet, ORC, and CSV (from v3.3.0 onwards) file formats. ##### Advantages of INSERT+FILES()[​](#advantages-of-insertfiles "Direct link to Advantages of INSERT+FILES()") [`FILES()`](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md) can read the file stored in cloud storage based on the path-related properties you specify, infer the table schema of the data in the file, and then return the data from the file as data rows. With `FILES()`, you can: * Query the data directly from MinIO using [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md). * Create and load a table using [CREATE TABLE AS SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE_AS_SELECT.md) (CTAS). * Load the data into an existing table using [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md). ##### Typical examples[​](#typical-examples "Direct link to Typical examples") ###### Querying directly from MinIO using SELECT[​](#querying-directly-from-minio-using-select "Direct link to Querying directly from MinIO using SELECT") Querying directly from MinIO using SELECT+`FILES()` can give a good preview of the content of a dataset before you create a table. For example: * Get a preview of the dataset without storing the data. * Query for the min and max values and decide what data types to use. * Check for `NULL` values. The following example queries the sample dataset previously added to your MinIO system. tip The highlighted section of the command includes the settings that you may need to change: * Set the `endpoint` and `path` to match your MinIO system. * If your MinIO system uses SSL set `enable_ssl` to `true`. * Substitute your MinIO access key and secret for `AAA` and `BBB`. ```sql SELECT * FROM FILES ( "aws.s3.endpoint" = "http://minio:9000", "path" = "s3://starrocks/user_behavior_ten_million_rows.parquet", "aws.s3.enable_ssl" = "false", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "format" = "parquet", "aws.s3.use_aws_sdk_default_behavior" = "false", "aws.s3.use_instance_profile" = "false", "aws.s3.enable_path_style_access" = "true" ) LIMIT 3; ``` The system returns the following query result: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 543711 | 829192 | 2355072 | pv | 2017-11-27 08:22:37 | | 543711 | 2056618 | 3645362 | pv | 2017-11-27 10:16:46 | | 543711 | 1165492 | 3645362 | pv | 2017-11-27 10:17:00 | +--------+---------+------------+--------------+---------------------+ 3 rows in set (0.41 sec) ``` info Notice that the column names returned above are provided by the Parquet file. ###### Creating and loading a table using CTAS[​](#creating-and-loading-a-table-using-ctas "Direct link to Creating and loading a table using CTAS") This is a continuation of the previous example. The previous query is wrapped in CREATE TABLE AS SELECT (CTAS) to automate the table creation using schema inference. This means StarRocks will infer the table schema, create the table you want, and then load the data into the table. The column names and types are not required to create a table when using the `FILES()` table function with Parquet files as the Parquet format includes the column names. note The syntax of CREATE TABLE when using schema inference does not allow setting the number of replicas, so set it before creating the table. The example below is for a system with a single replica: ```sql ADMIN SET FRONTEND CONFIG ('default_replication_num' = '1'); ``` Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Use CTAS to create a table and load the data of the sample dataset previously added to your MinIO system. tip The highlighted section of the command includes the settings that you may need to change: * Set the `endpoint` and `path` to match your MinIO system. * If your MinIO system uses SSL set `enable_ssl` to `true`. * Substitute your MinIO access key and secret key for `AAA` and `BBB`. ```sql CREATE TABLE user_behavior_inferred AS SELECT * FROM FILES ( "aws.s3.endpoint" = "http://minio:9000", "path" = "s3://starrocks/user_behavior_ten_million_rows.parquet", "aws.s3.enable_ssl" = "false", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "format" = "parquet", "aws.s3.use_aws_sdk_default_behavior" = "false", "aws.s3.use_instance_profile" = "false", "aws.s3.enable_path_style_access" = "true" ); ``` ```plaintext Query OK, 10000000 rows affected (3.17 sec) {'label':'insert_a5da3ff5-9ee4-11ee-90b0-02420a060004', 'status':'VISIBLE', 'txnId':'17'} ``` After creating the table, you can view its schema by using [DESCRIBE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DESCRIBE.md): ```sql DESCRIBE user_behavior_inferred; ``` The system returns the following query result: ```plaintext +--------------+------------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------+------------------+------+-------+---------+-------+ | UserID | bigint | YES | true | NULL | | | ItemID | bigint | YES | true | NULL | | | CategoryID | bigint | YES | true | NULL | | | BehaviorType | varchar(1048576) | YES | false | NULL | | | Timestamp | varchar(1048576) | YES | false | NULL | | +--------------+------------------+------+-------+---------+-------+ ``` Query the table to verify that the data has been loaded into it. Example: ```sql SELECT * from user_behavior_inferred LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plaintext +--------+--------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+--------+------------+--------------+---------------------+ | 58 | 158350 | 2355072 | pv | 2017-11-27 13:06:51 | | 58 | 158590 | 3194735 | pv | 2017-11-27 02:21:04 | | 58 | 215073 | 3002561 | pv | 2017-11-30 10:55:42 | +--------+--------+------------+--------------+---------------------+ ``` ###### Loading into an existing table using INSERT[​](#loading-into-an-existing-table-using-insert "Direct link to Loading into an existing table using INSERT") You may want to customize the table that you are inserting into, for example, the: * column data type, nullable setting, or default values * key types and columns * data partitioning and bucketing tip Creating the most efficient table structure requires knowledge of how the data will be used and the content of the columns. This topic does not cover table design. For information about table design, see [Table types](https://docs.starrocks.io/docs/table_design/StarRocks_table_design.md). In this example, we are creating a table based on knowledge of how the table will be queried and the data in the Parquet file. The knowledge of the data in the Parquet file can be gained by querying the file directly in MinIO. * Since a query of the dataset in MinIO indicates that the `Timestamp` column contains data that matches a `datetime` data type, the column type is specified in the following DDL. * By querying the data in MinIO, you can find that there are no `NULL` values in the dataset, so the DDL does not set any columns as nullable. * Based on knowledge of the expected query types, the sort key and bucketing column are set to the column `UserID`. Your use case might be different for this data, so you might decide to use `ItemID` in addition to or instead of `UserID` for the sort key. Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand (we recommend that the table have the same schema as the Parquet file you want to load from MinIO): ```sql CREATE TABLE user_behavior_declared ( UserID int(11) NOT NULL, ItemID int(11) NOT NULL, CategoryID int(11) NOT NULL, BehaviorType varchar(65533) NOT NULL, Timestamp datetime NOT NULL ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID) PROPERTIES ( 'replication_num' = '1' ); ``` Display the schema so that you can compare it with the inferred schema produced by the `FILES()` table function: ```sql DESCRIBE user_behavior_declared; ``` ```plaintext +--------------+----------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------+----------------+------+-------+---------+-------+ | UserID | int | NO | true | NULL | | | ItemID | int | NO | false | NULL | | | CategoryID | int | NO | false | NULL | | | BehaviorType | varchar(65533) | NO | false | NULL | | | Timestamp | datetime | NO | false | NULL | | +--------------+----------------+------+-------+---------+-------+ 5 rows in set (0.00 sec) ``` tip Compare the schema you just created with the schema inferred earlier using the `FILES()` table function. Look at: * data types * nullable * key fields To better control the schema of the destination table and for better query performance, we recommend that you specify the table schema by hand in production environments. Having a `datetime` data type for the timestamp field is more efficient than using a `varchar`. After creating the table, you can load it with INSERT INTO SELECT FROM FILES(): tip The highlighted section of the command includes the settings that you may need to change: * Set the `endpoint` and `path` to match your MinIO system. * If your MinIO system uses SSL set `enable_ssl` to `true`. * Substitute your MinIO access key and secret key for `AAA` and `BBB`. ```sql INSERT INTO user_behavior_declared SELECT * FROM FILES ( "aws.s3.endpoint" = "http://minio:9000", "path" = "s3://starrocks/user_behavior_ten_million_rows.parquet", "aws.s3.enable_ssl" = "false", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "format" = "parquet", "aws.s3.use_aws_sdk_default_behavior" = "false", "aws.s3.use_instance_profile" = "false", "aws.s3.enable_path_style_access" = "true" ); ``` After the load is complete, you can query the table to verify that the data has been loaded into it. Example: ```sql SELECT * from user_behavior_declared LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 58 | 4309692 | 1165503 | pv | 2017-11-25 14:06:52 | | 58 | 181489 | 1165503 | pv | 2017-11-25 14:07:22 | | 58 | 3722956 | 1165503 | pv | 2017-11-25 14:09:28 | +--------+---------+------------+--------------+---------------------+ ``` ###### Check load progress[​](#check-load-progress "Direct link to Check load progress") You can query the progress of INSERT jobs from the [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md) view in the StarRocks Information Schema. This feature is supported from v3.1 onwards. Example: ```sql SELECT * FROM information_schema.loads ORDER BY JOB_ID DESC; ``` For information about the fields provided in the `loads` view, see [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md). If you have submitted multiple load jobs, you can filter on the `LABEL` associated with the job. Example: ```sql SELECT * FROM information_schema.loads WHERE LABEL = 'insert_e3b882f5-7eb3-11ee-ae77-00163e267b60' \G *************************** 1. row *************************** JOB_ID: 10243 LABEL: insert_e3b882f5-7eb3-11ee-ae77-00163e267b60 DATABASE_NAME: mydatabase STATE: FINISHED PROGRESS: ETL:100%; LOAD:100% TYPE: INSERT PRIORITY: NORMAL SCAN_ROWS: 10000000 FILTERED_ROWS: 0 UNSELECTED_ROWS: 0 SINK_ROWS: 10000000 ETL_INFO: TASK_INFO: resource:N/A; timeout(s):300; max_filter_ratio:0.0 CREATE_TIME: 2023-11-09 11:56:01 ETL_START_TIME: 2023-11-09 11:56:01 ETL_FINISH_TIME: 2023-11-09 11:56:01 LOAD_START_TIME: 2023-11-09 11:56:01 LOAD_FINISH_TIME: 2023-11-09 11:56:44 JOB_DETAILS: {"All backends":{"e3b882f5-7eb3-11ee-ae77-00163e267b60":[10142]},"FileNumber":0,"FileSize":0,"InternalTableLoadBytes":311710786,"InternalTableLoadRows":10000000,"ScanBytes":581574034,"ScanRows":10000000,"TaskNumber":1,"Unfinished backends":{"e3b882f5-7eb3-11ee-ae77-00163e267b60":[]}} ERROR_MSG: NULL TRACKING_URL: NULL TRACKING_SQL: NULL REJECTED_RECORD_PATH: NULL ``` tip INSERT is a synchronous command. If an INSERT job is still running, you need to open another session to check its execution status. ##### Compare the table sizes on disk[​](#compare-the-table-sizes-on-disk "Direct link to Compare the table sizes on disk") This query compares the table with the inferred schema and the one where the schema is declared. Because the inferred schema has nullable columns and a varchar for the timestamp the data length is larger: ```sql SELECT TABLE_NAME, TABLE_ROWS, AVG_ROW_LENGTH, DATA_LENGTH FROM information_schema.tables WHERE TABLE_NAME like 'user_behavior%'\G ``` ```plaintext *************************** 1. row *************************** TABLE_NAME: user_behavior_declared TABLE_ROWS: 10000000 AVG_ROW_LENGTH: 10 DATA_LENGTH: 102562516 *************************** 2. row *************************** TABLE_NAME: user_behavior_inferred TABLE_ROWS: 10000000 AVG_ROW_LENGTH: 17 DATA_LENGTH: 176803880 2 rows in set (0.04 sec) ``` #### Use Broker Load[​](#use-broker-load "Direct link to Use Broker Load") An asynchronous Broker Load process handles making the connection to MinIO, pulling the data, and storing the data in StarRocks. This method supports the following file formats: * Parquet * ORC * CSV * JSON (supported from v3.2.3 onwards) ##### Advantages of Broker Load[​](#advantages-of-broker-load "Direct link to Advantages of Broker Load") * Broker Load runs in the background and clients do not need to stay connected for the job to continue. * Broker Load is preferred for long-running jobs, with the default timeout spanning 4 hours. * In addition to Parquet and ORC file format, Broker Load supports CSV file format and JSON file format (JSON file format is supported from v3.2.3 onwards). ##### Data flow[​](#data-flow "Direct link to Data flow") ![Workflow of Broker Load](/assets/images/broker_load_how-to-work_en-bb36de70866e6366b2b21808f0f77be8.png) 1. The user creates a load job. 2. The frontend (FE) creates a query plan and distributes the plan to the backend nodes (BEs) or compute nodes (CNs). 3. The BEs or CNs pull the data from the source and load the data into StarRocks. ##### Typical example[​](#typical-example "Direct link to Typical example") Create a table, start a load process that pulls the sample dataset previously loaded to your MinIO system. ###### Create a database and a table[​](#create-a-database-and-a-table "Direct link to Create a database and a table") Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand (we recommend that the table has the same schema as the Parquet file that you want to load from MinIO): ```sql CREATE TABLE user_behavior ( UserID int(11) NOT NULL, ItemID int(11) NOT NULL, CategoryID int(11) NOT NULL, BehaviorType varchar(65533) NOT NULL, Timestamp datetime NOT NULL ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID) PROPERTIES ( 'replication_num' = '1' ); ``` ###### Start a Broker Load[​](#start-a-broker-load "Direct link to Start a Broker Load") Run the following command to start a Broker Load job that loads data from the sample dataset `user_behavior_ten_million_rows.parquet` to the `user_behavior` table: tip The highlighted section of the command includes the settings that you may need to change: * Set the `endpoint` and `DATA INFILE` to match your MinIO system. * If your MinIO system uses SSL set `enable_ssl` to `true`. * Substitute your MinIO access key and secret for `AAA` and `BBB`. ```sql LOAD LABEL UserBehavior ( DATA INFILE("s3://starrocks/user_behavior_ten_million_rows.parquet") INTO TABLE user_behavior ) WITH BROKER ( "aws.s3.endpoint" = "http://minio:9000", "aws.s3.enable_ssl" = "false", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "aws.s3.use_aws_sdk_default_behavior" = "false", "aws.s3.use_instance_profile" = "false", "aws.s3.enable_path_style_access" = "true" ) PROPERTIES ( "timeout" = "72000" ); ``` This job has four main sections: * `LABEL`: A string used when querying the state of the load job. * `LOAD` declaration: The source URI, source data format, and destination table name. * `BROKER`: The connection details for the source. * `PROPERTIES`: The timeout value and any other properties to apply to the load job. For detailed syntax and parameter descriptions, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). ###### Check load progress[​](#check-load-progress-1 "Direct link to Check load progress") You can query the progress of Broker Load jobs from the [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md) view in the StarRocks Information Schema. This feature is supported from v3.1 onwards. ```sql SELECT * FROM information_schema.loads; ``` For information about the fields provided in the `loads` view, see [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md). If you have submitted multiple load jobs, you can filter on the `LABEL` associated with the job. Example: ```sql SELECT * FROM information_schema.loads WHERE LABEL = 'UserBehavior'\G ``` ```plaintext *************************** 1. row *************************** JOB_ID: 10176 LABEL: userbehavior DATABASE_NAME: mydatabase STATE: FINISHED PROGRESS: ETL:100%; LOAD:100% TYPE: BROKER PRIORITY: NORMAL SCAN_ROWS: 10000000 FILTERED_ROWS: 0 UNSELECTED_ROWS: 0 SINK_ROWS: 10000000 ETL_INFO: TASK_INFO: resource:N/A; timeout(s):72000; max_filter_ratio:0.0 CREATE_TIME: 2023-12-19 23:02:41 ETL_START_TIME: 2023-12-19 23:02:44 ETL_FINISH_TIME: 2023-12-19 23:02:44 LOAD_START_TIME: 2023-12-19 23:02:44 LOAD_FINISH_TIME: 2023-12-19 23:02:46 JOB_DETAILS: {"All backends":{"4aeec563-a91e-4c1e-b169-977b660950d1":[10004]},"FileNumber":1,"FileSize":132251298,"InternalTableLoadBytes":311710786,"InternalTableLoadRows":10000000,"ScanBytes":132251298,"ScanRows":10000000,"TaskNumber":1,"Unfinished backends":{"4aeec563-a91e-4c1e-b169-977b660950d1":[]}} ERROR_MSG: NULL TRACKING_URL: NULL TRACKING_SQL: NULL REJECTED_RECORD_PATH: NULL 1 row in set (0.02 sec) ``` After you confirm that the load job has finished, you can check a subset of the destination table to see if the data has been successfully loaded. Example: ```sql SELECT * from user_behavior LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 142 | 2869980 | 2939262 | pv | 2017-11-25 03:43:22 | | 142 | 2522236 | 1669167 | pv | 2017-11-25 15:14:12 | | 142 | 3031639 | 3607361 | pv | 2017-11-25 15:19:25 | +--------+---------+------------+--------------+---------------------+ ``` --- ### Load data using Routine Load tip Try Routine Load out in this [Quick Start](https://docs.starrocks.io/docs/quick_start/routine-load.md) This topic introduces how to create a Routine Load job to stream Kafka messages (events) into StarRocks, and familiarizes you with some basic concepts about Routine Load. To continuously load messages of a stream into StarRocks, you can store the message stream in a Kafka topic, and create a Routine Load job to consume the messages. The Routine Load job persists in StarRocks, generates a series of load tasks to consume the messages in all or part of the partitions in the topic, and loads the messages into StarRocks. A Routine Load job supports exactly-once delivery semantics to guarantee the data loaded into StarRocks is neither lost nor duplicated. Routine Load supports data transformation at data loading and supports data changes made by UPSERT and DELETE operations during data loading. For more information, see [Transform data at loading](https://docs.starrocks.io/docs/loading/Etl_in_loading.md) and [Change data through loading](https://docs.starrocks.io/docs/loading/Load_to_Primary_Key_tables.md). You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. #### Supported data formats[​](#supported-data-formats "Direct link to Supported data formats") Routine Load now supports consuming CSV, JSON, and Avro (supported since v3.0.1) formatted data from a Kafka cluster. > **NOTE** > > For CSV data, take note of the following points: > > * You can use a UTF-8 string, such as a comma (,), tab, or pipe (|), whose length does not exceed 50 bytes as a text delimiter. > * Null values are denoted by using `\N`. For example, a data file consists of three columns, and a record from that data file holds data in the first and third columns but no data in the second column. In this situation, you need to use `\N` in the second column to denote a null value. This means the record must be compiled as `a,\N,b` instead of `a,,b`. `a,,b` denotes that the second column of the record holds an empty string. #### Basic concepts[​](#basic-concepts "Direct link to Basic concepts") ![routine load](/assets/images/4.5.2-1-a6d6d07699a1f32c703d4b80921e2e44.png) ##### Terminology[​](#terminology "Direct link to Terminology") * **Load job** A Routine Load job is a long-running job. As long as its status is RUNNING, a load job continuously generates one or multiple concurrent load tasks which consume the messages in a topic of a Kafka cluster and load the data into StarRocks. * **Load task** A load job is split into multiple load tasks by certain rules. A load task is the basic unit of data loading. As an individual event, a load task implements the load mechanism based on [Stream Load](https://docs.starrocks.io/docs/loading/StreamLoad.md). Multiple load tasks concurrently consume the messages from different partitions of a topic, and load the data into StarRocks. ##### Workflow[​](#workflow "Direct link to Workflow") 1. **Create a Routine Load job.** To load data from Kafka, you need to create a Routine Load job by running the [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md) statement. The FE parses the statement, and creates the job according to the properties you have specified. 2. **The FE splits the job into multiple load tasks.** The FE split the job into multiple load tasks based on certain rules. Each load task is an individual transaction. The splitting rules are as follows: * The FE calculates the actual concurrent number of the load tasks according to the desired concurrent number `desired_concurrent_number`, the partition number in the Kafka topic, and the number of the BE nodes that are alive. * The FE splits the job into load tasks based on the actual concurrent number calculated, and arranges the tasks in the task queue. Each Kafka topic consists of multiple partitions. The relation between the topic partition and the load task is as follows: * A partition is uniquely assigned to a load task, and all messages from the partition are consumed by the load task. * A load task can consume messages from one or more partitions. * All partitions are distributed evenly among load tasks. 3. **Multiple load tasks run concurrently to consume the messages from multiple Kafka topic partitions, and load the data into StarRocks** 1. **The FE schedules and submits load tasks**: the FE schedules the load tasks in the queue on a timely basis, and assigns them to selected Coordinator BE nodes. The interval between load tasks is defined by the configuration item `max_batch_interval`. The FE distributes the load tasks evenly to all BE nodes. See [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md#examples) for more information about `max_batch_interval`. 2. The Coordinator BE starts the load task, consumes messages in partitions, parses and filters the data. A load task lasts until the pre-defined amount of messages are consumed or the pre-defined time limit is reached. The message batch size and time limit are defined in the FE configurations `max_routine_load_batch_size` and `routine_load_task_consume_second`. For detailed information, see [FE Configuration](https://docs.starrocks.io/docs/administration/management/FE_configuration.md). The Coordinator BE then distributes the messages to the Executor BEs. The Executor BEs write the messages to disks. > **NOTE** > > StarRocks supports access to Kafka via security protocols including SASL\_SSL, SAS\_PLAINTEXT, SSL, and PLAINTEXT. This topic uses connecting to Kafka via PLAINTEXT as an example. If you need to connect to Kafka via other security protocols, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). 4. **The FE generates new load tasks to load data continuously.** After the Executor BEs has written the data to disks, the Coordinator BE reports the result of the load task to the FE. Based on the result, the FE then generates new load tasks to load the data continuously. Or the FE retries the failed tasks to make sure the data loaded into StarRocks is neither lost nor duplicated. #### Create a Routine Load job[​](#create-a-routine-load-job "Direct link to Create a Routine Load job") The following three examples describe how to consume CSV-format, JSON-format and Avro-format data in Kafka, and load the data into StarRocks by creating a Routine Load job. For detailed syntax and parameter descriptions, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). ##### Load CSV-format data[​](#load-csv-format-data "Direct link to Load CSV-format data") This section describes how to create a Routine Load job to consume CSV-format data in a Kafka cluster, and load the data into StarRocks. ###### Prepare a dataset[​](#prepare-a-dataset "Direct link to Prepare a dataset") Suppose there is a CSV-format dataset in the topic `ordertest1` in a Kafka cluster. Every message in the dataset includes six fields: order ID, payment date, customer name, nationality, gender, and price. ```plain 2020050802,2020-05-08,Johann Georg Faust,Deutschland,male,895 2020050802,2020-05-08,Julien Sorel,France,male,893 2020050803,2020-05-08,Dorian Grey,UK,male,1262 2020050901,2020-05-09,Anna Karenina",Russia,female,175 2020051001,2020-05-10,Tess Durbeyfield,US,female,986 2020051101,2020-05-11,Edogawa Conan,japan,male,8924 ``` ###### Create a table[​](#create-a-table "Direct link to Create a table") According to the fields of CSV-format data, create the table `example_tbl1` in the database `example_db`. The following example creates a table with 5 fields excluding the field of customer gender in the CSV-format data. ```sql CREATE TABLE example_db.example_tbl1 ( `order_id` bigint NOT NULL COMMENT "Order ID", `pay_dt` date NOT NULL COMMENT "Payment date", `customer_name` varchar(26) NULL COMMENT "Customer name", `nationality` varchar(26) NULL COMMENT "Nationality", `price`double NULL COMMENT "Price" ) ENGINE=OLAP DUPLICATE KEY (order_id,pay_dt) DISTRIBUTED BY HASH(`order_id`); ``` > **NOTICE** > > Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). ###### Submit a Routine Load job[​](#submit-a-routine-load-job "Direct link to Submit a Routine Load job") Execute the following statement to submit a Routine Load job named `example_tbl1_ordertest1` to consume the messages in the topic `ordertest1` and load the data into the table `example_tbl1`. The load task consumes the messages from the initial offset in the specified partitions of the topic. ```sql CREATE ROUTINE LOAD example_db.example_tbl1_ordertest1 ON example_tbl1 COLUMNS TERMINATED BY ",", COLUMNS (order_id, pay_dt, customer_name, nationality, temp_gender, price) PROPERTIES ( "desired_concurrent_number" = "5" ) FROM KAFKA ( "kafka_broker_list" = ":,:", "kafka_topic" = "ordertest1", "kafka_partitions" = "0,1,2,3,4", "property.kafka_default_offsets" = "OFFSET_BEGINNING" ); ``` After submitting the load job, you can execute the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement to check the status of the load job. * **load job name** There could be multiple load job on a table. Therefore, we recommend you name a load job with the corresponding Kafka topic and the time when the load job is submitted. It helps you distinguish the load job on each table. * **Column separator** The property `COLUMN TERMINATED BY` defines the column separator of the CSV-format data. The default is `\t`. * **Kafka topic partition and offset** You can specify the properties `kafka_partitions` and `kafka_offsets` to specify the partitions and offsets to consume the messages. For example, if you want the load job to consume messages from the Kafka partitions `"0,1,2,3,4"` of the topic `ordertest1` all with the initial offsets, you can specify the properties as follows: If you want the load job to consume messages from the Kafka partitions `"0,1,2,3,4"`and you need to specify a separate starting offset for each partition, you can configure as follows: ```sql "kafka_partitions" ="0,1,2,3,4", "kafka_offsets" = "OFFSET_BEGINNING, OFFSET_END, 1000, 2000, 3000" ``` You can also set the default offsets of all partitions with the property `property.kafka_default_offsets`. ```sql "kafka_partitions" ="0,1,2,3,4", "property.kafka_default_offsets" = "OFFSET_BEGINNING" ``` For detailed information, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). * **Data mapping and transformation** To specify the mapping and transformation relationship between the CSV-format data, and the StarRocks table, you need to use the `COLUMNS` parameter. **Data mapping:** * StarRocks extracts the columns in the CSV-format data and maps them **in sequence** onto the fields declared in the `COLUMNS` parameter. * StarRocks extracts the fields declared in the `COLUMNS` parameter and maps them **by name** onto the columns of StarRocks table. **Data transformation:** And because the example excludes the column of customer gender from the CSV-format data, the field `temp_gender` in `COLUMNS` parameter is used as a placeholder for this field. The other fields are mapped to columns of the StarRocks table `example_tbl1` directly. For more information about data transformation, see [Transform data at loading](https://docs.starrocks.io/docs/loading/Etl_in_loading.md). > **NOTE** > > You do not need to specify the `COLUMNS` parameter if the names, number, and order of the columns in the CSV-format data completely correspond to those of the StarRocks table. * **Task concurrency** When there are many Kafka topic partitions and enough BE nodes, you can accelerate the loading by increasing the task concurrency. To increase the actual load task concurrency, you can increase the desired load task concurrency `desired_concurrent_number` when you create a routine load job. You can also set the dynamic configuration item of FE `max_routine_load_task_concurrent_num` ( default maximum load task currency ) to a larger value. For more information about `max_routine_load_task_concurrent_num`, please see [FE configuration items](https://docs.starrocks.io/docs/administration/management/FE_configuration.md). The actual task concurrency is defined by the minimum value among the number of BE nodes that are alive, the number of the pre-specified Kafka topic partitions, and the values of `desired_concurrent_number` and `max_routine_load_task_concurrent_num`. In the example, the number of BE nodes that are alive is `5`, the number of the pre-specified Kafka topic partitions is `5`, and the value of `max_routine_load_task_concurrent_num` is `5`. To increase the actual load task concurrency, you can increase the `desired_concurrent_number` from the default value `3` to `5`. For more about the properties, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). ##### Load JSON-format data[​](#load-json-format-data "Direct link to Load JSON-format data") This section describes how to create a Routine Load job to consume JSON-format data in a Kafka cluster, and load the data into StarRocks. ###### Prepare a dataset[​](#prepare-a-dataset-1 "Direct link to Prepare a dataset") Suppose there is a JSON-format dataset in the topic `ordertest2` in a Kafka cluster. The dataset includes six keys: commodity ID, customer name, nationality, payment time, and price. Besides, you want to transform the payment time column into the DATE type, and load it into the `pay_dt` column in the StarRocks table. ```json {"commodity_id": "1", "customer_name": "Mark Twain", "country": "US","pay_time": 1589191487,"price": 875} {"commodity_id": "2", "customer_name": "Oscar Wilde", "country": "UK","pay_time": 1589191487,"price": 895} {"commodity_id": "3", "customer_name": "Antoine de Saint-Exupéry","country": "France","pay_time": 1589191487,"price": 895} ``` > **CAUTION** Each JSON object in a row must be in one Kafka message, otherwise a JSON parsing error is returned. ###### Create a table[​](#create-a-table-1 "Direct link to Create a table") According to the keys of the JSON-format data, create the table `example_tbl2` in the database `example_db`. ```sql CREATE TABLE `example_tbl2` ( `commodity_id` varchar(26) NULL COMMENT "Commodity ID", `customer_name` varchar(26) NULL COMMENT "Customer name", `country` varchar(26) NULL COMMENT "Country", `pay_time` bigint(20) NULL COMMENT "Payment time", `pay_dt` date NULL COMMENT "Payment date", `price`double SUM NULL COMMENT "Price" ) ENGINE=OLAP AGGREGATE KEY(`commodity_id`,`customer_name`,`country`,`pay_time`,`pay_dt`) DISTRIBUTED BY HASH(`commodity_id`); ``` > **NOTICE** > > Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). ###### Submit a Routine Load job[​](#submit-a-routine-load-job-1 "Direct link to Submit a Routine Load job") Execute the following statement to submit a Routine Load job named `example_tbl2_ordertest2` to consume the messages in the topic `ordertest2` and load the data into the table `example_tbl2`. The load task consumes the messages from the initial offset in the specified partitions of the topic. ```sql CREATE ROUTINE LOAD example_db.example_tbl2_ordertest2 ON example_tbl2 COLUMNS(commodity_id, customer_name, country, pay_time, price, pay_dt=from_unixtime(pay_time, '%Y%m%d')) PROPERTIES ( "desired_concurrent_number" = "5", "format" = "json", "jsonpaths" = "[\"$.commodity_id\",\"$.customer_name\",\"$.country\",\"$.pay_time\",\"$.price\"]" ) FROM KAFKA ( "kafka_broker_list" =":,:", "kafka_topic" = "ordertest2", "kafka_partitions" ="0,1,2,3,4", "property.kafka_default_offsets" = "OFFSET_BEGINNING" ); ``` After submitting the load job, you can execute the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement to check the status of the load job. * **Data format** You need to specify `"format" = "json"` in the clause `PROPERTIES` to define that the data format is JSON. * **Data mapping and transformation** To specify the mapping and transformation relationship between the JSON-format data, and the StarRocks table, you need to specify the parameter `COLUMNS` and property`jsonpaths`. The order of fields specified in the `COLUMNS` parameter must match that of the JSON-format data, and the name of fields must match that of the StarRocks table. The property `jsonpaths` is used to extract the required fields from the JSON data. These fields are then named by the property `COLUMNS`. Because the example needs to transform the payment time field to the DATE data type, and load the data into the `pay_dt` column in the StarRocks table, you need to use the from\_unixtime function. The other fields are mapped to fields of the table `example_tbl2` directly. **Data mapping:** * StarRocks extracts the `name` and `code` keys of JSON-format data and maps them onto the keys declared in the `jsonpaths` property. * StarRocks extracts the keys declared in the `jsonpaths` property and maps them **in sequence** onto the fields declared in the `COLUMNS` parameter. * StarRocks extracts the fields declared in the `COLUMNS` parameter and maps them **by name** onto the columns of StarRocks table. **Data transformation**: * Because the example needs to transform the key `pay_time` to the DATE data type, and load the data into the `pay_dt` column in the StarRocks table, you need to use the from\_unixtime function in `COLUMNS` parameter. The other fields are mapped to fields of the table `example_tbl2` directly. * And because the example excludes the column of customer gender from the JSON-format data, the field `temp_gender` in `COLUMNS` parameter is used as a placeholder for this field. The other fields are mapped to columns of the StarRocks table `example_tbl1` directly. For more information about data transformation, see [Transform data at loading](https://docs.starrocks.io/docs/loading/Etl_in_loading.md). > **NOTE** > > You do not need to specify the `COLUMNS` parameter if the names and number of the keys in the JSON object completely match those of fields in the StarRocks table. ##### Load Avro-format data[​](#load-avro-format-data "Direct link to Load Avro-format data") Since v3.0.1, StarRocks supports loading Avro data by using Routine Load. ###### Prepare a dataset[​](#prepare-a-dataset-2 "Direct link to Prepare a dataset") ###### Avro schema[​](#avro-schema "Direct link to Avro schema") 1. Create the following Avro schema file `avro_schema.avsc`: ```json { "type": "record", "name": "sensor_log", "fields" : [ {"name": "id", "type": "long"}, {"name": "name", "type": "string"}, {"name": "checked", "type" : "boolean"}, {"name": "data", "type": "double"}, {"name": "sensor_type", "type": {"type": "enum", "name": "sensor_type_enum", "symbols" : ["TEMPERATURE", "HUMIDITY", "AIR-PRESSURE"]}} ] } ``` 2. Register the Avro schema in the [Schema Registry](https://docs.confluent.io/cloud/current/get-started/schema-registry.html#create-a-schema). ###### Avro data[​](#avro-data "Direct link to Avro data") Prepare the Avro data and send it to the Kafka topic `topic_0`. ###### Create a table[​](#create-a-table-2 "Direct link to Create a table") According to the fields of Avro data, create a table `sensor_log` in the target database `example_db` in the StarRocks cluster. The column names of the table must match the field names in the Avro data. For the data type mapping between the table columns and the Avro data fields, see \[Data types mapping]\(#Data types mapping). ```sql CREATE TABLE example_db.sensor_log ( `id` bigint NOT NULL COMMENT "sensor id", `name` varchar(26) NOT NULL COMMENT "sensor name", `checked` boolean NOT NULL COMMENT "checked", `data` double NULL COMMENT "sensor data", `sensor_type` varchar(26) NOT NULL COMMENT "sensor type" ) ENGINE=OLAP DUPLICATE KEY (id) DISTRIBUTED BY HASH(`id`); ``` > **NOTICE** > > Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). ###### Submit a Routine Load job[​](#submit-a-routine-load-job-2 "Direct link to Submit a Routine Load job") Execute the following statement to submit a Routine Load job named `sensor_log_load_job` to consume the Avro messages in the Kafka topic `topic_0` and load the data into the table `sensor_log` in the database `sensor`. The load job consumes the messages from the initial offset in the specified partitions of the topic. ```sql CREATE ROUTINE LOAD example_db.sensor_log_load_job ON sensor_log PROPERTIES ( "format" = "avro" ) FROM KAFKA ( "kafka_broker_list" = ":,:,...", "confluent.schema.registry.url" = "http://172.xx.xxx.xxx:8081", "kafka_topic" = "topic_0", "kafka_partitions" = "0,1,2,3,4,5", "property.kafka_default_offsets" = "OFFSET_BEGINNING" ); ``` * Data Format You need to specify `"format = "avro"` in the clause `PROPERTIES` to define that the data format is Avro. * Schema Registry You need to configure `confluent.schema.registry.url` to specify the URL of the Schema Registry where the Avro schema is registered. StarRocks retrieves the Avro schema by using this URL. The format is as follows: ```plaintext confluent.schema.registry.url = http[s]://[:@][:] ``` * Data mapping and transformation To specify the mapping and transformation relationship between the Avro-format data and the StarRocks table, you need to specify the parameter `COLUMNS` and property `jsonpaths`. The order of fields specified in the `COLUMNS` parameter must match that of the fields in the property `jsonpaths`, and the names of fields must match these of the StarRocks table. The property `jsonpaths` is used to extract the required fields from the Avro data. These fields are then named by the property `COLUMNS`. For more information about data transformation, see [Transform data at loading](https://docs.starrocks.io/docs/loading/Etl_in_loading.md). > NOTE > > You do not need to specify the `COLUMNS` parameter if the names and number of the fields in the Avro record completely match those of columns in the StarRocks table. After submitting the load job, you can execute the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement to check the status of the load job. ###### Data types mapping[​](#data-types-mapping "Direct link to Data types mapping") The data type mapping between the Avro data fields you want to load and the StarRocks table columns is as follows: ###### Primitive types[​](#primitive-types "Direct link to Primitive types") | Avro | StarRocks | | ------- | --------- | | nul | NULL | | boolean | BOOLEAN | | int | INT | | long | BIGINT | | float | FLOAT | | double | DOUBLE | | bytes | STRING | | string | STRING | ###### Complex types[​](#complex-types "Direct link to Complex types") | Avro | StarRocks | | -------------- | -------------------------------------------------------------------------- | | record | STRUCT, or load the entire RECORD or its subfields into StarRocks as JSON. | | enums | STRING | | arrays | ARRAY | | maps | MAP or JSON | | union(T, null) | NULLABLE(T) | | fixed | STRING | ###### Limits[​](#limits "Direct link to Limits") * Currently, StarRocks does not support schema evolution. * Each Kafka message must only contain a single Avro data record. ##### Access source message metadata[​](#access-source-message-metadata "Direct link to Access source message metadata") When loading JSON- or Avro-format data, you can populate destination columns from a message's Kafka/Pulsar metadata — topic, partition, offset, timestamp, key, and headers — instead of from the message payload. You declare the metadata in an `INCLUDE METADATA (...)` clause that binds each metadata key to an alias; the alias is an ordinary source column you reference from `COLUMNS`. This is useful for auditing (which topic/partition/offset a row came from), event-time processing (using the message timestamp), and routing on a header value. ###### Syntax[​](#syntax "Direct link to Syntax") ```sql INCLUDE METADATA ( [AS ] [, [AS ] ...] ) ``` `INCLUDE METADATA` is a load property; place it among the other load properties (such as `COLUMNS` and `WHERE`), in any order, before the `PROPERTIES` and `FROM` clauses. `AS ` is optional. If it is omitted, the alias defaults to ``. The alias must be unique within the clause, and it must not collide with a payload field, a destination-table column, or a reserved column name. ###### Metadata keys[​](#metadata-keys "Direct link to Metadata keys") The supported keys depend on the data source. | Source | Key | Type | Description | | ------ | ----------------- | ---------------------- | ------------------------------------------------------------------------------------------------- | | KAFKA | `TOPIC` | VARCHAR | Topic name. | | KAFKA | `PARTITION` | INT | Partition number. | | KAFKA | `OFFSET` | BIGINT | Message offset within the partition. | | KAFKA | `TIMESTAMP_MS` | BIGINT | Record timestamp in milliseconds since epoch. `NULL` when the broker reports no timestamp. | | KAFKA | `KEY` | VARCHAR | Message key as raw bytes. `NULL` when the message has no key. | | KAFKA | `HEADERS` | MAP\ | All headers as a map. On duplicate keys, the last value wins. | | PULSAR | `TOPIC` | VARCHAR | The logical topic the job consumes (a partitioned topic's `-partition-N` suffix is not included). | | PULSAR | `PARTITION` | INT | Partition index, parsed from the per-message topic name. `NULL` for a non-partitioned topic. | | PULSAR | `KEY` | VARCHAR | Partition key. `NULL` when the message has no key. | | PULSAR | `MESSAGE_ID` | VARCHAR | Message ID. | | PULSAR | `PUBLISH_TIME_MS` | BIGINT | Publish time in milliseconds since epoch. | | PULSAR | `EVENT_TIME_MS` | BIGINT | Event time in milliseconds since epoch. `NULL` when the producer did not set it. | | PULSAR | `PROPERTIES` | MAP\ | All properties as a map. On duplicate keys, the last value wins. | To read a single header/property value, use `element_at(, '')` over the `HEADERS`/`PROPERTIES` map: the last value wins on a duplicate key, and the result is `NULL` when the key is absent. Header/property values are raw bytes placed into VARCHAR as-is (no UTF-8 validation). ###### Usage notes[​](#usage-notes "Direct link to Usage notes") * `INCLUDE METADATA` is available for `format = json` (Kafka and Pulsar) and `format = avro` (Kafka only; Pulsar Routine Load does not support Avro). It is not supported for CSV, where one message can expand into many rows and per-message metadata would be ambiguous. * A metadata alias is an ordinary source column: reference it anywhere in the `COLUMNS` expressions. If a payload field has the same name as a metadata key, specify a different alias with `AS ` to avoid ambiguity. * `OFFSET` is Kafka-only and `MESSAGE_ID` is Pulsar-only; using a key unsupported by the source raises an error that lists the keys supported for that source. * `HEADERS`/`PROPERTIES` is a `MAP\`. The source headers/properties are an ordered list that may repeat a key; duplicates collapse into the map with the last value winning (an `element_at(map, 'name')` lookup is likewise last-wins, and returns `NULL` when the key is absent). Values are raw bytes stored in VARCHAR as-is — there is no UTF-8 validation or decoding. ###### Example[​](#example "Direct link to Example") Load the order payload field `order_id` together with the source topic, partition, offset, the message timestamp converted to a `DATETIME`, and a `trace-id` header: ```sql CREATE TABLE example_db.orders_with_meta ( order_id BIGINT, src_topic VARCHAR(256), src_partition INT, src_offset BIGINT, msg_time DATETIME, trace_id VARCHAR(128) ) ENGINE = OLAP DUPLICATE KEY(order_id) DISTRIBUTED BY HASH(order_id); CREATE ROUTINE LOAD example_db.orders_with_meta_job ON orders_with_meta INCLUDE METADATA ( TOPIC AS m_topic, PARTITION AS m_partition, OFFSET AS m_offset, TIMESTAMP_MS AS m_timestamp, HEADERS AS m_headers ), COLUMNS ( order_id, src_topic = m_topic, src_partition = m_partition, src_offset = m_offset, msg_time = from_unixtime(m_timestamp / 1000), trace_id = element_at(m_headers, 'trace-id') ) PROPERTIES ( "format" = "json", "jsonpaths" = "[\"$.order_id\"]" ) FROM KAFKA ( "kafka_broker_list" = ":,...", "kafka_topic" = "topic_orders", "property.kafka_default_offsets" = "OFFSET_BEGINNING" ); ``` A metadata alias may be used inside an expression (as with `from_unixtime(m_timestamp / 1000)` above); only payload columns are listed in `jsonpaths`. #### Check a load job and task[​](#check-a-load-job-and-task "Direct link to Check a load job and task") ##### Check a load job[​](#check-a-load-job "Direct link to Check a load job") Execute the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement to check the status of the load job `example_tbl2_ordertest2`. StarRocks returns the execution state `State`, the statistical information (including the total rows consumed and the total rows loaded) `Statistics`, and the progress of the load job `progress`. If the state of the load job is automatically changed to **PAUSED**, it is possibly because the number of error rows has exceeded the threshold. For detailed instructions on setting this threshold, see [CREATE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD.md). You can check the files `ReasonOfStateChanged` and `ErrorLogUrls` to identify and troubleshoot the problem. Having fixed the problem, you can then execute the [RESUME ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/RESUME_ROUTINE_LOAD.md) statement to resume the **PAUSED** load job. If the state of the load job is **CANCELLED**, it is possibly because the load job encounters an exception (such as the table has been dropped). You can check the files `ReasonOfStateChanged` and `ErrorLogUrls` to identify and troubleshoot the problem. However, you cannot resume a **CANCELLED** load job. ```sql MySQL [example_db]> SHOW ROUTINE LOAD FOR example_tbl2_ordertest2 \G *************************** 1. row *************************** Id: 63013 Name: example_tbl2_ordertest2 CreateTime: 2022-08-10 17:09:00 PauseTime: NULL EndTime: NULL DbName: default_cluster:example_db TableName: example_tbl2 State: RUNNING DataSourceType: KAFKA CurrentTaskNum: 3 JobProperties: {"partitions":"*","partial_update":"false","columnToColumnExpr":"commodity_id,customer_name,country,pay_time,pay_dt=from_unixtime(`pay_time`, '%Y%m%d'),price","maxBatchIntervalS":"20","whereExpr":"*","dataFormat":"json","timezone":"Asia/Shanghai","format":"json","json_root":"","strict_mode":"false","jsonpaths":"[\"$.commodity_id\",\"$.customer_name\",\"$.country\",\"$.pay_time\",\"$.price\"]","desireTaskConcurrentNum":"3","maxErrorNum":"0","strip_outer_array":"false","currentTaskConcurrentNum":"3","maxBatchRows":"200000"} DataSourceProperties: {"topic":"ordertest2","currentKafkaPartitions":"0,1,2,3,4","brokerList":":,:"} CustomProperties: {"kafka_default_offsets":"OFFSET_BEGINNING"} Statistic: {"receivedBytes":230,"errorRows":0,"committedTaskNum":1,"loadedRows":2,"loadRowsRate":0,"abortedTaskNum":0,"totalRows":2,"unselectedRows":0,"receivedBytesRate":0,"taskExecuteTimeMs":522} Progress: {"0":"1","1":"OFFSET_ZERO","2":"OFFSET_ZERO","3":"OFFSET_ZERO","4":"OFFSET_ZERO"} ReasonOfStateChanged: ErrorLogUrls: OtherMsg: ``` > **CAUTION** > > You cannot check a load job that has stopped or has not yet started. ##### Check a load task[​](#check-a-load-task "Direct link to Check a load task") Execute the [SHOW ROUTINE LOAD TASK](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD_TASK.md) statement to check the load tasks of the load job `example_tbl2_ordertest2`, such as how many tasks are currently running, the Kafka topic partitions that are consumed and the consumption progress `DataSourceProperties`, and the corresponding Coordinator BE node `BeId`. ```sql MySQL [example_db]> SHOW ROUTINE LOAD TASK WHERE JobName = "example_tbl2_ordertest2" \G *************************** 1. row *************************** TaskId: 18c3a823-d73e-4a64-b9cb-b9eced026753 TxnId: -1 TxnStatus: UNKNOWN JobId: 63013 CreateTime: 2022-08-10 17:09:05 LastScheduledTime: 2022-08-10 17:47:27 ExecuteStartTime: NULL Timeout: 60 BeId: -1 DataSourceProperties: {"1":0,"4":0} Message: there is no new data in kafka, wait for 20 seconds to schedule again *************************** 2. row *************************** TaskId: f76c97ac-26aa-4b41-8194-a8ba2063eb00 TxnId: -1 TxnStatus: UNKNOWN JobId: 63013 CreateTime: 2022-08-10 17:09:05 LastScheduledTime: 2022-08-10 17:47:26 ExecuteStartTime: NULL Timeout: 60 BeId: -1 DataSourceProperties: {"2":0} Message: there is no new data in kafka, wait for 20 seconds to schedule again *************************** 3. row *************************** TaskId: 1a327a34-99f4-4f8d-8014-3cd38db99ec6 TxnId: -1 TxnStatus: UNKNOWN JobId: 63013 CreateTime: 2022-08-10 17:09:26 LastScheduledTime: 2022-08-10 17:47:27 ExecuteStartTime: NULL Timeout: 60 BeId: -1 DataSourceProperties: {"0":2,"3":0} Message: there is no new data in kafka, wait for 20 seconds to schedule again ``` #### Pause a load job[​](#pause-a-load-job "Direct link to Pause a load job") You can execute the [PAUSE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/PAUSE_ROUTINE_LOAD.md) statement to pause a load job. The state of the load job will be **PAUSED** after the statement is executed. However, it has not stopped. You can execute the [RESUME ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/RESUME_ROUTINE_LOAD.md) statement to resume it. You can also check its status with the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement. The following example pauses the load job `example_tbl2_ordertest2`: ```sql PAUSE ROUTINE LOAD FOR example_tbl2_ordertest2; ``` #### Resume a load job[​](#resume-a-load-job "Direct link to Resume a load job") You can execute the [RESUME ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/RESUME_ROUTINE_LOAD.md) statement to resume a paused load job. The state of the load job will be **NEED\_SCHEDULE** temporarily (because the load job is being re-scheduled), and then become **RUNNING**. You can check its status with the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement. The following example resumes the paused load job `example_tbl2_ordertest2`: ```sql RESUME ROUTINE LOAD FOR example_tbl2_ordertest2; ``` #### Alter a load job[​](#alter-a-load-job "Direct link to Alter a load job") Before altering a load job, you must pause it with the [PAUSE ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/PAUSE_ROUTINE_LOAD.md) statement. Then you can execute the [ALTER ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/ALTER_ROUTINE_LOAD.md). After altering it, you can execute the [RESUME ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/RESUME_ROUTINE_LOAD.md) statement to resume it, and check its status with the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement. Suppose the number of the BE nodes that are alive increases to `6` and the Kafka topic partitions to be consumed is `"0,1,2,3,4,5,6,7"`. If you want to increase the actual load task concurrency, you can execute the following statement to increase the number of desired task concurrency `desired_concurrent_number` to `6` (greater than or equal to the number of BE nodes that are alive), and specify the Kafka topic partitions and initial offsets. > **NOTE** > > Because the actual task concurrency is determined by the minimum value of multiple parameters, you must make sure that the value of the FE dynamic parameter `max_routine_load_task_concurrent_num` is greater than or equal to `6`. ```sql ALTER ROUTINE LOAD FOR example_tbl2_ordertest2 PROPERTIES ( "desired_concurrent_number" = "6" ) FROM kafka ( "kafka_partitions" = "0,1,2,3,4,5,6,7", "kafka_offsets" = "OFFSET_BEGINNING,OFFSET_BEGINNING,OFFSET_BEGINNING,OFFSET_BEGINNING,OFFSET_END,OFFSET_END,OFFSET_END,OFFSET_END" ); ``` #### Stop a load job[​](#stop-a-load-job "Direct link to Stop a load job") You can execute the [STOP ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/STOP_ROUTINE_LOAD.md) statement to stop a load job. The state of the load job will be **STOPPED** after the statement is executed, and you cannot resume a stopped load job. You cannot check the status of a stopped load job with the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md) statement. The following example stops the load job `example_tbl2_ordertest2`: ```sql STOP ROUTINE LOAD FOR example_tbl2_ordertest2; ``` --- ### Load data from AWS S3 StarRocks provides the following options for loading data from AWS S3: * Synchronous loading using [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md)+[`FILES()`](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md) * Asynchronous loading using [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) * Continuous asynchronous loading using [Pipe](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/CREATE_PIPE.md) Each of these options has its own advantages, which are detailed in the following sections. In most cases, we recommend that you use the INSERT+`FILES()` method, which is much easier to use. However, the INSERT+`FILES()` method currently supports only the Parquet, ORC, and CSV file formats. Therefore, if you need to load data of other file formats such as JSON, or perform data changes such as DELETE during data loading, you can resort to Broker Load. If you need to load a large number of data files with a significant data volume in total (for example, more than 100 GB or even 1 TB), we recommend that you use the Pipe method. Pipe can split the files based on their number or size, breaking down the load job into smaller, sequential tasks. This approach ensures that errors in one file do not impact the entire load job and minimizes the need for retries due to data errors. #### Before you begin[​](#before-you-begin "Direct link to Before you begin") ##### Make source data ready[​](#make-source-data-ready "Direct link to Make source data ready") Make sure the source data you want to load into StarRocks is properly stored in an S3 bucket. You may also consider where the data and the database are located, because data transfer costs are much lower when your bucket and your StarRocks cluster are located in the same region. In this topic, we provide you with a sample dataset in an S3 bucket, `s3://starrocks-examples/user-behavior-10-million-rows.parquet`. You can access that dataset with any valid credentials as the object is readable by any AWS authenticated user. ##### Check privileges[​](#check-privileges "Direct link to Check privileges") You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. ##### Gather authentication details[​](#gather-authentication-details "Direct link to Gather authentication details") The examples in this topic use IAM user-based authentication. To ensure that you have permission to read data from AWS S3, we recommend that you read [Preparation for IAM user-based authentication](https://docs.starrocks.io/docs/integrations/authenticate_to_aws_resources.md) and follow the instructions to create an IAM user with proper [IAM policies](https://docs.starrocks.io/docs/sql-reference/aws_iam_policies.md) configured. In a nutshell, if you practice IAM user-based authentication, you need to gather information about the following AWS resources: * The S3 bucket that stores your data. * The S3 object key (object name) if accessing a specific object in the bucket. Note that the object key can include a prefix if your S3 objects are stored in sub-folders. * The AWS region to which the S3 bucket belongs. * The access key and secret key used as access credentials. For information about all the authentication methods available, see [Authenticate to AWS resources](https://docs.starrocks.io/docs/integrations/authenticate_to_aws_resources.md). #### Use INSERT+FILES()[​](#use-insertfiles "Direct link to Use INSERT+FILES()") This method is available from v3.1 onwards and currently supports only the Parquet, ORC, and CSV (from v3.3.0 onwards) file formats. ##### Advantages of INSERT+FILES()[​](#advantages-of-insertfiles "Direct link to Advantages of INSERT+FILES()") [`FILES()`](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md) can read the file stored in cloud storage based on the path-related properties you specify, infer the table schema of the data in the file, and then return the data from the file as data rows. With `FILES()`, you can: * Query the data directly from S3 using [SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SELECT.md). * Create and load a table using [CREATE TABLE AS SELECT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE_AS_SELECT.md) (CTAS). * Load the data into an existing table using [INSERT](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT.md). ##### Typical examples[​](#typical-examples "Direct link to Typical examples") ###### Querying directly from S3 using SELECT[​](#querying-directly-from-s3-using-select "Direct link to Querying directly from S3 using SELECT") Querying directly from S3 using SELECT+`FILES()` can give a good preview of the content of a dataset before you create a table. For example: * Get a preview of the dataset without storing the data. * Query for the min and max values and decide what data types to use. * Check for `NULL` values. The following example queries the sample dataset `s3://starrocks-examples/user-behavior-10-million-rows.parquet`: ```sql SELECT * FROM FILES ( "path" = "s3://starrocks-examples/user-behavior-10-million-rows.parquet", "format" = "parquet", "aws.s3.region" = "us-east-1", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB" ) LIMIT 3; ``` > **NOTE** > > Substitute your credentials for `AAA` and `BBB` in the above command. Any valid `aws.s3.access_key` and `aws.s3.secret_key` can be used, as the object is readable by any AWS authenticated user. The system returns the following query result: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 1 | 2576651 | 149192 | pv | 2017-11-25 01:21:25 | | 1 | 3830808 | 4181361 | pv | 2017-11-25 07:04:53 | | 1 | 4365585 | 2520377 | pv | 2017-11-25 07:49:06 | +--------+---------+------------+--------------+---------------------+ ``` > **NOTE** > > Notice that the column names as returned above are provided by the Parquet file. ###### Creating and loading a table using CTAS[​](#creating-and-loading-a-table-using-ctas "Direct link to Creating and loading a table using CTAS") This is a continuation of the previous example. The previous query is wrapped in CREATE TABLE AS SELECT (CTAS) to automate the table creation using schema inference. This means StarRocks will infer the table schema, create the table you want, and then load the data into the table. The column names and types are not required to create a table when using the `FILES()` table function with Parquet files as the Parquet format includes the column names. > **NOTE** > > The syntax of CREATE TABLE when using schema inference does not allow setting the number of replicas, so set it before creating the table. The example below is for a system with one replica: > > ```sql > ADMIN SET FRONTEND CONFIG ('default_replication_num' = "1"); > > ``` Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Use CTAS to create a table and load the data of the sample dataset `s3://starrocks-examples/user-behavior-10-million-rows.parquet` into the table: ```sql CREATE TABLE user_behavior_inferred AS SELECT * FROM FILES ( "path" = "s3://starrocks-examples/user-behavior-10-million-rows.parquet", "format" = "parquet", "aws.s3.region" = "us-east-1", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB" ); ``` > **NOTE** > > Substitute your credentials for `AAA` and `BBB` in the above command. Any valid `aws.s3.access_key` and `aws.s3.secret_key` can be used, as the object is readable by any AWS authenticated user. After creating the table, you can view its schema by using [DESCRIBE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DESCRIBE.md): ```sql DESCRIBE user_behavior_inferred; ``` The system returns the following query result: ```plain +--------------+------------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------+------------------+------+-------+---------+-------+ | UserID | bigint | YES | true | NULL | | | ItemID | bigint | YES | true | NULL | | | CategoryID | bigint | YES | true | NULL | | | BehaviorType | varchar(1048576) | YES | false | NULL | | | Timestamp | varchar(1048576) | YES | false | NULL | | +--------------+------------------+------+-------+---------+-------+ ``` Query the table to verify that the data has been loaded into it. Example: ```sql SELECT * from user_behavior_inferred LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 225586 | 3694958 | 1040727 | pv | 2017-12-01 00:58:40 | | 225586 | 3726324 | 965809 | pv | 2017-12-01 02:16:02 | | 225586 | 3732495 | 1488813 | pv | 2017-12-01 00:59:46 | +--------+---------+------------+--------------+---------------------+ ``` ###### Loading into an existing table using INSERT[​](#loading-into-an-existing-table-using-insert "Direct link to Loading into an existing table using INSERT") You may want to customize the table that you are inserting into, for example, the: * column data type, nullable setting, or default values * key types and columns * data partitioning and bucketing > **NOTE** > > Creating the most efficient table structure requires knowledge of how the data will be used and the content of the columns. This topic does not cover table design. For information about table design, see [Table types](https://docs.starrocks.io/docs/table_design/StarRocks_table_design.md). In this example, we are creating a table based on knowledge of how the table will be queried and the data in the Parquet file. The knowledge of the data in the Parquet file can be gained by querying the file directly in S3. * Since a query of the dataset in S3 indicates that the `Timestamp` column contains data that matches a VARCHAR data type, and StarRocks can cast from VARCHAR to DATETIME, the data type is changed to DATETIME in the following DDL. * By querying the data in S3, you can find that there are no `NULL` values in the dataset, so the DDL could also set all columns as non-nullable. * Based on knowledge of the expected query types, the sort key and bucketing column are set to the column `UserID`. Your use case might be different for this data, so you might decide to use `ItemID` in addition to, or instead of, `UserID` for the sort key. Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand: ```sql CREATE TABLE user_behavior_declared ( UserID int(11), ItemID int(11), CategoryID int(11), BehaviorType varchar(65533), Timestamp datetime ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID); ``` Display the schema so that you can compare it with the inferred schema produced by the `FILES()` table function: ```sql DESCRIBE user_behavior_declared; ``` ```plaintext +--------------+----------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +--------------+----------------+------+-------+---------+-------+ | UserID | int | YES | true | NULL | | | ItemID | int | YES | false | NULL | | | CategoryID | int | YES | false | NULL | | | BehaviorType | varchar(65533) | YES | false | NULL | | | Timestamp | datetime | YES | false | NULL | | +--------------+----------------+------+-------+---------+-------+ ``` tip Compare the schema you just created with the schema inferred earlier using the `FILES()` table function. Look at: * data types * nullable * key fields To better control the schema of the destination table and for better query performance, we recommend that you specify the table schema by hand in production environments. After creating the table, you can load it with INSERT INTO SELECT FROM FILES(): ```sql INSERT INTO user_behavior_declared SELECT * FROM FILES ( "path" = "s3://starrocks-examples/user-behavior-10-million-rows.parquet", "format" = "parquet", "aws.s3.region" = "us-east-1", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB" ); ``` > **NOTE** > > Substitute your credentials for `AAA` and `BBB` in the above command. Any valid `aws.s3.access_key` and `aws.s3.secret_key` can be used, as the object is readable by any AWS authenticated user. After the load is complete, you can query the table to verify that the data has been loaded into it. Example: ```sql SELECT * from user_behavior_declared LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 393529 | 3715112 | 883960 | pv | 2017-12-02 02:45:44 | | 393529 | 2650583 | 883960 | pv | 2017-12-02 02:45:59 | | 393529 | 3715112 | 883960 | pv | 2017-12-02 03:00:56 | +--------+---------+------------+--------------+---------------------+ ``` ###### Check load progress[​](#check-load-progress "Direct link to Check load progress") You can query the progress of INSERT jobs from the [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md) view in the StarRocks Information Schema. This feature is supported from v3.1 onwards. Example: ```sql SELECT * FROM information_schema.loads ORDER BY JOB_ID DESC; ``` For information about the fields provided in the `loads` view, see [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md). If you have submitted multiple load jobs, you can filter on the `LABEL` associated with the job. Example: ```sql SELECT * FROM information_schema.loads WHERE LABEL = 'insert_e3b882f5-7eb3-11ee-ae77-00163e267b60' \G *************************** 1. row *************************** JOB_ID: 10243 LABEL: insert_e3b882f5-7eb3-11ee-ae77-00163e267b60 DATABASE_NAME: mydatabase STATE: FINISHED PROGRESS: ETL:100%; LOAD:100% TYPE: INSERT PRIORITY: NORMAL SCAN_ROWS: 10000000 FILTERED_ROWS: 0 UNSELECTED_ROWS: 0 SINK_ROWS: 10000000 ETL_INFO: TASK_INFO: resource:N/A; timeout(s):300; max_filter_ratio:0.0 CREATE_TIME: 2023-11-09 11:56:01 ETL_START_TIME: 2023-11-09 11:56:01 ETL_FINISH_TIME: 2023-11-09 11:56:01 LOAD_START_TIME: 2023-11-09 11:56:01 LOAD_FINISH_TIME: 2023-11-09 11:56:44 JOB_DETAILS: {"All backends":{"e3b882f5-7eb3-11ee-ae77-00163e267b60":[10142]},"FileNumber":0,"FileSize":0,"InternalTableLoadBytes":311710786,"InternalTableLoadRows":10000000,"ScanBytes":581574034,"ScanRows":10000000,"TaskNumber":1,"Unfinished backends":{"e3b882f5-7eb3-11ee-ae77-00163e267b60":[]}} ERROR_MSG: NULL TRACKING_URL: NULL TRACKING_SQL: NULL REJECTED_RECORD_PATH: NULL ``` > **NOTE** > > INSERT is a synchronous command. If an INSERT job is still running, you need to open another session to check its execution status. #### Use Broker Load[​](#use-broker-load "Direct link to Use Broker Load") An asynchronous Broker Load process handles making the connection to S3, pulling the data, and storing the data in StarRocks. This method supports the following file formats: * Parquet * ORC * CSV * JSON (supported from v3.2.3 onwards) ##### Advantages of Broker Load[​](#advantages-of-broker-load "Direct link to Advantages of Broker Load") * Broker Load runs in the background and clients do not need to stay connected for the job to continue. * Broker Load is preferred for long-running jobs, with the default timeout spanning 4 hours. * In addition to Parquet and ORC file format, Broker Load supports CSV file format and JSON file format (JSON file format is supported from v3.2.3 onwards). ##### Data flow[​](#data-flow "Direct link to Data flow") ![Workflow of Broker Load](/assets/images/broker_load_how-to-work_en-bb36de70866e6366b2b21808f0f77be8.png) 1. The user creates a load job. 2. The frontend (FE) creates a query plan and distributes the plan to the backend nodes (BEs) or compute nodes (CNs). 3. The BEs or CNs pull the data from the source and load the data into StarRocks. ##### Typical example[​](#typical-example "Direct link to Typical example") Create a table, start a load process that pulls the sample dataset `s3://starrocks-examples/user-behavior-10-million-rows.parquet` from S3, and verify the progress and success of the data loading. ###### Create a database and a table[​](#create-a-database-and-a-table "Direct link to Create a database and a table") Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand (we recommend that the table has the same schema as the Parquet file that you want to load from AWS S3): ```sql CREATE TABLE user_behavior ( UserID int(11), ItemID int(11), CategoryID int(11), BehaviorType varchar(65533), Timestamp datetime ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID); ``` ###### Start a Broker Load[​](#start-a-broker-load "Direct link to Start a Broker Load") Run the following command to start a Broker Load job that loads data from the sample dataset `s3://starrocks-examples/user-behavior-10-million-rows.parquet` to the `user_behavior` table: ```sql LOAD LABEL user_behavior ( DATA INFILE("s3://starrocks-examples/user-behavior-10-million-rows.parquet") INTO TABLE user_behavior FORMAT AS "parquet" ) WITH BROKER ( "aws.s3.enable_ssl" = "true", "aws.s3.use_instance_profile" = "false", "aws.s3.region" = "us-east-1", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB" ) PROPERTIES ( "timeout" = "72000" ); ``` > **NOTE** > > Substitute your credentials for `AAA` and `BBB` in the above command. Any valid `aws.s3.access_key` and `aws.s3.secret_key` can be used, as the object is readable by any AWS authenticated user. This job has four main sections: * `LABEL`: A string used when querying the state of the load job. * `LOAD` declaration: The source URI, source data format, and destination table name. * `BROKER`: The connection details for the source. * `PROPERTIES`: The timeout value and any other properties to apply to the load job. For detailed syntax and parameter descriptions, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). ###### Check load progress[​](#check-load-progress-1 "Direct link to Check load progress") You can query the progress of the Broker Load job from the [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md) view in the StarRocks Information Schema. This feature is supported from v3.1 onwards. ```sql SELECT * FROM information_schema.loads WHERE LABEL = 'user_behavior'; ``` For information about the fields provided in the `loads` view, see [`loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md). This record shows a state of `LOADING`, and the progress is 39%. If you see something similar, then run the command again until you see a state of `FINISHED`. ```plaintext JOB_ID: 10466 LABEL: user_behavior DATABASE_NAME: mydatabase STATE: LOADING PROGRESS: ETL:100%; LOAD:39% TYPE: BROKER PRIORITY: NORMAL SCAN_ROWS: 4620288 FILTERED_ROWS: 0 UNSELECTED_ROWS: 0 SINK_ROWS: 4620288 ETL_INFO: TASK_INFO: resource:N/A; timeout(s):72000; max_filter_ratio:0.0 CREATE_TIME: 2024-02-28 22:11:36 ETL_START_TIME: 2024-02-28 22:11:41 ETL_FINISH_TIME: 2024-02-28 22:11:41 LOAD_START_TIME: 2024-02-28 22:11:41 LOAD_FINISH_TIME: NULL JOB_DETAILS: {"All backends":{"2fb97223-b14c-404b-9be1-83aa9b3a7715":[10004]},"FileNumber":1,"FileSize":136901706,"InternalTableLoadBytes":144032784,"InternalTableLoadRows":4620288,"ScanBytes":143969616,"ScanRows":4620288,"TaskNumber":1,"Unfinished backends":{"2fb97223-b14c-404b-9be1-83aa9b3a7715":[10004]}} ERROR_MSG: NULL TRACKING_URL: NULL TRACKING_SQL: NULL REJECTED_RECORD_PATH: NULL ``` After you confirm that the load job has finished, you can check a subset of the destination table to see if the data has been successfully loaded. Example: ```sql SELECT * from user_behavior LIMIT 3; ``` The following query result is returned, indicating that the data has been successfully loaded: ```plaintext +--------+---------+------------+--------------+---------------------+ | UserID | ItemID | CategoryID | BehaviorType | Timestamp | +--------+---------+------------+--------------+---------------------+ | 34 | 856384 | 1029459 | pv | 2017-11-27 14:43:27 | | 34 | 5079705 | 1029459 | pv | 2017-11-27 14:44:13 | | 34 | 4451615 | 1029459 | pv | 2017-11-27 14:45:52 | +--------+---------+------------+--------------+---------------------+ ``` #### Use Pipe[​](#use-pipe "Direct link to Use Pipe") Starting from v3.2, StarRocks provides the Pipe loading method, which currently supports only the Parquet and ORC file formats. ##### Advantages of Pipe[​](#advantages-of-pipe "Direct link to Advantages of Pipe") Pipe is ideal for continuous data loading and large-scale data loading: * **Large-scale data loading in micro-batches helps reduce the cost of retries caused by data errors.** With the help of Pipe, StarRocks enables the efficient loading of a large number of data files with a significant data volume in total. Pipe automatically splits the files based on their number or size, breaking down the load job into smaller, sequential tasks. This approach ensures that errors in one file do not impact the entire load job. The load status of each file is recorded by Pipe, allowing you to easily identify and fix files that contain errors. By minimizing the need for retries due to data errors, this approach helps to reduce costs. * **Continuous data loading helps reduce manpower.** Pipe helps you write new or updated data files to a specific location and continuously load the new data from these files into StarRocks. After you create a Pipe job with `"AUTO_INGEST" = "TRUE"` specified, it will constantly monitor changes to the data files stored in the specified path and automatically load new or updated data from the data files into the destination StarRocks table. Additionally, Pipe performs file uniqueness checks to help prevent duplicate data loading.During the loading process, Pipe checks the uniqueness of each data file based on the file name and digest. If a file with a specific file name and digest has already been processed by a Pipe job, the Pipe job will skip all subsequent files with the same file name and digest. Note that object storage like AWS S3 uses ETag as file digest. The load status of each data file is recorded and saved to the `information_schema.pipe_files` view. After a Pipe job associated with the view is deleted, the records about the files loaded in that job will also be deleted. ##### Data flow[​](#data-flow "Direct link to Data flow") ![Pipe data flow](/assets/images/pipe_data_flow-2a4dc0b44a06c987d9afc0ecf632f5d9.png) Pipe is ideal for continuous data loading and large-scale data loading: * **Large-scale data loading in micro-batches helps reduce the cost of retries caused by data errors.** With the help of Pipe, StarRocks enables the efficient loading of a large number of data files with a significant data volume in total. Pipe automatically splits the files based on their number or size, breaking down the load job into smaller, sequential tasks. This approach ensures that errors in one file do not impact the entire load job. The load status of each file is recorded by Pipe, allowing you to easily identify and fix files that contain errors. By minimizing the need for retries due to data errors, this approach helps to reduce costs. * **Continuous data loading helps reduce manpower.** Pipe helps you write new or updated data files to a specific location and continuously load the new data from these files into StarRocks. After you create a Pipe job with `"AUTO_INGEST" = "TRUE"` specified, it will constantly monitor changes to the data files stored in the specified path and automatically load new or updated data from the data files into the destination StarRocks table. Additionally, Pipe performs file uniqueness checks to help prevent duplicate data loading.During the loading process, Pipe checks the uniqueness of each data file based on the file name and digest. If a file with a specific file name and digest has already been processed by a Pipe job, the Pipe job will skip all subsequent files with the same file name and digest. Note that object storage like AWS S3 uses `ETag` as file digest. The load status of each data file is recorded and saved to the `information_schema.pipe_files` view. After a Pipe job associated with the view is deleted, the records about the files loaded in that job will also be deleted. ##### Differences between Pipe and INSERT+FILES()[​](#differences-between-pipe-and-insertfiles "Direct link to Differences between Pipe and INSERT+FILES()") A Pipe job is split into one or more transactions based on the size and number of rows in each data file. Users can query the intermediate results during the loading process. In contrast, an INSERT+`FILES()` job is processed as a single transaction, and users are unable to view the data during the loading process. ##### File loading sequence[​](#file-loading-sequence "Direct link to File loading sequence") For each Pipe job, StarRocks maintains a file queue, from which it fetches and loads data files as micro-batches. Pipe does not ensure that the data files are loaded in the same order as they are uploaded. Therefore, newer data may be loaded prior to older data. ##### Typical example[​](#typical-example-1 "Direct link to Typical example") ###### Create a database and a table[​](#create-a-database-and-a-table-1 "Direct link to Create a database and a table") Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a table by hand (we recommend that the table have the same schema as the Parquet file you want to load from AWS S3): ```sql CREATE TABLE user_behavior_from_pipe ( UserID int(11), ItemID int(11), CategoryID int(11), BehaviorType varchar(65533), Timestamp datetime ) ENGINE = OLAP DUPLICATE KEY(UserID) DISTRIBUTED BY HASH(UserID); ``` ###### Start a Pipe job[​](#start-a-pipe-job "Direct link to Start a Pipe job") Run the following command to start a Pipe job that loads data from the sample dataset `s3://starrocks-examples/user-behavior-10-million-rows/` to the `user_behavior_from_pipe` table. This pipe job uses both micro batches, and continuous loading (described above) pipe-specific features. The other examples in this guide load a single Parquet file with 10 million rows. For the pipe example, the same dataset is split into 57 separate files, and these are all stored in one S3 folder. Note in the `CREATE PIPE` command below the `path` is the URI for an S3 folder and rather than providing a filename the URI ends in `/*`. By setting `AUTO_INGEST` and specifying a folder rather than an individual file the pipe job will poll the S3 folder for new files and ingest them as they are added to the folder. ```sql CREATE PIPE user_behavior_pipe PROPERTIES ( "AUTO_INGEST" = "TRUE" ) AS INSERT INTO user_behavior_from_pipe SELECT * FROM FILES ( "path" = "s3://starrocks-examples/user-behavior-10-million-rows/*", "format" = "parquet", "aws.s3.region" = "us-east-1", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB" ); ``` > **NOTE** > > Substitute your credentials for `AAA` and `BBB` in the above command. Any valid `aws.s3.access_key` and `aws.s3.secret_key` can be used, as the object is readable by any AWS authenticated user. This job has four main sections: * `pipe_name`: The name of the pipe. The pipe name must be unique within the database to which the pipe belongs. * `INSERT_SQL`: The INSERT INTO SELECT FROM FILES statement that is used to load data from the specified source data file to the destination table. * `PROPERTIES`: A set of optional parameters that specify how to execute the pipe. These include `AUTO_INGEST`, `POLL_INTERVAL`, `BATCH_SIZE`, and `BATCH_FILES`. Specify these properties in the `"key" = "value"` format. For detailed syntax and parameter descriptions, see [CREATE PIPE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/CREATE_PIPE.md). ###### Check load progress[​](#check-load-progress-2 "Direct link to Check load progress") * Query the progress of the Pipe job by using [SHOW PIPES](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/SHOW_PIPES.md) in the current database to which the Pipe job belongs. ```sql SHOW PIPES WHERE NAME = 'user_behavior_pipe' \G ``` The following result is returned: tip In the output shown below the pipe is in the `RUNNING` state. A pipe will stay in the `RUNNING` state until you manually stop it. The output also shows the number of files loaded (57) and the last time that a file was loaded. ```sql *************************** 1. row *************************** DATABASE_NAME: mydatabase PIPE_ID: 10476 PIPE_NAME: user_behavior_pipe STATE: RUNNING TABLE_NAME: mydatabase.user_behavior_from_pipe LOAD_STATUS: {"loadedFiles":57,"loadedBytes":295345637,"loadingFiles":0,"lastLoadedTime":"2024-02-28 22:14:19"} LAST_ERROR: NULL CREATED_TIME: 2024-02-28 22:13:41 1 row in set (0.02 sec) ``` * Query the progress of the Pipe job from the [`pipes`](https://docs.starrocks.io/docs/sql-reference/information_schema/pipes.md) view in the StarRocks Information Schema. ```sql SELECT * FROM information_schema.pipes WHERE pipe_name = 'user_behavior_replica' \G ``` The following result is returned: tip Some of the queries in this guide end in `\G` instead of a semicolon (`;`). This causes the MySQL client to output the results in vertical format. If you are using DBeaver or another client you may need to use a semicolon (`;`) rather than `\G`. ```sql *************************** 1. row *************************** DATABASE_NAME: mydatabase PIPE_ID: 10217 PIPE_NAME: user_behavior_replica STATE: RUNNING TABLE_NAME: mydatabase.user_behavior_replica LOAD_STATUS: {"loadedFiles":1,"loadedBytes":132251298,"loadingFiles":0,"lastLoadedTime":"2023-11-09 15:35:42"} LAST_ERROR: CREATED_TIME: 9891-01-15 07:51:45 1 row in set (0.01 sec) ``` ###### Check file status[​](#check-file-status "Direct link to Check file status") You can query the load status of the files loaded from the [`pipe_files`](https://docs.starrocks.io/docs/sql-reference/information_schema/pipe_files.md) view in the StarRocks Information Schema. ```sql SELECT * FROM information_schema.pipe_files WHERE pipe_name = 'user_behavior_replica' \G ``` The following result is returned: ```sql *************************** 1. row *************************** DATABASE_NAME: mydatabase PIPE_ID: 10217 PIPE_NAME: user_behavior_replica FILE_NAME: s3://starrocks-examples/user-behavior-10-million-rows.parquet FILE_VERSION: e29daa86b1120fea58ad0d047e671787-8 FILE_SIZE: 132251298 LAST_MODIFIED: 2023-11-06 13:25:17 LOAD_STATE: FINISHED STAGED_TIME: 2023-11-09 15:35:02 START_LOAD_TIME: 2023-11-09 15:35:03 FINISH_LOAD_TIME: 2023-11-09 15:35:42 ERROR_MSG: 1 row in set (0.03 sec) ``` ###### Manage Pipe jobs[​](#manage-pipe-jobs "Direct link to Manage Pipe jobs") You can alter, suspend or resume, drop, or query the pipes you have created and retry to load specific data files. For more information, see [ALTER PIPE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/ALTER_PIPE.md), [SUSPEND or RESUME PIPE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/SUSPEND_or_RESUME_PIPE.md), [DROP PIPE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/DROP_PIPE.md), [SHOW PIPES](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/SHOW_PIPES.md), and [RETRY FILE](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/pipe/RETRY_FILE.md). --- ### s3_compatible --- ### Load data using Spark connector (recommended) StarRocks provides a self-developed connector named StarRocks Connector for Apache Spark™ (Spark connector for short) to help you load data into a StarRocks table by using Spark. The basic principle is to accumulate the data and then load it all at a time into StarRocks through [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). The Spark connector is implemented based on Spark DataSource V2. A DataSource can be created by using Spark DataFrames or Spark SQL. And both batch and structured streaming modes are supported. > **NOTICE** > > Only users with the SELECT and INSERT privileges on a StarRocks table can load data into this table. You can follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant these privileges to a user. #### Version requirements[​](#version-requirements "Direct link to Version requirements") | Spark connector | Spark | StarRocks | Java | Scala | | --------------- | ------------------ | ------------- | ---- | ----- | | 1.1.2 | 3.2, 3.3, 3.4, 3.5 | 2.5 and later | 8 | 2.12 | | 1.1.1 | 3.2, 3.3, or 3.4 | 2.5 and later | 8 | 2.12 | | 1.1.0 | 3.2, 3.3, or 3.4 | 2.5 and later | 8 | 2.12 | > **NOTICE** > > * Please see [Upgrade Spark connector](#upgrade-spark-connector) for behavior changes among different versions of the Spark connector. > * The Spark connector does not provide MySQL JDBC driver since version 1.1.1, and you need import the driver to the spark classpath manually. You can find the driver on [MySQL site](https://dev.mysql.com/downloads/connector/j/) or [Maven Central](https://repo1.maven.org/maven2/mysql/mysql-connector-java/). #### Obtain Spark connector[​](#obtain-spark-connector "Direct link to Obtain Spark connector") You can obtain the Spark connector JAR file in the following ways: * Directly download the compiled Spark Connector JAR file. * Add the Spark connector as a dependency in your Maven project and then download the JAR file. * Compile the source code of the Spark Connector into a JAR file by yourself. The naming format of the Spark connector JAR file is `starrocks-spark-connector-${spark_version}_${scala_version}-${connector_version}.jar`. For example, if you install Spark 3.2 and Scala 2.12 in your environment and you want to use Spark connector 1.1.0, you can use `starrocks-spark-connector-3.2_2.12-1.1.0.jar`. > **NOTICE** > > In general, the latest version of the Spark connector only maintains compatibility with the three most recent versions of Spark. ##### Download the compiled Jar file[​](#download-the-compiled-jar-file "Direct link to Download the compiled Jar file") Directly download the corresponding version of the Spark connector JAR from the [Maven Central Repository](https://repo1.maven.org/maven2/com/starrocks). ##### Maven Dependency[​](#maven-dependency "Direct link to Maven Dependency") 1. In your Maven project's `pom.xml` file, add the Spark connector as a dependency according to the following format. Replace `spark_version`, `scala_version`, and `connector_version` with the respective versions. ```xml com.starrocks starrocks-spark-connector-${spark_version}_${scala_version} ${connector_version} ``` 2. For example, if the version of Spark in your environment is 3.2, the version of Scala is 2.12, and you choose Spark connector 1.1.0, you need to add the following dependency: ```xml com.starrocks starrocks-spark-connector-3.2_2.12 1.1.0 ``` ##### Compile by yourself[​](#compile-by-yourself "Direct link to Compile by yourself") 1. Download the [Spark connector package](https://github.com/StarRocks/starrocks-connector-for-apache-spark). 2. Execute the following command to compile the source code of Spark connector into a JAR file. Note that `spark_version` is replaced with the corresponding Spark version. ```bash sh build.sh ``` For example, if the Spark version in your environment is 3.2, you need to execute the following command: ```bash sh build.sh 3.2 ``` 3. Go to the `target/` directory to find the Spark connector JAR file, such as `starrocks-spark-connector-3.2_2.12-1.1.0-SNAPSHOT.jar` , generated upon compilation. > **NOTE** > > The name of Spark connector which is not formally released contains the `SNAPSHOT` suffix. #### Parameters[​](#parameters "Direct link to Parameters") ##### starrocks.fe.http.url[​](#starrocksfehttpurl "Direct link to starrocks.fe.http.url") **Required**: YES
**Default value**: None
**Description**: The HTTP URL of the FE in your StarRocks cluster. You can specify multiple URLs, which must be separated by a comma (,). Format: `:,:`. Since version 1.1.1, you can also add `http://` prefix to the URL, such as `http://:,http://:`. ##### starrocks.fe.jdbc.url[​](#starrocksfejdbcurl "Direct link to starrocks.fe.jdbc.url") **Required**: YES
**Default value**: None
**Description**: The address that is used to connect to the MySQL server of the FE. Format: `jdbc:mysql://:`. ##### starrocks.table.identifier[​](#starrockstableidentifier "Direct link to starrocks.table.identifier") **Required**: YES
**Default value**: None
**Description**: The name of the StarRocks table. Format: `.`. ##### starrocks.user[​](#starrocksuser "Direct link to starrocks.user") **Required**: YES
**Default value**: None
**Description**: The username of your StarRocks cluster account. The user needs the [SELECT and INSERT privileges](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) on the StarRocks table. ##### starrocks.password[​](#starrockspassword "Direct link to starrocks.password") **Required**: YES
**Default value**: None
**Description**: The password of your StarRocks cluster account. ##### starrocks.write.label.prefix[​](#starrockswritelabelprefix "Direct link to starrocks.write.label.prefix") **Required**: NO
**Default value**: spark-
**Description**: The label prefix used by Stream Load. ##### starrocks.write.enable.transaction-stream-load[​](#starrockswriteenabletransaction-stream-load "Direct link to starrocks.write.enable.transaction-stream-load") **Required**: NO
**Default value**: TRUE
**Description**: Whether to use [Stream Load transaction interface](https://docs.starrocks.io/docs/loading/Stream_Load_transaction_interface.md) to load data. It requires StarRocks v2.5 or later. This feature can load more data in a transaction with less memory usage, and improve performance.
**NOTICE:** Since 1.1.1, this parameter takes effect only when the value of `starrocks.write.max.retries` is non-positive because Stream Load transaction interface does not support retry. ##### starrocks.write.buffer.size[​](#starrockswritebuffersize "Direct link to starrocks.write.buffer.size") **Required**: NO
**Default value**: 104857600
**Description**: The maximum size of data that can be accumulated in memory before being sent to StarRocks at a time. Setting this parameter to a larger value can improve loading performance but may increase loading latency. ##### starrocks.write.buffer.rows[​](#starrockswritebufferrows "Direct link to starrocks.write.buffer.rows") **Required**: NO
**Default value**: Integer.MAX\_VALUE
**Description**: Supported since version 1.1.1. The maximum number of rows that can be accumulated in memory before being sent to StarRocks at a time. ##### starrocks.write.flush.interval.ms[​](#starrockswriteflushintervalms "Direct link to starrocks.write.flush.interval.ms") **Required**: NO
**Default value**: 300000
**Description**: The interval at which data is sent to StarRocks. This parameter is used to control the loading latency. ##### starrocks.write.max.retries[​](#starrockswritemaxretries "Direct link to starrocks.write.max.retries") **Required**: NO
**Default value**: 3
**Description**: Supported since version 1.1.1. The number of times that the connector retries to perform the Stream Load for the same batch of data if the load fails.
**NOTICE:** Because Stream Load transaction interface does not support retry. If this parameter is positive, the connector always use Stream Load interface and ignore the value of `starrocks.write.enable.transaction-stream-load`. ##### starrocks.write.retry.interval.ms[​](#starrockswriteretryintervalms "Direct link to starrocks.write.retry.interval.ms") **Required**: NO
**Default value**: 10000
**Description**: Supported since version 1.1.1. The interval to retry the Stream Load for the same batch of data if the load fails. ##### starrocks.columns[​](#starrockscolumns "Direct link to starrocks.columns") **Required**: NO
**Default value**: None
**Description**: The StarRocks table column into which you want to load data. You can specify multiple columns, which must be separated by commas (,), for example, `"col0,col1,col2"`. ##### starrocks.column.types[​](#starrockscolumntypes "Direct link to starrocks.column.types") **Required**: NO
**Default value**: None
**Description**: Supported since version 1.1.1. Customize the column data types for Spark instead of using the defaults inferred from the StarRocks table and the [default mapping](#data-type-mapping-between-spark-and-starrocks). The parameter value is a schema in DDL format same as the output of Spark [StructType#toDDL](https://github.com/apache/spark/blob/master/sql/api/src/main/scala/org/apache/spark/sql/types/StructType.scala#L449) , such as `col0 INT, col1 STRING, col2 BIGINT`. Note that you only need to specify columns that need customization. One use case is to load data into columns of [BITMAP](#load-data-into-columns-of-bitmap-type) or [HLL](#load-data-into-columns-of-hll-type) type. ##### starrocks.write.properties.\*[​](#starrockswriteproperties "Direct link to starrocks.write.properties.*") **Required**: NO
**Default value**: None
**Description**: The parameters that are used to control Stream Load behavior. For example, the parameter `starrocks.write.properties.format` specifies the format of the data to be loaded, such as CSV or JSON. For a list of supported parameters and their descriptions, see [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). ##### starrocks.write.properties.format[​](#starrockswritepropertiesformat "Direct link to starrocks.write.properties.format") **Required**: NO
**Default value**: CSV
**Description**: The file format based on which the Spark connector transforms each batch of data before the data is sent to StarRocks. Valid values: CSV and JSON. ##### starrocks.write.properties.row\_delimiter[​](#starrockswritepropertiesrow_delimiter "Direct link to starrocks.write.properties.row_delimiter") **Required**: NO
**Default value**: \n
**Description**: The row delimiter for CSV-formatted data. ##### starrocks.write.properties.column\_separator[​](#starrockswritepropertiescolumn_separator "Direct link to starrocks.write.properties.column_separator") **Required**: NO
**Default value**: \t
**Description**: The column separator for CSV-formatted data. ##### starrocks.write.properties.partial\_update[​](#starrockswritepropertiespartial_update "Direct link to starrocks.write.properties.partial_update") **Required**: NO
**Default value**: `FALSE`
**Description**: Whether to use partial updates. Valid values: `TRUE` and `FALSE`. Default value: `FALSE`, indicating to disable this feature. ##### starrocks.write.properties.partial\_update\_mode[​](#starrockswritepropertiespartial_update_mode "Direct link to starrocks.write.properties.partial_update_mode") **Required**: NO
**Default value**: `row`
**Description**: Specifies the mode for partial updates. Valid values: `row` and `column`. * The value `row` (default) means partial updates in row mode, which is more suitable for real-time updates with many columns and small batches. * The value `column` means partial updates in column mode, which is more suitable for batch updates with few columns and many rows. In such scenarios, enabling the column mode offers faster update speeds. For example, in a table with 100 columns, if only 10 columns (10% of the total) are updated for all rows, the update speed of the column mode is 10 times faster. ##### starrocks.write.num.partitions[​](#starrockswritenumpartitions "Direct link to starrocks.write.num.partitions") **Required**: NO
**Default value**: None
**Description**: The number of partitions into which Spark can write data in parallel. When the data volume is small, you can reduce the number of partitions to lower the loading concurrency and frequency. The default value for this parameter is determined by Spark. However, this method may cause Spark Shuffle cost. ##### starrocks.write.partition.columns[​](#starrockswritepartitioncolumns "Direct link to starrocks.write.partition.columns") **Required**: NO
**Default value**: None
**Description**: The partitioning columns in Spark. The parameter takes effect only when `starrocks.write.num.partitions` is specified. If this parameter is not specified, all columns being written are used for partitioning. ##### starrocks.timezone[​](#starrockstimezone "Direct link to starrocks.timezone") **Required**: NO
**Default value**: Default timezone of JVM
**Description**: Supported since 1.1.1. The timezone used to convert Spark `TimestampType` to StarRocks `DATETIME`. The default is the timezone of JVM returned by `ZoneId#systemDefault()`. The format can be a timezone name such as `Asia/Shanghai`, or a zone offset such as `+08:00`. #### Data type mapping between Spark and StarRocks[​](#data-type-mapping-between-spark-and-starrocks "Direct link to Data type mapping between Spark and StarRocks") * The default data type mapping is as follows: | Spark data type | StarRocks data type | | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | BooleanType | BOOLEAN | | ByteType | TINYINT | | ShortType | SMALLINT | | IntegerType | INT | | LongType | BIGINT | | StringType | LARGEINT | | FloatType | FLOAT | | DoubleType | DOUBLE | | DecimalType | DECIMAL | | StringType | CHAR | | StringType | VARCHAR | | StringType | STRING | | StringType | JSON | | DateType | DATE | | TimestampType | DATETIME | | ArrayType | ARRAY
**NOTE:**
**Supported since version 1.1.1**. For detailed steps, see [Load data into columns of ARRAY type](#load-data-into-columns-of-array-type). | * You can also customize the data type mapping. For example, a StarRocks table contains BITMAP and HLL columns, but Spark does not support the two data types. You need to customize the corresponding data types in Spark. For detailed steps, see load data into [BITMAP](#load-data-into-columns-of-bitmap-type) and [HLL](#load-data-into-columns-of-hll-type) columns. **BITMAP and HLL are supported since version 1.1.1**. #### Upgrade Spark connector[​](#upgrade-spark-connector "Direct link to Upgrade Spark connector") ##### Upgrade from version 1.1.0 to 1.1.1[​](#upgrade-from-version-110-to-111 "Direct link to Upgrade from version 1.1.0 to 1.1.1") * Since 1.1.1, the Spark connector does not provide `mysql-connector-java` which is the official JDBC driver for MySQL, because of the limitations of the GPL license used by `mysql-connector-java`. However, the Spark connector still needs the MySQL JDBC driver to connect to StarRocks for the table metadata, so you need to add the driver to the Spark classpath manually. You can find the driver on [MySQL site](https://dev.mysql.com/downloads/connector/j/) or [Maven Central](https://repo1.maven.org/maven2/mysql/mysql-connector-java/). * Since 1.1.1, the connector uses Stream Load interface by default rather than Stream Load transaction interface in version 1.1.0. If you still want to use Stream Load transaction interface, you can set the option `starrocks.write.max.retries` to `0`. Please see the description of `starrocks.write.enable.transaction-stream-load` and `starrocks.write.max.retries` for details. #### Examples[​](#examples "Direct link to Examples") The following examples show how to use the Spark connector to load data into a StarRocks table with Spark DataFrames or Spark SQL. The Spark DataFrames supports both Batch and Structured Streaming modes. For more examples, see [Spark Connector Examples](https://github.com/StarRocks/starrocks-connector-for-apache-spark/tree/main/src/test/java/com/starrocks/connector/spark/examples). ##### Preparations[​](#preparations "Direct link to Preparations") ###### Create a StarRocks table[​](#create-a-starrocks-table "Direct link to Create a StarRocks table") Create a database `test` and create a Primary Key table `score_board`. ```sql CREATE DATABASE `test`; CREATE TABLE `test`.`score_board` ( `id` int(11) NOT NULL COMMENT "", `name` varchar(65533) NULL DEFAULT "" COMMENT "", `score` int(11) NOT NULL DEFAULT "0" COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`id`) COMMENT "OLAP" DISTRIBUTED BY HASH(`id`); ``` ###### Network configuration[​](#network-configuration "Direct link to Network configuration") Ensure that the machine where Spark is located can access the FE nodes of the StarRocks cluster via the [`http_port`](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#http_port) (default: `8030`) and [`query_port`](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#query_port) (default: `9030`), and the BE nodes via the [`be_http_port`](https://docs.starrocks.io/docs/administration/management/BE_configuration.md#be_http_port) (default: `8040`). ###### Set up your Spark environment[​](#set-up-your-spark-environment "Direct link to Set up your Spark environment") Note that the following examples are run in Spark 3.2.4 and use `spark-shell`, `pyspark` and `spark-sql`. Before running the examples, make sure to place the Spark connector JAR file in the `$SPARK_HOME/jars` directory. ##### Load data with Spark DataFrames[​](#load-data-with-spark-dataframes "Direct link to Load data with Spark DataFrames") The following two examples explain how to load data with Spark DataFrames Batch or Structured Streaming mode. ###### Batch[​](#batch "Direct link to Batch") Construct data in memory and load data into the StarRocks table. 1. You can write the spark application using Scala or Python. For Scala, run the following code snippet in `spark-shell`: ```scala // 1. Create a DataFrame from a sequence. val data = Seq((1, "starrocks", 100), (2, "spark", 100)) val df = data.toDF("id", "name", "score") // 2. Write to StarRocks by configuring the format as "starrocks" and the following options. // You need to modify the options according your own environment. df.write.format("starrocks") .option("starrocks.fe.http.url", "127.0.0.1:8030") .option("starrocks.fe.jdbc.url", "jdbc:mysql://127.0.0.1:9030") .option("starrocks.table.identifier", "test.score_board") .option("starrocks.user", "root") .option("starrocks.password", "") .mode("append") .save() ``` For Python, run the following code snippet in `pyspark`: ```python from pyspark.sql import SparkSession spark = SparkSession \ .builder \ .appName("StarRocks Example") \ .getOrCreate() # 1. Create a DataFrame from a sequence. data = [(1, "starrocks", 100), (2, "spark", 100)] df = spark.sparkContext.parallelize(data) \ .toDF(["id", "name", "score"]) # 2. Write to StarRocks by configuring the format as "starrocks" and the following options. # You need to modify the options according your own environment. df.write.format("starrocks") \ .option("starrocks.fe.http.url", "127.0.0.1:8030") \ .option("starrocks.fe.jdbc.url", "jdbc:mysql://127.0.0.1:9030") \ .option("starrocks.table.identifier", "test.score_board") \ .option("starrocks.user", "root") \ .option("starrocks.password", "") \ .mode("append") \ .save() ``` 2. Query data in the StarRocks table. ```sql MySQL [test]> SELECT * FROM `score_board`; +------+-----------+-------+ | id | name | score | +------+-----------+-------+ | 1 | starrocks | 100 | | 2 | spark | 100 | +------+-----------+-------+ 2 rows in set (0.00 sec) ``` ###### Structured Streaming[​](#structured-streaming "Direct link to Structured Streaming") Construct a streaming read of data from a CSV file and load data into the StarRocks table. 1. In the directory `csv-data`, create a CSV file `test.csv` with the following data: ```csv 3,starrocks,100 4,spark,100 ``` 2. You can write the Spark application using Scala or Python. For Scala, run the following code snippet in `spark-shell`: ```scala import org.apache.spark.sql.types.StructType // 1. Create a DataFrame from CSV. val schema = (new StructType() .add("id", "integer") .add("name", "string") .add("score", "integer") ) val df = (spark.readStream .option("sep", ",") .schema(schema) .format("csv") // Replace it with your path to the directory "csv-data". .load("/path/to/csv-data") ) // 2. Write to StarRocks by configuring the format as "starrocks" and the following options. // You need to modify the options according your own environment. val query = (df.writeStream.format("starrocks") .option("starrocks.fe.http.url", "127.0.0.1:8030") .option("starrocks.fe.jdbc.url", "jdbc:mysql://127.0.0.1:9030") .option("starrocks.table.identifier", "test.score_board") .option("starrocks.user", "root") .option("starrocks.password", "") // replace it with your checkpoint directory .option("checkpointLocation", "/path/to/checkpoint") .outputMode("append") .start() ) ``` For Python, run the following code snippet in `pyspark`: ```python from pyspark.sql import SparkSession from pyspark.sql.types import IntegerType, StringType, StructType, StructField spark = SparkSession \ .builder \ .appName("StarRocks SS Example") \ .getOrCreate() # 1. Create a DataFrame from CSV. schema = StructType([ StructField("id", IntegerType()), StructField("name", StringType()), StructField("score", IntegerType()) ]) df = ( spark.readStream .option("sep", ",") .schema(schema) .format("csv") # Replace it with your path to the directory "csv-data". .load("/path/to/csv-data") ) # 2. Write to StarRocks by configuring the format as "starrocks" and the following options. # You need to modify the options according your own environment. query = ( df.writeStream.format("starrocks") .option("starrocks.fe.http.url", "127.0.0.1:8030") .option("starrocks.fe.jdbc.url", "jdbc:mysql://127.0.0.1:9030") .option("starrocks.table.identifier", "test.score_board") .option("starrocks.user", "root") .option("starrocks.password", "") # replace it with your checkpoint directory .option("checkpointLocation", "/path/to/checkpoint") .outputMode("append") .start() ) ``` 3. Query data in the StarRocks table. ```sql MySQL [test]> select * from score_board; +------+-----------+-------+ | id | name | score | +------+-----------+-------+ | 4 | spark | 100 | | 3 | starrocks | 100 | +------+-----------+-------+ 2 rows in set (0.67 sec) ``` ##### Load data with Spark SQL[​](#load-data-with-spark-sql "Direct link to Load data with Spark SQL") The following example explains how to load data with Spark SQL by using the `INSERT INTO` statement in the [Spark SQL CLI](https://spark.apache.org/docs/latest/sql-distributed-sql-engine-spark-sql-cli.html). 1. Execute the following SQL statement in the `spark-sql`: ```sql -- 1. Create a table by configuring the data source as `starrocks` and the following options. -- You need to modify the options according your own environment. CREATE TABLE `score_board` USING starrocks OPTIONS( "starrocks.fe.http.url"="127.0.0.1:8030", "starrocks.fe.jdbc.url"="jdbc:mysql://127.0.0.1:9030", "starrocks.table.identifier"="test.score_board", "starrocks.user"="root", "starrocks.password"="" ); -- 2. Insert two rows into the table. INSERT INTO `score_board` VALUES (5, "starrocks", 100), (6, "spark", 100); ``` 2. Query data in the StarRocks table. ```sql MySQL [test]> select * from score_board; +------+-----------+-------+ | id | name | score | +------+-----------+-------+ | 6 | spark | 100 | | 5 | starrocks | 100 | +------+-----------+-------+ 2 rows in set (0.00 sec) ``` #### Best Practices[​](#best-practices "Direct link to Best Practices") ##### Load data to Primary Key table[​](#load-data-to-primary-key-table "Direct link to Load data to Primary Key table") This section will show how to load data to StarRocks Primary Key table to achieve partial updates, and conditional updates. You can see [Change data through loading](https://docs.starrocks.io/docs/loading/Load_to_Primary_Key_tables.md) for the detailed introduction of these features. These examples use Spark SQL. ###### Preparations[​](#preparations-1 "Direct link to Preparations") Create a database `test` and create a Primary Key table `score_board` in StarRocks. ```sql CREATE DATABASE `test`; CREATE TABLE `test`.`score_board` ( `id` int(11) NOT NULL COMMENT "", `name` varchar(65533) NULL DEFAULT "" COMMENT "", `score` int(11) NOT NULL DEFAULT "0" COMMENT "" ) ENGINE=OLAP PRIMARY KEY(`id`) COMMENT "OLAP" DISTRIBUTED BY HASH(`id`); ``` ###### Partial updates[​](#partial-updates "Direct link to Partial updates") This example will show how to only update data in the column `name` through loading: 1. Insert initial data to StarRocks table in MySQL client. ```sql mysql> INSERT INTO `score_board` VALUES (1, 'starrocks', 100), (2, 'spark', 100); mysql> select * from score_board; +------+-----------+-------+ | id | name | score | +------+-----------+-------+ | 1 | starrocks | 100 | | 2 | spark | 100 | +------+-----------+-------+ 2 rows in set (0.02 sec) ``` 2. Create a Spark table `score_board` in Spark SQL client. * Set the option `starrocks.write.properties.partial_update` to `true` which tells the connector to do partial update. * Set the option `starrocks.columns` to `"id,name"` to tell the connector which columns to write. ```sql CREATE TABLE `score_board` USING starrocks OPTIONS( "starrocks.fe.http.url"="127.0.0.1:8030", "starrocks.fe.jdbc.url"="jdbc:mysql://127.0.0.1:9030", "starrocks.table.identifier"="test.score_board", "starrocks.user"="root", "starrocks.password"="", "starrocks.write.properties.partial_update"="true", "starrocks.columns"="id,name" ); ``` 3. Insert data into the table in Spark SQL client, and only update the column `name`. ```sql INSERT INTO `score_board` VALUES (1, 'starrocks-update'), (2, 'spark-update'); ``` 4. Query the StarRocks table in MySQL client. You can see that only values for `name` change, and the values for `score` does not change. ```sql mysql> select * from score_board; +------+------------------+-------+ | id | name | score | +------+------------------+-------+ | 1 | starrocks-update | 100 | | 2 | spark-update | 100 | +------+------------------+-------+ 2 rows in set (0.02 sec) ``` ###### Conditional updates[​](#conditional-updates "Direct link to Conditional updates") This example will show how to do conditional updates according to the values of column `score`. The update for an `id` takes effect only when the new value for `score` is has a greater or equal to the old value. 1. Insert initial data to StarRocks table in MySQL client. ```sql mysql> INSERT INTO `score_board` VALUES (1, 'starrocks', 100), (2, 'spark', 100); mysql> select * from score_board; +------+-----------+-------+ | id | name | score | +------+-----------+-------+ | 1 | starrocks | 100 | | 2 | spark | 100 | +------+-----------+-------+ 2 rows in set (0.02 sec) ``` 2. Create a Spark table `score_board` in the following ways. * Set the option `starrocks.write.properties.merge_condition` to `score` which tells the connector to use the column `score` as the condition. * Make sure that the Spark connector use Stream Load interface to load data, rather than Stream Load transaction interface, because the latter does not support this feature. ```sql CREATE TABLE `score_board` USING starrocks OPTIONS( "starrocks.fe.http.url"="127.0.0.1:8030", "starrocks.fe.jdbc.url"="jdbc:mysql://127.0.0.1:9030", "starrocks.table.identifier"="test.score_board", "starrocks.user"="root", "starrocks.password"="", "starrocks.write.properties.merge_condition"="score" ); ``` 3. Insert data to the table in Spark SQL client, and update the row whose `id` is 1 with a smaller score value, and the row whose `id` is 2 with a larger score value. ```sql INSERT INTO `score_board` VALUES (1, 'starrocks-update', 99), (2, 'spark-update', 101); ``` 4. Query the StarRocks table in MySQL client. You can see that only the row whose `id` is 2 changes, and the row whose `id` is 1 does not change. ```sql mysql> select * from score_board; +------+--------------+-------+ | id | name | score | +------+--------------+-------+ | 1 | starrocks | 100 | | 2 | spark-update | 101 | +------+--------------+-------+ 2 rows in set (0.03 sec) ``` ##### Load data into columns of BITMAP type[​](#load-data-into-columns-of-bitmap-type "Direct link to Load data into columns of BITMAP type") [`BITMAP`](https://docs.starrocks.io/docs/sql-reference/data-types/other-data-types/BITMAP.md) is often used to accelerate count distinct, such as counting UV, see [Use Bitmap for exact Count Distinct](https://docs.starrocks.io/docs/using_starrocks/distinct_values/Using_bitmap.md). Here we take the counting of UV as an example to show how to load data into columns of the `BITMAP` type. **`BITMAP` is supported since version 1.1.1**. 1. Create a StarRocks Aggregate table. In the database `test`, create an Aggregate table `page_uv` where the column `visit_users` is defined as the `BITMAP` type and configured with the aggregate function `BITMAP_UNION`. ```sql CREATE TABLE `test`.`page_uv` ( `page_id` INT NOT NULL COMMENT 'page ID', `visit_date` datetime NOT NULL COMMENT 'access time', `visit_users` BITMAP BITMAP_UNION NOT NULL COMMENT 'user ID' ) ENGINE=OLAP AGGREGATE KEY(`page_id`, `visit_date`) DISTRIBUTED BY HASH(`page_id`); ``` 2. Create a Spark table. The schema of the Spark table is inferred from the StarRocks table, and the Spark does not support the `BITMAP` type. So you need to customize the corresponding column data type in Spark, for example as `BIGINT`, by configuring the option `"starrocks.column.types"="visit_users BIGINT"`. When using Stream Load to ingest data, the connector uses the [`to_bitmap`](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/to_bitmap.md) function to convert the data of `BIGINT` type into `BITMAP` type. Run the following DDL in `spark-sql`: ```sql CREATE TABLE `page_uv` USING starrocks OPTIONS( "starrocks.fe.http.url"="127.0.0.1:8030", "starrocks.fe.jdbc.url"="jdbc:mysql://127.0.0.1:9030", "starrocks.table.identifier"="test.page_uv", "starrocks.user"="root", "starrocks.password"="", "starrocks.column.types"="visit_users BIGINT" ); ``` 3. Load data into StarRocks table. Run the following DML in `spark-sql`: ```sql INSERT INTO `page_uv` VALUES (1, CAST('2020-06-23 01:30:30' AS TIMESTAMP), 13), (1, CAST('2020-06-23 01:30:30' AS TIMESTAMP), 23), (1, CAST('2020-06-23 01:30:30' AS TIMESTAMP), 33), (1, CAST('2020-06-23 02:30:30' AS TIMESTAMP), 13), (2, CAST('2020-06-23 01:30:30' AS TIMESTAMP), 23); ``` 4. Calculate page UVs from the StarRocks table. ```sql MySQL [test]> SELECT `page_id`, COUNT(DISTINCT `visit_users`) FROM `page_uv` GROUP BY `page_id`; +---------+-----------------------------+ | page_id | count(DISTINCT visit_users) | +---------+-----------------------------+ | 2 | 1 | | 1 | 3 | +---------+-----------------------------+ 2 rows in set (0.01 sec) ``` > **NOTICE:** > > The connector uses [`to_bitmap`](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/to_bitmap.md) function to convert data of the `TINYINT`, `SMALLINT`, `INTEGER`, and `BIGINT` types in Spark to the `BITMAP` type in StarRocks, and uses [`bitmap_hash`](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_hash.md) or [`bitmap_hash64`](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_hash64.md) function for other Spark data types. ##### Load data into columns of HLL type[​](#load-data-into-columns-of-hll-type "Direct link to Load data into columns of HLL type") [`HLL`](https://docs.starrocks.io/docs/sql-reference/data-types/other-data-types/HLL.md) can be used for approximate count distinct, see [Use HLL for approximate count distinct](https://docs.starrocks.io/docs/using_starrocks/distinct_values/Using_HLL.md). Here we take the counting of UV as an example to show how to load data into columns of the `HLL` type. **`HLL` is supported since version 1.1.1**. 1. Create a StarRocks Aggregate table. In the database `test`, create an Aggregate table `hll_uv` where the column `visit_users` is defined as the `HLL` type and configured with the aggregate function `HLL_UNION`. ```sql CREATE TABLE `hll_uv` ( `page_id` INT NOT NULL COMMENT 'page ID', `visit_date` datetime NOT NULL COMMENT 'access time', `visit_users` HLL HLL_UNION NOT NULL COMMENT 'user ID' ) ENGINE=OLAP AGGREGATE KEY(`page_id`, `visit_date`) DISTRIBUTED BY HASH(`page_id`); ``` 2. Create a Spark table. The schema of the Spark table is inferred from the StarRocks table, and the Spark does not support the `HLL` type. So you need to customize the corresponding column data type in Spark, for example as `BIGINT`, by configuring the option `"starrocks.column.types"="visit_users BIGINT"`. When using Stream Load to ingest data, the connector uses the [`hll_hash`](https://docs.starrocks.io/docs/sql-reference/sql-functions/scalar-functions/hll_hash.md) function to convert the data of `BIGINT` type into `HLL` type. Run the following DDL in `spark-sql`: ```sql CREATE TABLE `hll_uv` USING starrocks OPTIONS( "starrocks.fe.http.url"="127.0.0.1:8030", "starrocks.fe.jdbc.url"="jdbc:mysql://127.0.0.1:9030", "starrocks.table.identifier"="test.hll_uv", "starrocks.user"="root", "starrocks.password"="", "starrocks.column.types"="visit_users BIGINT" ); ``` 3. Load data into StarRocks table. Run the following DML in `spark-sql`: ```sql INSERT INTO `hll_uv` VALUES (3, CAST('2023-07-24 12:00:00' AS TIMESTAMP), 78), (4, CAST('2023-07-24 13:20:10' AS TIMESTAMP), 2), (3, CAST('2023-07-24 12:30:00' AS TIMESTAMP), 674); ``` 4. Calculate page UVs from the StarRocks table. ```sql MySQL [test]> SELECT `page_id`, COUNT(DISTINCT `visit_users`) FROM `hll_uv` GROUP BY `page_id`; +---------+-----------------------------+ | page_id | count(DISTINCT visit_users) | +---------+-----------------------------+ | 4 | 1 | | 3 | 2 | +---------+-----------------------------+ 2 rows in set (0.01 sec) ``` ##### Load data into columns of ARRAY type[​](#load-data-into-columns-of-array-type "Direct link to Load data into columns of ARRAY type") The following example explains how to load data into columns of the [`ARRAY`](https://docs.starrocks.io/docs/sql-reference/data-types/semi_structured/Array.md) type. 1. Create a StarRocks table. In the database `test`, create a Primary Key table `array_tbl` that includes one `INT` column and two `ARRAY` columns. ```sql CREATE TABLE `array_tbl` ( `id` INT NOT NULL, `a0` ARRAY, `a1` ARRAY> ) ENGINE=OLAP PRIMARY KEY(`id`) DISTRIBUTED BY HASH(`id`) ; ``` 2. Write data to StarRocks. Because some versions of StarRocks does not provide the metadata of `ARRAY` column, the connector can not infer the corresponding Spark data type for this column. However, you can explicitly specify the corresponding Spark data type of the column in the option `starrocks.column.types`. In this example, you can configure the option as `a0 ARRAY,a1 ARRAY>`. Run the following codes in `spark-shell`: ```scala val data = Seq( | (1, Seq("hello", "starrocks"), Seq(Seq(1, 2), Seq(3, 4))), | (2, Seq("hello", "spark"), Seq(Seq(5, 6, 7), Seq(8, 9, 10))) | ) val df = data.toDF("id", "a0", "a1") df.write .format("starrocks") .option("starrocks.fe.http.url", "127.0.0.1:8030") .option("starrocks.fe.jdbc.url", "jdbc:mysql://127.0.0.1:9030") .option("starrocks.table.identifier", "test.array_tbl") .option("starrocks.user", "root") .option("starrocks.password", "") .option("starrocks.column.types", "a0 ARRAY,a1 ARRAY>") .mode("append") .save() ``` 3. Query data in the StarRocks table. ```sql MySQL [test]> SELECT * FROM `array_tbl`; +------+-----------------------+--------------------+ | id | a0 | a1 | +------+-----------------------+--------------------+ | 1 | ["hello","starrocks"] | [[1,2],[3,4]] | | 2 | ["hello","spark"] | [[5,6,7],[8,9,10]] | +------+-----------------------+--------------------+ 2 rows in set (0.01 sec) ``` --- ### Load data in bulk using Spark Load This load uses external Apache Spark™ resources to pre-process imported data, which improves import performance and saves compute resources. It is mainly used for **initial migration** and **large data import** into StarRocks (data volume up to TB level). Spark load is an **asynchronous** import method that requires users to create Spark-type import jobs via the MySQL protocol and view the import results using `SHOW LOAD`. > **NOTICE** > > * Only users with the INSERT privilege on a StarRocks table can load data into this table. You can follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the required privilege. > * Spark Load can not be used to load data into a Primary Key table. #### Terminology explanation[​](#terminology-explanation "Direct link to Terminology explanation") * **Spark ETL**: Mainly responsible for ETL of data in the import process, including global dictionary construction (BITMAP type), partitioning, sorting, aggregation, etc. * **Broker**: Broker is an independent stateless process. It encapsulates the file system interface and provides StarRocks with the ability to read files from remote storage systems. * **Global Dictionary**: Saves the data structure that maps data from the original value to the encoded value. The original value can be any data type, while the encoded value is an integer. The global dictionary is mainly used in scenarios where exact count distinct is precomputed. #### Fundamentals[​](#fundamentals "Direct link to Fundamentals") The user submits a Spark type import job through the MySQL client;the FE records the metadata and returns the submission result. The execution of the spark load task is divided into the following main phases. 1. The user submits the spark load job to the FE. 2. The FE schedules the submission of the ETL task to the Apache Spark™ cluster for execution. 3. The Apache Spark™ cluster executes the ETL task that includes global dictionary construction (BITMAP type), partitioning, sorting, aggregation, etc. 4. After the ETL task is completed, the FE gets the data path of each preprocessed slice and schedules the relevant BE to execute the Push task. 5. The BE reads data through Broker process from HDFS and converts it into StarRocks storage format. > If you choose not to use Broker process, the BE reads data from HDFS directly. 6. The FE schedules the effective version and completes the import job. The following diagram illustrates the main flow of spark load. ![Spark load](/assets/images/4.3.2-1-7104f83d68f711e7dd819b5f11391eb7.png) *** #### Global Dictionary[​](#global-dictionary "Direct link to Global Dictionary") ##### Applicable Scenarios[​](#applicable-scenarios "Direct link to Applicable Scenarios") Currently, the BITMAP column in StarRocks is implemented using the Roaringbitmap, which only has integer to be the input data type. So if you want to implement precomputation for the BITMAP column in the import process, then you need to convert the input data type to integer. In the existing import process of StarRocks, the data structure of the global dictionary is implemented based on the Hive table, which saves the mapping from the original value to the encoded value. ##### Build Process[​](#build-process "Direct link to Build Process") 1. Read the data from the upstream data source and generate a temporary Hive table, named `hive-table`. 2. Extract the values of the de-emphasized fields of `hive-table` to generate a new Hive table named `distinct-value-table`. 3. Create a new global dictionary table named `dict-table` with one column for the original values and one column for the encoded values. 4. Left join between `distinct-value-table` and `dict-table`, and then use the window function to encode this set. Finally both the original value and the encoded value of the de-duplicated column are written back to `dict-table`. 5. Join between `dict-table` and `hive-table` to finish the job of replacing the original value in `hive-table` with the integer encoded value. 6. `hive-table` will be read by the next time data pre-processing, and then imported into StarRocks after calculation. #### Data Pre-processing[​](#data-pre-processing "Direct link to Data Pre-processing") The basic process of data pre-processing is as follows: 1. Read data from the upstream data source (HDFS file or Hive table). 2. Complete field mapping and calculation for the read data, then generate `bucket-id` based on the partition information. 3. Generate RollupTree based on the Rollup metadata of StarRocks table. 4. Iterate through the RollupTree and perform hierarchical aggregation operations. The Rollup of the next hierarchy can be calculated from the Rollup of the previous hierarchy. 5. Each time the aggregation calculation is completed, the data is bucketed according to `bucket-id` and then written to HDFS. 6. The subsequent Broker process will pull the files from HDFS and import them into the StarRocks BE node. #### Basic Operations[​](#basic-operations "Direct link to Basic Operations") ##### Configuring ETL Clusters[​](#configuring-etl-clusters "Direct link to Configuring ETL Clusters") Apache Spark™ is used as an external computational resource in StarRocks for ETL work. There may be other external resources added to StarRocks, such as Spark/GPU for query, HDFS/S3 for external storage, MapReduce for ETL, etc. Therefore, we introduce `Resource Management` to manage these external resources used by StarRocks. Before submitting a Apache Spark™ import job, configure the Apache Spark™ cluster for performing ETL tasks. The syntax for operation is as follows: ```sql -- create Apache Spark™ resource CREATE EXTERNAL RESOURCE resource_name PROPERTIES ( type = spark, spark_conf_key = spark_conf_value, working_dir = path, broker = broker_name, broker.property_key = property_value ); -- drop Apache Spark™ resource DROP RESOURCE resource_name; -- show resources SHOW RESOURCES SHOW PROC "/resources"; -- privileges GRANT USAGE_PRIV ON RESOURCE resource_name TO user_identityGRANT USAGE_PRIV ON RESOURCE resource_name TO ROLE role_name; REVOKE USAGE_PRIV ON RESOURCE resource_name FROM user_identityREVOKE USAGE_PRIV ON RESOURCE resource_name FROM ROLE role_name; ``` * Create resource **For example**: ```sql -- yarn cluster mode CREATE EXTERNAL RESOURCE "spark0" PROPERTIES ( "type" = "spark", "spark.master" = "yarn", "spark.submit.deployMode" = "cluster", "spark.jars" = "xxx.jar,yyy.jar", "spark.files" = "/tmp/aaa,/tmp/bbb", "spark.executor.memory" = "1g", "spark.yarn.queue" = "queue0", "spark.hadoop.yarn.resourcemanager.address" = "127.0.0.1:9999", "spark.hadoop.fs.defaultFS" = "hdfs://127.0.0.1:10000", "working_dir" = "hdfs://127.0.0.1:10000/tmp/starrocks", "broker" = "broker0", "broker.username" = "user0", "broker.password" = "password0" ); -- yarn HA cluster mode CREATE EXTERNAL RESOURCE "spark1" PROPERTIES ( "type" = "spark", "spark.master" = "yarn", "spark.submit.deployMode" = "cluster", "spark.hadoop.yarn.resourcemanager.ha.enabled" = "true", "spark.hadoop.yarn.resourcemanager.ha.rm-ids" = "rm1,rm2", "spark.hadoop.yarn.resourcemanager.hostname.rm1" = "host1", "spark.hadoop.yarn.resourcemanager.hostname.rm2" = "host2", "spark.hadoop.fs.defaultFS" = "hdfs://127.0.0.1:10000", "working_dir" = "hdfs://127.0.0.1:10000/tmp/starrocks", "broker" = "broker1" ); ``` `resource-name` is the name of the Apache Spark™ resource configured in StarRocks. `PROPERTIES` includes parameters relating to the Apache Spark™ resource, as follows: > **Note** > > For detailed description of Apache Spark™ resource PROPERTIES, please see [CREATE RESOURCE](https://docs.starrocks.io/docs/sql-reference/sql-statements/Resource/CREATE_RESOURCE.md) * Spark related parameters: * `type`: Resource type, required, currently only supports `spark`. * `spark.master`: Required, currently only supports `yarn`. * `spark.submit.deployMode`: The deployment mode of the Apache Spark™ program, required, currently supports both `cluster` and `client`. * `spark.hadoop.fs.defaultFS`: Required if master is yarn. * Parameters related to yarn resource manager, required. * one ResourceManager on a single node `spark.hadoop.yarn.resourcemanager.address`: Address of the single point resource manager. * ResourceManager HA > You can choose to specify ResourceManager's hostname or address. * `spark.hadoop.yarn.resourcemanager.ha.enabled`: Enable the resource manager HA, set to `true`. * `spark.hadoop.yarn.resourcemanager.ha.rm-ids`: list of resource manager logical ids. * `spark.hadoop.yarn.resourcemanager.hostname.rm-id`: For each rm-id, specify the hostname corresponding to the resource manager. * `spark.hadoop.yarn.resourcemanager.address.rm-id`: For each rm-id, specify `host:port` for the client to submit jobs to. * `*working_dir`: The directory used by ETL. Required if Apache Spark™ is used as an ETL resource. For example: `hdfs://host:port/tmp/starrocks`. * Broker related parameters: * `broker`: Broker name. Required if Apache Spark™ is used as an ETL resource. You need to use the `ALTER SYSTEM ADD BROKER` command to complete the configuration in advance. * `broker.property_key`: Information (e.g.authentication information) to be specified when Broker process reads the intermediate file generated by the ETL. **Precaution**: The above is a description of parameters for loading through Broker process. If you intend to load data without Broker process, the following should be noted. * You do not need to specify `broker`. * If you need to configure user authentication, and HA for NameNode nodes, you need to configure the parameters in the hdfs-site.xml file in the HDFS cluster, see [broker\_properties](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md#hdfs) for descriptions of parameters. and you need to move the **hdfs-site.xml** file under **$FE\_HOME/conf** for each FE and **$BE\_HOME/conf** for each BE. > Note > > If the HDFS file can only be accessed by a specific user, you still need to specify the HDFS username in `broker.name` and the user password in `broker.password`. * View resources Regular accounts can only view resources to which they have `USAGE-PRIV` access. The root and admin accounts can view all resources. * Resource Permissions Resource permissions are managed through `GRANT REVOKE`, which currently only supports `USAGE-PRIV` permissions. You can give `USAGE-PRIV` permissions to a user or a role. ```sql -- Grant access to spark0 resources to user0 GRANT USAGE_PRIV ON RESOURCE "spark0" TO "user0"@"%"; -- Grant access to spark0 resources to role0 GRANT USAGE_PRIV ON RESOURCE "spark0" TO ROLE "role0"; -- Grant access to all resources to user0 GRANT USAGE_PRIV ON RESOURCE* TO "user0"@"%"; -- Grant access to all resources to role0 GRANT USAGE_PRIV ON RESOURCE* TO ROLE "role0"; -- Revoke the use privileges of spark0 resources from user user0 REVOKE USAGE_PRIV ON RESOURCE "spark0" FROM "user0"@"%"; ``` ##### Configuring Spark Client[​](#configuring-spark-client "Direct link to Configuring Spark Client") Configure the Spark client for FE so that the latter can submit Spark tasks by executing the `spark-submit` command. It is recommended to use the official version of Spark2 2.4.5 or above [spark download address](https://archive.apache.org/dist/spark/). After downloading, please use the following steps to complete the configuration. * Configure `SPARK-HOME` Place the Spark client in a directory on the same machine as the FE, and configure `spark_home_default_dir` in the FE configuration file to this directory, which by default is the `lib/spark2x` path in the FE root directory, and cannot be empty. * **Configure SPARK dependency package** To configure the dependency package, zip and archive all jar files in the jars folder under the Spark client, and configure the `spark_resource_path` item in the FE configuration to this zip file. If this configuration is empty, the FE will try to find the `lib/spark2x/jars/spark-2x.zip` file in the FE root directory. If the FE fails to find it, it will report an error. When the spark load job is submitted, the archived dependency files will be uploaded to the remote repository. The default repository path is under the `working_dir/{cluster_id}` directory named with `--spark-repository--{resource-name}`, which means that a resource in the cluster corresponds to a remote repository. The directory structure is referenced as follows: ```bash ---spark-repository--spark0/ |---archive-1.0.0/ | |\---lib-990325d2c0d1d5e45bf675e54e44fb16-spark-dpp-1.0.0\-jar-with-dependencies.jar | |\---lib-7670c29daf535efe3c9b923f778f61fc-spark-2x.zip |---archive-1.1.0/ | |\---lib-64d5696f99c379af2bee28c1c84271d5-spark-dpp-1.1.0\-jar-with-dependencies.jar | |\---lib-1bbb74bb6b264a270bc7fca3e964160f-spark-2x.zip |---archive-1.2.0/ | |-... ``` In addition to the spark dependencies (named `spark-2x.zip` by default), the FE also uploads the DPP dependencies to the remote repository. If all the dependencies submitted by the spark load already exist in the remote repository, then there is no need to upload the dependencies again, saving the time of repeatedly uploading a large number of files each time. ##### Configuring YARN Client[​](#configuring-yarn-client "Direct link to Configuring YARN Client") Configure the yarn client for FE so that the FE can execute yarn commands to get the status of the running application or kill it.It is recommended to use the official version of Hadoop2 2.5.2 or above ([hadoop download address](https://archive.apache.org/dist/hadoop/common/)). After downloading, please use the following steps to complete the configuration: * **Configure the YARN executable path** Place the downloaded yarn client in a directory on the same machine as the FE, and configure the `yarn_client_path` item in the FE configuration file to the binary executable file of yarn, which by default is the `lib/yarn-client/hadoop/bin/yarn` path in the FE root directory. * **Configure the path to the configuration file needed to generate YARN (optional)** When the FE goes through the yarn client to get the status of the application, or to kill the application, by default StarRocks generates the configuration file required to execute the yarn command in the `lib/yarn-config` path of the FE root directory This path can be modified by configuring the `yarn_config_dir` entry in the FE configuration file, which currently includes `core-site.xml` and `yarn-site.xml`. ##### Create Import Job[​](#create-import-job "Direct link to Create Import Job") **Syntax:** ```sql LOAD LABEL load_label (data_desc, ...) WITH RESOURCE resource_name [resource_properties] [PROPERTIES (key1=value1, ... )] * load_label: db_name.label_name * data_desc: DATA INFILE ('file_path', ...) [NEGATIVE] INTO TABLE tbl_name [PARTITION (p1, p2)] [COLUMNS TERMINATED BY separator ] [(col1, ...)] [COLUMNS FROM PATH AS (col2, ...)] [SET (k1=f1(xx), k2=f2(xx))] [WHERE predicate] DATA FROM TABLE hive_external_tbl [NEGATIVE] INTO TABLE tbl_name [PARTITION (p1, p2)] [SET (k1=f1(xx), k2=f2(xx))] [WHERE predicate] * resource_properties: (key2=value2, ...) ``` **Example 1**: The case where the upstream data source is HDFS ```sql LOAD LABEL db1.label1 ( DATA INFILE("hdfs://abc.com:8888/user/starrocks/test/ml/file1") INTO TABLE tbl1 COLUMNS TERMINATED BY "," (tmp_c1,tmp_c2) SET ( id=tmp_c2, name=tmp_c1 ), DATA INFILE("hdfs://abc.com:8888/user/starrocks/test/ml/file2") INTO TABLE tbl2 COLUMNS TERMINATED BY "," (col1, col2) where col1 > 1 ) WITH RESOURCE 'spark0' ( "spark.executor.memory" = "2g", "spark.shuffle.compress" = "true" ) PROPERTIES ( "timeout" = "3600" ); ``` **Example 2**: The case where the upstream data source is Hive. * Step 1: Create a new hive resource ```sql CREATE EXTERNAL RESOURCE hive0 PROPERTIES ( "type" = "hive", "hive.metastore.uris" = "thrift://xx.xx.xx.xx:8080" ); ``` * Step 2: Create a new hive external table ```sql CREATE EXTERNAL TABLE hive_t1 ( k1 INT, K2 SMALLINT, k3 varchar(50), uuid varchar(100) ) ENGINE=hive PROPERTIES ( "resource" = "hive0", "database" = "tmp", "table" = "t1" ); ``` * Step 3: Submit the load command, requiring that the columns in the imported StarRocks table exist in the hive external table. ```sql LOAD LABEL db1.label1 ( DATA FROM TABLE hive_t1 INTO TABLE tbl1 SET ( uuid=bitmap_dict(uuid) ) ) WITH RESOURCE 'spark0' ( "spark.executor.memory" = "2g", "spark.shuffle.compress" = "true" ) PROPERTIES ( "timeout" = "3600" ); ``` Introduction to the parameters in the Spark load: * **Label** Label of the import job. Each import job has a Label that is unique within the database, following the same rules as broker load. * **Data description class parameters** Currently, supported data sources are CSV and Hive table. Other rules are the same as broker load. * **Import Job Parameters** Import job parameters refer to the parameters belonging to the `opt_properties` section of the import statement. These parameters are applicable to the entire import job. The rules are the same as broker load. * **Spark Resource Parameters** Spark resources need to be configured into StarRocks in advance and users need to be given USAGE-PRIV permissions before they can apply the resources to Spark load. Spark resource parameters can be set when the user has a temporary need, such as adding resources for a job and modifying Spark configs. The setting only takes effect on this job and does not affect the existing configurations in the StarRocks cluster. ```sql WITH RESOURCE 'spark0' ( "spark.driver.memory" = "1g", "spark.executor.memory" = "3g" ) ``` * **Import when the data source is Hive** Currently, to use a Hive table in the import process, you need to create an external table of the `Hive` type and then specify its name when submitting the import command. * **Import process to build a global dictionary** In the load command, you can specify the required fields for building the global dictionary in the following format: `StarRocks field name=bitmap_dict(hive table field name)` Note that currently **the global dictionary is only supported when the upstream data source is a Hive table**. * **Load binary type data** Since v2.5.17, Spark Load supports the bitmap\_from\_binary function, which can convert binary data into bitmap data. If the column type of the Hive table or HDFS file is binary and the corresponding column in the StarRocks table is a bitmap-type aggregate column, you can specify the fields in the load command in the following format, `StarRocks field name=bitmap_from_binary(Hive table field name)`. This eliminates the need for building a global dictionary. #### Viewing Import Jobs[​](#viewing-import-jobs "Direct link to Viewing Import Jobs") The Spark load import is asynchronous, as is the broker load. The user must record the label of the import job and use it in the `SHOW LOAD` command to view the import results. The command to view the import is common to all import methods. The example is as follows. Refer to Broker Load for a detailed explanation of returned parameters.The differences are as follows. ```sql mysql> show load order by createtime desc limit 1\G *************************** 1. row *************************** JobId: 76391 Label: label1 State: FINISHED Progress: ETL:100%; LOAD:100% Type: SPARK EtlInfo: unselected.rows=4; dpp.abnorm.ALL=15; dpp.norm.ALL=28133376 TaskInfo: cluster:cluster0; timeout(s):10800; max_filter_ratio:5.0E-5 ErrorMsg: N/A CreateTime: 2019-07-27 11:46:42 EtlStartTime: 2019-07-27 11:46:44 EtlFinishTime: 2019-07-27 11:49:44 LoadStartTime: 2019-07-27 11:49:44 LoadFinishTime: 2019-07-27 11:50:16 URL: http://1.1.1.1:8089/proxy/application_1586619723848_0035/ JobDetails: {"ScannedRows":28133395,"TaskNumber":1,"FileNumber":1,"FileSize":200000} ``` * **State** The current stage of the imported job. PENDING: The job is committed. ETL: Spark ETL is committed. LOADING: The FE schedule an BE to execute push operation. FINISHED: The push is completed and the version is effective. There are two final stages of the import job – `CANCELLED` and `FINISHED`, both indicating the load job is completed. `CANCELLED` indicates import failure and `FINISHED` indicates import success. * **Progress** Description of the import job progress. There are two types of progress –ETL and LOAD, which correspond to the two phases of the import process, ETL and LOADING. * The range of progress for LOAD is 0~100%. `LOAD progress = the number of currently completed tablets of all replications imports / the total number of tablets of this import job * 100%`. * If all tables have been imported, the LOAD progress is 99%, and changes to 100% when the import enters the final validation phase. * The import progress is not linear. If there is no change in progress for a period of time, it does not mean that the import is not executing. * **Type** The type of the import job. SPARK for spark load. * **CreateTime/EtlStartTime/EtlFinishTime/LoadStartTime/LoadFinishTime** These values represent the time when the import was created, when the ETL phase started, when the ETL phase completed, when the LOADING phase started, and when the entire import job was completed. * **JobDetails** Displays the detailed running status of the job, including the number of imported files, total size (in bytes), number of subtasks, number of raw rows being processed, etc. For example: ```json {"ScannedRows":139264,"TaskNumber":1,"FileNumber":1,"FileSize":940754064} ``` * **URL** You can copy the input to your browser to access the web interface of the corresponding application. ##### View Apache Spark™ Launcher commit logs[​](#view-apache-spark-launcher-commit-logs "Direct link to View Apache Spark™ Launcher commit logs") Sometimes users need to view the detailed logs generated during a Apache Spark™ job commit. By default, the logs are saved in the path `log/spark_launcher_log` in the FE root directory named as `spark-launcher-{load-job-id}-{label}.log`. The logs are saved in this directory for a period of time and will be erased when the import information in FE metadata is cleaned up. The default retention time is 3 days. ##### Cancel Import[​](#cancel-import "Direct link to Cancel Import") When the Spark load job status is not `CANCELLED` or `FINISHED`, it can be cancelled manually by the user by specifying the Label of the import job. *** #### Related System Configurations[​](#related-system-configurations "Direct link to Related System Configurations") **FE Configuration:** The following configuration is the system-level configuration of Spark load, which applies to all Spark load import jobs. The configuration values can be adjusted mainly by modifying `fe.conf`. * enable-spark-load: Enable Spark load and resource creation with a default value of false. * spark-load-default-timeout-second: The default timeout for the job is 259200 seconds (3 days). * spark-home-default-dir: The Spark client path (`fe/lib/spark2x`). * spark-resource-path: The path to the packaged S park dependency file (empty by default). * spark-launcher-log-dir: The directory where the commit log of the Spark client is stored (`fe/log/spark-launcher-log`). * yarn-client-path: The path to the yarn binary executable (`fe/lib/yarn-client/hadoop/bin/yarn`). * yarn-config-dir: Yarn's configuration file path (`fe/lib/yarn-config`). *** #### Best Practices[​](#best-practices "Direct link to Best Practices") The most suitable scenario for using Spark load is when the raw data is in the file system (HDFS) and the data volume is in the tens of GB to TB level. Use Stream Load or Broker Load for smaller data volumes. For the full spark load import example, refer to the demo on github: #### FAQs[​](#faqs "Direct link to FAQs") * `Error: When running with master 'yarn' either HADOOP-CONF-DIR or YARN-CONF-DIR must be set in the environment.` Using Spark Load without configuring the `HADOOP-CONF-DIR` environment variable in `spark-env.sh` of the Spark client. * `Error: Cannot run program "xxx/bin/spark-submit": error=2, No such file or directory` The `spark_home_default_dir` configuration item does not specify the Spark client root directory when using Spark Load. * `Error: File xxx/jars/spark-2x.zip does not exist.` The `spark-resource-path` configuration item does not point to the packed zip file when using Spark load. * `Error: yarn client does not exist in path: xxx/yarn-client/hadoop/bin/yarn` The yarn-client-path configuration item does not specify the yarn executable when using Spark load. * `ERROR: Cannot execute hadoop-yarn/bin/... /libexec/yarn-config.sh` When using Hadoop with CDH, you need to configure the `HADOOP_LIBEXEC_DIR` environment variable. Since `hadoop-yarn` and hadoop directories are different, the default `libexec` directory will look for `hadoop-yarn/bin/... /libexec`, while `libexec` is in the hadoop directory. The \`\`\`yarn application status\`\` command to get the Spark task status reported an error causing the import job to fail. --- ### SQL Transaction Beta feature [Advice on use of Beta features](https://docs.starrocks.io/docs/introduction/maturity.md) Start a simple SQL transaction to commit multiple DML statements in a batch. #### Overview[​](#overview "Direct link to Overview") From v3.5.0, StarRocks supports SQL transactions to assure the atomicity of the updated tables when manipulating data within multiple tables. A transaction consists of multiple SQL statements that are processed within the same atomic unit. The statements in the transaction are either applied or undone together, thus guaranteeing the ACID (atomicity, consistency, isolation, and durability) properties of the transaction. Currently, the SQL transaction in StarRocks supports the following operations: * INSERT INTO * UPDATE * DELETE note * INSERT OVERWRITE is not supported currently. * Multiple INSERT statements against the same table within a transaction are supported only in shared-data clusters from v4.0 onwards. * UPDATE and DELETE are supported only in shared-data clusters from v4.0 onwards. From v4.0 onwards, within one SQL transaction: * **Multiple INSERT statements** against the one table are supported. * **Only one UPDATE *OR* DELETE** statement against one table is allowed. * **An UPDATE *OR* DELETE** statement **after** INSERT statements against the same table is **not allowed**. The ACID properties of the transaction are guaranteed only on the limited READ COMMITTED isolation level, that is: * A statement operates only on data that was committed before the statement began. * Two successive statements within the same transaction may operate on different data if another transaction is committed between the execution of the first and the second statements. * Data changes brought by preceding DML statements are invisible to subsequent statements within the same transaction. A transaction is associated with a single session. Multiple sessions cannot share the same transaction. #### Usage[​](#usage "Direct link to Usage") 1. A transaction must be started by executing a START TRANSACTION statement. StarRocks also supports the synonym BEGIN. ```sql { START TRANSACTION | BEGIN [ WORK ] } ``` 2. After starting the transaction, you can define multiple DML statements in the transaction. For detailed information, see [Usage notes](#usage-notes). 3. A transaction must be ended explicitly by executing `COMMIT` or `ROLLBACK`. * To apply (commit) the transaction, use the following syntax: ```sql COMMIT [ WORK ] ``` * To undo (roll back) the transaction, use the following syntax: ```sql ROLLBACK [ WORK ] ``` #### Example[​](#example "Direct link to Example") 1. Create the demo table `desT` in a shared-data cluster, and load data into it. note If you want to try this example in a shared-nothing cluster, you must skip Step 3 and define only one INSERT statement in Step 4. ```sql CREATE TABLE desT ( k int, v int ) PRIMARY KEY(k); INSERT INTO desT VALUES (1,1), (2,2), (3,3); ``` 2. Start a transaction. ```sql START TRANSACTION; ``` Or ```sql BEGIN WORK; ``` 3. Define an UPDATE or DELETE statement. ```sql UPDATE desT SET v = v + 1 WHERE k = 1, ``` Or ```sql DELETE FROM desT WHERE k = 1; ``` 4. Define multiple INSERT statements. ```sql -- Insert data with specified values. INSERT INTO desT VALUES (4,4); -- Insert data from a native table to another. INSERT INTO desT SELECT * FROM srcT; -- Insert data from remote storage. INSERT INTO desT SELECT * FROM FILES( "path" = "s3://inserttest/parquet/srcT.parquet", "format" = "parquet", "aws.s3.access_key" = "XXXXXXXXXX", "aws.s3.secret_key" = "YYYYYYYYYY", "aws.s3.region" = "us-west-2" ); ``` 5. Apply or undo the transaction. * To apply the SQL statements in the transaction. ```sql COMMIT WORK; ``` * To undo the SQL statements in the transaction. ```sql ROLLBACK WORK; ``` #### Usage notes[​](#usage-notes "Direct link to Usage notes") * Currently, StarRocks supports SELECT, INSERT, UPDATE, and DELETE statements in SQL transactions. UPDATE and DELETE are supported only in shared-data clusters from v4.0 onwards. * SELECT statements against the tables whose data have been changed in the same transaction are not allowed. * Multiple INSERT statements against the same table within a transaction are supported only in shared-data clusters from v4.0 onwards. * Within a transaction, you can only define one UPDATE or DELETE statement against each table, and it must precede the INSERT statements. * Subsequent DML statements cannot read the uncommitted changes brought by preceding statements within the same transaction. For example, the target table of the preceding INSERT statement cannot be the source table of subsequent statements. Otherwise, the system returns an error. * For the same reason, a partial column update (an INSERT that writes only a subset of a Primary Key table's columns) against a table that was already modified earlier in the same transaction is not allowed. A partial update must implicitly read the table's other columns to complete each row, which would read the preceding statements' uncommitted changes. Otherwise, the system returns an error. * All target tables of the DML statements in a transaction must be within the same database. Cross-database operations are not allowed. * Currently, INSERT OVERWRITE is not supported. * Nesting transactions are not allowed. You cannot specify BEGIN WORK within a BEGIN-COMMIT/ROLLBACK pair. * If the session where an on-going transaction belongs is terminated or closed, the transaction is automatically rolled back. * StarRock only supports limited READ COMMITTED for Transaction Isolation Level as described above. * Write conflict checks are not supported. When two transactions write to the same table simultaneously, both transactions can be committed successfully. The visibility (order) of the data changes depends on the execution order of the COMMIT WORK statements. --- ### Load data using Stream Load transaction interface From v2.4 onwards, StarRocks provides a Stream Load transaction interface to implement two-phase commit (2PC) for transactions that are run to load data from external systems such as Apache Flink® and Apache Kafka®. The Stream Load transaction interface helps improve the performance of highly concurrent stream loads. From v4.0 onwards, the Stream Load transaction interface supports Multi-table Transaction, that is, loading data into multiple tables within the same database. This topic describes the Stream Load transaction interface and how to load data into StarRocks by using this interface. #### Description[​](#description "Direct link to Description") The Stream Load transaction interface supports using an HTTP protocol-compatible tool or language to call API operations. This topic uses curl as an example to explain how to use this interface. This interface provides various features, such as transaction management, data write, transaction pre-commit, transaction deduplication, and transaction timeout management. note Stream Load supports CSV and JSON file formats. This method is recommended if you want to load data from a small number of files whose individual sizes do not exceed 10 GB. Stream Load does not support Parquet file format. If you need to load data from Parquet files, use [INSERT+files()](https://docs.starrocks.io/docs/loading/InsertInto.md#insert-data-directly-from-files-in-an-external-source-using-files). ##### Transaction management[​](#transaction-management "Direct link to Transaction management") The Stream Load transaction interface provides the following API operations, which are used to manage transactions: * `/api/transaction/begin`: starts a new transaction. * `/api/transaction/prepare`: pre-commits the current transaction and make data changes temporarily persistent. After you pre-commit a transaction, you can proceed to commit or roll back the transaction. If your cluster crashes after a transaction is pre-committed, you can still proceed to commit the transaction after the cluster is restored. * `/api/transaction/commit`: commits the current transaction to make data changes persistent. * `/api/transaction/rollback`: rolls back the current transaction to abort data changes. > **NOTE** > > After the transaction is pre-committed, do not continue to write data using the transaction. If you continue to write data using the transaction, your write request returns errors. The following diagram shows the relationship between transaction states and operations: ```mermaid stateDiagram-v2 direction LR [*] --> PREPARE : begin PREPARE --> PREPARED : prepare PREPARE --> ABORTED : rollback PREPARED --> COMMITTED : commit PREPARED --> ABORTED : rollback ``` ##### Data write[​](#data-write "Direct link to Data write") The Stream Load transaction interface provides the `/api/transaction/load` operation, which is used to write data. You can call this operation multiple times within one transaction. From v4.0 onwards, you can call `/api/transaction/load` operations on different tables to load data into multiple tables within the same database. ##### Transaction deduplication[​](#transaction-deduplication "Direct link to Transaction deduplication") The Stream Load transaction interface carries over the labeling mechanism of StarRocks. You can bind a unique label to each transaction to achieve at-most-once guarantees for transactions. ##### Transaction timeout management[​](#transaction-timeout-management "Direct link to Transaction timeout management") When you begin a transaction, you can use the `timeout` field in the HTTP request header to specify a timeout period (in seconds) for the transaction from `PREPARE` to `PREPARED` state. If the transaction has not been prepared after this period, it will be automatically aborted. If this field is not specified, the default value is determined by the FE configuration [`stream_load_default_timeout_second`](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#stream_load_default_timeout_second) (Default: 600 seconds). When you begin a transaction, you can also use the `idle_transaction_timeout` field in the HTTP request header to specify a timeout period (in seconds) within which the transaction can stay idle. If no data is written within this period, the transaction will be automatically rolled back. When you prepare a transaction, you can use the `prepared_timeout` field in the HTTP request header to specify a timeout period (in seconds) for the transaction from `PREPARED` to `COMMITTED` state. If the transaction has not been committed after this period, it will be automatically aborted. If this field is not specified, the default value is determined by the FE configuration [`prepared_transaction_default_timeout_second`](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#prepared_transaction_default_timeout_second) (Default: 86400 seconds). `prepared_timeout` is supported from v3.5.4 onwards. #### Benefits[​](#benefits "Direct link to Benefits") The Stream Load transaction interface brings the following benefits: * **Exactly-once semantics** A transaction is split into two phases, pre-commit and commit, which make it easy to load data across systems. For example, this interface can guarantee exactly-once semantics for data loads from Flink. * **Improved load performance** If you run a load job by using a program, the Stream Load transaction interface allows you to merge multiple mini-batches of data on demand and then send them all at once within one transaction by calling the `/api/transaction/commit` operation. As such, fewer data versions need to be loaded, and load performance is improved. #### Limits[​](#limits "Direct link to Limits") The Stream Load transaction interface has the following limits: * **Single-database multi-table** transactions are supported from v4.0 onwards. Support for **multi-database multi-table** transactions is in development. * Only **concurrent data writes from one client** are supported. Support for **concurrent data writes from multiple clients** is in development. * The `/api/transaction/load` operation can be called multiple times within one transaction. In this case, the parameter settings (except `table`) specified for all of the `/api/transaction/load` operations that are called must be the same. * When you load CSV-formatted data by using the Stream Load transaction interface, make sure that each data record in your data file ends with a row delimiter. #### Precautions[​](#precautions "Direct link to Precautions") * If the `/api/transaction/begin`, `/api/transaction/load`, or `/api/transaction/prepare` operation that you have called returns errors, the transaction fails and is automatically rolled back. * When calling the `/api/transaction/begin` operation to start a new transaction, you must specify a label. Note that the subsequent `/api/transaction/load`, `/api/transaction/prepare`, and `/api/transaction/commit` operations must use the same label as the `/api/transaction/begin` operation. * If you use the label of an ongoing transaction to call the `/api/transaction/begin` operation to start a new transaction, the previous transaction will fail and be rolled back. * If you use a multi-table transaction to load data into different tables, you must specify the parameter `-H "transaction_type:multi"` for all operations involved in the transaction. * The default column separator and row delimiter that StarRocks supports for CSV-formatted data are `\t` and `\n`. If your data file does not use the default column separator or row delimiter, you must use `"column_separator: "` or `"row_delimiter: "` to specify the column separator or row delimiter that is actually used in your data file when calling the `/api/transaction/load` operation. #### Before you begin[​](#before-you-begin "Direct link to Before you begin") ##### Check privileges[​](#check-privileges "Direct link to Check privileges") You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. ###### Check network configuration[​](#check-network-configuration "Direct link to Check network configuration") Make sure that the machine on which the data you want to load resides can access the FE and BE nodes of the StarRocks cluster via the [`http_port`](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#http_port) (default: `8030`) and [`be_http_port`](https://docs.starrocks.io/docs/administration/management/BE_configuration.md#be_http_port) (default: `8040`) , respectively. #### Basic operations[​](#basic-operations "Direct link to Basic operations") ##### Prepare sample data[​](#prepare-sample-data "Direct link to Prepare sample data") This topic uses CSV-formatted data as an example. 1. In the `/home/disk1/` path of your local file system, create a CSV file named `example1.csv`. The file consists of three columns, which represent the user ID, user name, and user score in sequence. ```plain 1,Lily,23 2,Rose,23 3,Alice,24 4,Julia,25 ``` 2. In your StarRocks database `test_db`, create a Primary Key table named `table1`. The table consists of three columns: `id`, `name`, and `score`, of which `id` is the primary key. ```sql CREATE TABLE `table1` ( `id` int(11) NOT NULL COMMENT "user ID", `name` varchar(65533) NULL COMMENT "user name", `score` int(11) NOT NULL COMMENT "user score" ) ENGINE=OLAP PRIMARY KEY(`id`) DISTRIBUTED BY HASH(`id`) BUCKETS 10; ``` ##### Start a transaction[​](#start-a-transaction "Direct link to Start a transaction") ###### Syntax[​](#syntax "Direct link to Syntax") ```bash curl --location-trusted -u : -H "label:" \ -H "Expect:100-continue" \ [-H "transaction_type:multi"]\ # Optional. Initiates a multi-table transaction. -H "db:" -H "table:" \ -XPOST http://:/api/transaction/begin ``` > **NOTE** > > Specify `-H "transaction_type:multi"` in the command if you want to load data into different tables within the transaction. ###### Example[​](#example "Direct link to Example") ```bash curl --location-trusted -u :<123456> -H "label:streamload_txn_example1_table1" \ -H "Expect:100-continue" \ -H "db:test_db" -H "table:table1" \ -XPOST http://:/api/transaction/begin ``` > **NOTE** > > For this example, `streamload_txn_example1_table1` is specified as the label of the transaction. ###### Return result[​](#return-result "Direct link to Return result") * If the transaction is successfully started, the following result is returned: ```bash { "Status": "OK", "Message": "", "Label": "streamload_txn_example1_table1", "TxnId": 9032, "BeginTxnTimeMs": 0 } ``` * If the transaction is bound to a duplicate label, the following result is returned: ```bash { "Status": "LABEL_ALREADY_EXISTS", "ExistingJobStatus": "RUNNING", "Message": "Label [streamload_txn_example1_table1] has already been used." } ``` * If errors other than duplicate label occur, the following result is returned: ```bash { "Status": "FAILED", "Message": "" } ``` ##### Write data[​](#write-data "Direct link to Write data") ###### Syntax[​](#syntax-1 "Direct link to Syntax") ```bash curl --location-trusted -u : -H "label:" \ -H "Expect:100-continue" \ [-H "transaction_type:multi"]\ # Optional. Loads data via a multi-table transaction. -H "db:" -H "table:" \ -T \ -XPUT http://:/api/transaction/load ``` > **NOTE** > > * When calling the `/api/transaction/load` operation, you must use `` to specify the save path of the data file you want to load. > * You can call `/api/transaction/load` operations with different `table` parameter values to load data into different tables within the same database. In this case, you must specify `-H "transaction_type:multi"` in the command. ###### Example[​](#example-1 "Direct link to Example") ```bash curl --location-trusted -u :<123456> -H "label:streamload_txn_example1_table1" \ -H "Expect:100-continue" \ -H "db:test_db" -H "table:table1" \ -T /home/disk1/example1.csv \ -H "column_separator: ," \ -XPUT http://:/api/transaction/load ``` > **NOTE** > > For this example, the column separator used in the data file `example1.csv` is commas (`,`) instead of StarRocks‘s default column separator (`\t`). Therefore, when calling the `/api/transaction/load` operation, you must use `"column_separator: "` to specify commas (`,`) as the column separator. ###### Return result[​](#return-result-1 "Direct link to Return result") * If the data write is successful, the following result is returned: ```bash { "TxnId": 1, "Seq": 0, "Label": "streamload_txn_example1_table1", "Status": "OK", "Message": "", "NumberTotalRows": 5265644, "NumberLoadedRows": 5265644, "NumberFilteredRows": 0, "NumberUnselectedRows": 0, "LoadBytes": 10737418067, "LoadTimeMs": 418778, "StreamLoadPutTimeMs": 68, "ReceivedDataTimeMs": 38964, } ``` * If the transaction is considered unknown, the following result is returned: ```bash { "TxnId": 1, "Label": "streamload_txn_example1_table1", "Status": "FAILED", "Message": "TXN_NOT_EXISTS" } ``` * If the transaction is considered in an invalid state, the following result is returned: ```bash { "TxnId": 1, "Label": "streamload_txn_example1_table1", "Status": "FAILED", "Message": "Transcation State Invalid" } ``` * If errors other than unknown transaction and invalid status occur, the following result is returned: ```bash { "TxnId": 1, "Label": "streamload_txn_example1_table1", "Status": "FAILED", "Message": "" } ``` ##### Pre-commit a transaction[​](#pre-commit-a-transaction "Direct link to Pre-commit a transaction") ###### Syntax[​](#syntax-2 "Direct link to Syntax") ```bash curl --location-trusted -u : -H "label:" \ -H "Expect:100-continue" \ [-H "transaction_type:multi"]\ # Optional. Pre-commits a multi-table transaction. -H "db:" \ [-H "prepared_timeout:"] \ -XPOST http://:/api/transaction/prepare ``` > **NOTE** > > Specify `-H "transaction_type:multi"` in the command if the transaction you want to pre-commit is a multi-table transaction. ###### Example[​](#example-2 "Direct link to Example") ```bash curl --location-trusted -u :<123456> -H "label:streamload_txn_example1_table1" \ -H "Expect:100-continue" \ -H "db:test_db" \ -H "prepared_timeout:300" \ -XPOST http://:/api/transaction/prepare ``` > **NOTE** > > The `prepared_timeout` field is optional. If it is not specified, the default value is determined by the FE configuration [`prepared_transaction_default_timeout_second`](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#prepared_transaction_default_timeout_second) (Default: 86400 seconds). `prepared_timeout` is supported from v3.5.4 onwards. ###### Return result[​](#return-result-2 "Direct link to Return result") * If the pre-commit is successful, the following result is returned: ```bash { "TxnId": 1, "Label": "streamload_txn_example1_table1", "Status": "OK", "Message": "", "NumberTotalRows": 5265644, "NumberLoadedRows": 5265644, "NumberFilteredRows": 0, "NumberUnselectedRows": 0, "LoadBytes": 10737418067, "LoadTimeMs": 418778, "StreamLoadPutTimeMs": 68, "ReceivedDataTimeMs": 38964, "WriteDataTimeMs": 417851 "CommitAndPublishTimeMs": 1393 } ``` * If the transaction is considered not existent, the following result is returned: ```bash { "TxnId": 1, "Label": "streamload_txn_example1_table1", "Status": "FAILED", "Message": "Transcation Not Exist" } ``` * If the pre-commit times out, the following result is returned: ```bash { "TxnId": 1, "Label": "streamload_txn_example1_table1", "Status": "FAILED", "Message": "commit timeout", } ``` * If errors other than non-existent transaction and pre-commit timeout occur, the following result is returned: ```bash { "TxnId": 1, "Label": "streamload_txn_example1_table1", "Status": "FAILED", "Message": "publish timeout" } ``` ##### Commit a transaction[​](#commit-a-transaction "Direct link to Commit a transaction") ###### Syntax[​](#syntax-3 "Direct link to Syntax") ```bash curl --location-trusted -u : -H "label:" \ -H "Expect:100-continue" \ [-H "transaction_type:multi"]\ # Optional. Commits a multi-table transaction. -H "db:" \ -XPOST http://:/api/transaction/commit ``` > **NOTE** > > Specify `-H "transaction_type:multi"` in the command if the transaction you want to commit is a multi-table transaction. ###### Example[​](#example-3 "Direct link to Example") ```bash curl --location-trusted -u :<123456> -H "label:streamload_txn_example1_table1" \ -H "Expect:100-continue" \ -H "db:test_db" \ -XPOST http://:/api/transaction/commit ``` ###### Return result[​](#return-result-3 "Direct link to Return result") * If the commit is successful, the following result is returned: ```bash { "TxnId": 1, "Label": "streamload_txn_example1_table1", "Status": "OK", "Message": "", "NumberTotalRows": 5265644, "NumberLoadedRows": 5265644, "NumberFilteredRows": 0, "NumberUnselectedRows": 0, "LoadBytes": 10737418067, "LoadTimeMs": 418778, "StreamLoadPutTimeMs": 68, "ReceivedDataTimeMs": 38964, "WriteDataTimeMs": 417851 "CommitAndPublishTimeMs": 1393 } ``` * If the transaction has already been committed, the following result is returned: ```bash { "TxnId": 1, "Label": "streamload_txn_example1_table1", "Status": "OK", "Message": "Transaction already commited", } ``` * If the transaction is considered not existent, the following result is returned: ```bash { "TxnId": 1, "Label": "streamload_txn_example1_table1", "Status": "FAILED", "Message": "Transcation Not Exist" } ``` * If the commit times out, the following result is returned: ```bash { "TxnId": 1, "Label": "streamload_txn_example1_table1", "Status": "FAILED", "Message": "commit timeout", } ``` * If the data publish times out, the following result is returned: ```bash { "TxnId": 1, "Label": "streamload_txn_example1_table1", "Status": "FAILED", "Message": "publish timeout", "CommitAndPublishTimeMs": 1393 } ``` * If errors other than non-existent transaction and timeout occur, the following result is returned: ```bash { "TxnId": 1, "Label": "streamload_txn_example1_table1", "Status": "FAILED", "Message": "" } ``` ##### Roll back a transaction[​](#roll-back-a-transaction "Direct link to Roll back a transaction") ###### Syntax[​](#syntax-4 "Direct link to Syntax") ```bash curl --location-trusted -u : -H "label:" \ -H "Expect:100-continue" \ [-H "transaction_type:multi"]\ # Optional. Rolls back a multi-table transaction. -H "db:" \ -XPOST http://:/api/transaction/rollback ``` > **NOTE** > > Specify `-H "transaction_type:multi"` in the command if the transaction you want to roll back is a multi-table transaction. ###### Example[​](#example-4 "Direct link to Example") ```bash curl --location-trusted -u :<123456> -H "label:streamload_txn_example1_table1" \ -H "Expect:100-continue" \ -H "db:test_db" \ -XPOST http://:/api/transaction/rollback ``` ###### Return result[​](#return-result-4 "Direct link to Return result") * If the rollback is successful, the following result is returned: ```bash { "TxnId": 1, "Label": "streamload_txn_example1_table1", "Status": "OK", "Message": "" } ``` * If the transaction is considered not existent, the following result is returned: ```bash { "TxnId": 1, "Label": "streamload_txn_example1_table1", "Status": "FAILED", "Message": "Transcation Not Exist" } ``` * If errors other than not existent transaction occur, the following result is returned: ```bash { "TxnId": 1, "Label": "streamload_txn_example1_table1", "Status": "FAILED", "Message": "" } ``` #### References[​](#references "Direct link to References") For information about the suitable application scenarios and supported data file formats of Stream Load and about how Stream Load works, see [Loading from a local file system via Stream Load](https://docs.starrocks.io/docs/loading/StreamLoad.md#loading-from-a-local-file-system-via-stream-load). For information about the syntax and parameters for creating Stream Load jobs, see [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). --- ### Load data from a local file system StarRocks provides two methods of loading data from a local file system: * Synchronous loading using [Stream Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md) * Asynchronous loading using [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md) Each of these options has its own advantages: * Stream Load supports CSV and JSON file formats. This method is recommended if you want to load data from a small number of files whose individual sizes do not exceed 10 GB. * Broker Load supports Parquet, ORC, CSV, and JSON file formats (JSON file format is supported from v3.2.3 onwards). This method is recommended if you want to load data from a large number of files whose individual sizes exceed 10 GB, or if the files are stored in a network attached storage (NAS) device. **Using Broker Load to load data from a local file system is supported from v2.5 onwards.** For CSV data, take note of the following points: * You can use a UTF-8 string, such as a comma (,), tab, or pipe (|), whose length does not exceed 50 bytes as a text delimiter. * Null values are denoted by using `\N`. For example, a data file consists of three columns, and a record from that data file holds data in the first and third columns but no data in the second column. In this situation, you need to use `\N` in the second column to denote a null value. This means the record must be compiled as `a,\N,b` instead of `a,,b`. `a,,b` denotes that the second column of the record holds an empty string. Stream Load and Broker Load both support data transformation at data loading and supports data changes made by UPSERT and DELETE operations during data loading. For more information, see [Transform data at loading](https://docs.starrocks.io/docs/loading/Etl_in_loading.md) and [Change data through loading](https://docs.starrocks.io/docs/loading/Load_to_Primary_Key_tables.md). #### Before you begin[​](#before-you-begin "Direct link to Before you begin") ##### Check privileges[​](#check-privileges "Direct link to Check privileges") You can load data into StarRocks tables only as a user who has the INSERT privilege on those StarRocks tables. If you do not have the INSERT privilege, follow the instructions provided in [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT.md) to grant the INSERT privilege to the user that you use to connect to your StarRocks cluster. The syntax is `GRANT INSERT ON TABLE IN DATABASE TO { ROLE | USER }`. ###### Check network configuration[​](#check-network-configuration "Direct link to Check network configuration") Make sure that the machine on which the data you want to load resides can access the FE and BE nodes of the StarRocks cluster via the [`http_port`](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#http_port) (default: `8030`) and [`be_http_port`](https://docs.starrocks.io/docs/administration/management/BE_configuration.md#be_http_port) (default: `8040`) , respectively. #### Loading from a local file system via Stream Load[​](#loading-from-a-local-file-system-via-stream-load "Direct link to Loading from a local file system via Stream Load") Stream Load is an HTTP PUT-based synchronous loading method. After you submit a load job, StarRocks synchronously runs the job, and returns the result of the job after the job finishes. You can determine whether the job is successful based on the job result. > **NOTICE** > > After you load data into a StarRocks table by using Stream Load, the data of the materialized views that are created on that table is also updated. ##### How it works[​](#how-it-works "Direct link to How it works") You can submit a load request on your client to an FE according to HTTP, and the FE then uses an HTTP redirect to forward the load request to a specific BE or CN. You can also directly submit a load request on your client to a BE or CN of your choice. note If you submit load requests to an FE, the FE uses a polling mechanism to decide which BE or CN will serve as a coordinator to receive and process the load requests. The polling mechanism helps achieve load balancing within your StarRocks cluster. Therefore, we recommend that you send load requests to an FE. The BE or CN that receives the load request runs as the Coordinator BE or CN to split data based on the used schema into portions and assign each portion of the data to the other involved BEs or CNs. After the load finishes, the Coordinator BE or CN returns the result of the load job to your client. Note that if you stop the Coordinator BE or CN during the load, the load job fails. The following figure shows the workflow of a Stream Load job. ![Workflow of Stream Load](/assets/images/4.2-1-5e9185a6b2919c81f4697cceffd7734b.png) ##### Limits[​](#limits "Direct link to Limits") Stream Load does not support loading the data of a CSV file that contains a JSON-formatted column. ##### Typical example[​](#typical-example "Direct link to Typical example") This section uses curl as an example to describe how to load the data of a CSV or JSON file from your local file system into StarRocks. For detailed syntax and parameter descriptions, see [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). Note that in StarRocks some literals are used as reserved keywords by the SQL language. Do not directly use these keywords in SQL statements. If you want to use such a keyword in an SQL statement, enclose it in a pair of backticks (\`). See [Keywords](https://docs.starrocks.io/docs/sql-reference/sql-statements/keywords.md). ###### Load CSV data[​](#load-csv-data "Direct link to Load CSV data") ###### Prepare datasets[​](#prepare-datasets "Direct link to Prepare datasets") In your local file system, create a CSV file named `example1.csv`. The file consists of three columns, which represent the user ID, user name, and user score in sequence. ```plain 1,Lily,23 2,Rose,23 3,Alice,24 4,Julia,25 ``` ###### Create a database and a table[​](#create-a-database-and-a-table "Direct link to Create a database and a table") Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a Primary Key table named `table1`. The table consists of three columns: `id`, `name`, and `score`, of which `id` is the primary key. ```sql CREATE TABLE `table1` ( `id` int(11) NOT NULL COMMENT "user ID", `name` varchar(65533) NULL COMMENT "user name", `score` int(11) NOT NULL COMMENT "user score" ) ENGINE=OLAP PRIMARY KEY(`id`) DISTRIBUTED BY HASH(`id`); ``` note Since v2.5.7, StarRocks can automatically set the number of buckets (BUCKETS) when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). ###### Start a Stream Load[​](#start-a-stream-load "Direct link to Start a Stream Load") Run the following command to load the data of `example1.csv` into `table1`: ```bash curl --location-trusted -u : -H "label:123" \ -H "Expect:100-continue" \ -H "column_separator:," \ -H "columns: id, name, score" \ -T example1.csv -XPUT \ http://:/api/mydatabase/table1/_stream_load ``` note * If you use an account for which no password is set, you need to input only `:`. * You can use [SHOW FRONTENDS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_FRONTENDS.md) to view the IP address and HTTP port of the FE node. `example1.csv` consists of three columns, which are separated by commas (,) and can be mapped in sequence onto the `id`, `name`, and `score` columns of `table1`. Therefore, you need to use the `column_separator` parameter to specify the comma (,) as the column separator. You also need to use the `columns` parameter to temporarily name the three columns of `example1.csv` as `id`, `name`, and `score`, which are mapped in sequence onto the three columns of `table1`. After the load is complete, you can query `table1` to verify that the load is successful: ```sql SELECT * FROM table1; +------+-------+-------+ | id | name | score | +------+-------+-------+ | 1 | Lily | 23 | | 2 | Rose | 23 | | 3 | Alice | 24 | | 4 | Julia | 25 | +------+-------+-------+ 4 rows in set (0.00 sec) ``` ###### Load JSON data[​](#load-json-data "Direct link to Load JSON data") Since v3.2.7, Stream Load supports compressing JSON data during transmission, reducing network bandwidth overhead. Users can specify different compression algorithms using parameters `compression` and `Content-Encoding`. Supported compression algorithms including GZIP, BZIP2, LZ4\_FRAME, and ZSTD. For the syntax, see [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). ###### Prepare datasets[​](#prepare-datasets-1 "Direct link to Prepare datasets") In your local file system, create a JSON file named `example2.json`. The file consists of two columns, which represent city ID and city name in sequence. ```json {"name": "Beijing", "code": 2} ``` ###### Create a database and a table[​](#create-a-database-and-a-table-1 "Direct link to Create a database and a table") Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a Primary Key table named `table2`. The table consists of two columns: `id` and `city`, of which `id` is the primary key. ```sql CREATE TABLE `table2` ( `id` int(11) NOT NULL COMMENT "city ID", `city` varchar(65533) NULL COMMENT "city name" ) ENGINE=OLAP PRIMARY KEY(`id`) DISTRIBUTED BY HASH(`id`); ``` note Since v2.5.7, StarRocks can set the number of(BUCKETS) automatically when you create a table or add a partition. You no longer need to manually set the number of buckets. For detailed information, see [set the number of buckets](https://docs.starrocks.io/docs/table_design/data_distribution.md#set-the-number-of-buckets). ###### Start a Stream Load[​](#start-a-stream-load-1 "Direct link to Start a Stream Load") Run the following command to load the data of `example2.json` into `table2`: ```bash curl -v --location-trusted -u : -H "strict_mode: true" \ -H "Expect:100-continue" \ -H "format: json" -H "jsonpaths: [\"$.name\", \"$.code\"]" \ -H "columns: city,tmp_id, id = tmp_id * 100" \ -T example2.json -XPUT \ http://:/api/mydatabase/table2/_stream_load ``` note * If you use an account for which no password is set, you need to input only `:`. * You can use [SHOW FRONTENDS](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_FRONTENDS.md) to view the IP address and HTTP port of the FE node. `example2.json` consists of two keys, `name` and `code`, which are mapped onto the `id` and `city` columns of `table2`, as shown in the following figure. ![JSON - Column Mapping](/assets/images/4.2-2-c642e32999d9076ae9633ff470fc8764.png) The mappings shown in the preceding figure are described as follows: * StarRocks extracts the `name` and `code` keys of `example2.json` and maps them onto the `name` and `code` fields declared in the `jsonpaths` parameter. * StarRocks extracts the `name` and `code` fields declared in the `jsonpaths` parameter and **maps them in sequence** onto the `city` and `tmp_id` fields declared in the `columns` parameter. * StarRocks extracts the `city` and `tmp_id` fields declared in the `columns` parameter and **maps them by name** onto the `city` and `id` columns of `table2`. note In the preceding example, the value of `code` in `example2.json` is multiplied by 100 before it is loaded into the `id` column of `table2`. For detailed mappings between `jsonpaths`, `columns`, and the columns of the StarRocks table, see the "Column mappings" section in [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). After the load is complete, you can query `table2` to verify that the load is successful: ```sql SELECT * FROM table2; +------+--------+ | id | city | +------+--------+ | 200 | Beijing| +------+--------+ 4 rows in set (0.01 sec) ``` ###### Merge Stream Load requests[​](#merge-stream-load-requests "Direct link to Merge Stream Load requests") Beta feature [Advice on use of Beta features](https://docs.starrocks.io/docs/introduction/maturity.md) From v3.4.0, the system supports merging multiple Stream Load requests. warning Note that the Merge Commit optimization is suitable for the scenario with **concurrent** Stream Load jobs on a single table. It is not recommended if the concurrency is one. Meanwhile, think twice before setting `merge_commit_async` to `false` and `merge_commit_interval_ms` to a large value because they may cause load performance degradation. Merge Commit is an optimization for Stream Load, designed for high concurrency, small-batch (from KB to tens of MB) real-time loading scenarios. In earlier versions, each Stream Load request would generate a transaction and a data version, which led to the following issues in high concurrency loading scenarios: * Excessive data versions impact query performance, and limiting the number of versions may cause `too many versions` errors. * Data version merging through Compaction increases resource consumption. * It generates small files, increasing IOPS and I/O latency. And in shared-data clusters, this also raises cloud object storage costs. * Leader FE node, as the transaction manager, may become a single point of bottleneck. Merge Commit mitigates these issues by merging multiple concurrent Stream Load requests within a time window into a single transaction. This reduces the number of transactions and versions generated by high concurrency requests, thereby improving loading performance. Merge Commit supports both synchronous and asynchronous modes. Each mode has advantages and disadvantages. You can choose based on your use cases. * **Synchronous mode** The server returns only after the merged transaction is committed, ensuring the loading is successful and visible. * **Asynchronous mode** The server returns immediately after receiving the data. This mode does not ensure the loading is successful. | **Mode** | **Advantages** | **Disadvantages** | | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Synchronous | - Ensures data persistence and visibility upon request return.
- Guarantees that multiple sequential loading requests from the same client are executed in order. | Each loading request from the client is blocked until the server closes the merge window. It may reduce the data processing capability of a single client if the window is excessively large. | | Asynchronous | Allows a single client to send subsequent loading requests without waiting for the server to close the merge window, improving loading throughput. | - Does not guarantee data persistence or visibility upon return. The client must later verify the transaction status.
- Does not guarantee that multiple sequential loading requests from the same client are executed in order. | ###### Start a Stream Load[​](#start-a-stream-load-2 "Direct link to Start a Stream Load") * Run the following command to start a Stream Load job with Merge Commit enabled in synchronous mode, and set the merging window to `5000` milliseconds and degree of parallelism to `2`: ```bash curl --location-trusted -u : \ -H "Expect:100-continue" \ -H "column_separator:," \ -H "columns: id, name, score" \ -H "enable_merge_commit:true" \ -H "merge_commit_interval_ms:5000" \ -H "merge_commit_parallel:2" \ -T example1.csv -XPUT \ http://:/api/mydatabase/table1/_stream_load ``` * Run the following command to start a Stream Load job with Merge Commit enabled in asynchronous mode, and set the merging window to `60000` milliseconds and degree of parallelism to `2`: ```bash curl --location-trusted -u : \ -H "Expect:100-continue" \ -H "column_separator:," \ -H "columns: id, name, score" \ -H "enable_merge_commit:true" \ -H "merge_commit_async:true" \ -H "merge_commit_interval_ms:60000" \ -H "merge_commit_parallel:2" \ -T example1.csv -XPUT \ http://:/api/mydatabase/table1/_stream_load ``` note * Merge Commit only supports merging **homogeneous** loading requests into a single database and table. "Homogeneous" indicates that the Stream Load parameters are identical, including: common parameters, JSON format parameters, CSV format parameters, `opt_properties`, and Merge Commit parameters. * For loading CSV-formatted data, you must ensure that each row ends with a line separator. `skip_header` is not supported. * The server automatically generates labels for transactions. They will be ignored if specified. * Merge Commit merges multiple loading requests into a single transaction. If one request contains data quality issues, all requests in the transaction will fail. ###### Check Stream Load progress[​](#check-stream-load-progress "Direct link to Check Stream Load progress") After a load job is complete, StarRocks returns the result of the job in JSON format. For more information, see the "Return value" section in [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). Stream Load does not allow you to query the result of a load job by using the SHOW LOAD statement. ###### Cancel a Stream Load job[​](#cancel-a-stream-load-job "Direct link to Cancel a Stream Load job") Stream Load does not allow you to cancel a load job. If a load job times out or encounters errors, StarRocks automatically cancels the job. ##### Parameter configurations[​](#parameter-configurations "Direct link to Parameter configurations") This section describes some system parameters that you need to configure if you choose the loading method Stream Load. These parameter configurations take effect on all Stream Load jobs. * `streaming_load_max_mb`: the maximum size of each data file you want to load. The default maximum size is 10 GB. For more information, see [Configure BE or CN dynamic parameters](https://docs.starrocks.io/docs/administration/management/BE_configuration.md). We recommend that you do not load more than 10 GB of data at a time. If the size of a data file exceeds 10 GB, we recommend that you split the data file into small files that each are less than 10 GB in size and then load these files one by one. If you cannot split a data file greater than 10 GB, you can increase the value of this parameter based on the file size. After you increase the value of this parameter, the new value can take effect only after you restart the BEs or CNs of your StarRocks cluster. Additionally, system performance may deteriorate, and the costs of retries in the event of load failures also increase. note When you load the data of a JSON file, take note of the following points: * The size of each JSON object in the file cannot exceed 4 GB. If any JSON object in the file exceeds 4 GB, StarRocks throws an error "This parser can't support a document that big." * By default, the JSON body in an HTTP request cannot exceed 100 MB. If the JSON body exceeds 100 MB, StarRocks throws an error "The size of this batch exceed the max size \[104857600] of json type data data \[8617627793]. Set ignore\_json\_size to skip check, although it may lead huge memory consuming." To prevent this error, you can add `"ignore_json_size:true"` in the HTTP request header to ignore the check on the JSON body size. * `stream_load_default_timeout_second`: the timeout period of each load job. The default timeout period is 600 seconds. For more information, see [Configure FE dynamic parameters](https://docs.starrocks.io/docs/administration/management/FE_configuration.md#configure-fe-dynamic-parameters). If many of the load jobs that you create time out, you can increase the value of this parameter based on the calculation result that you obtain from the following formula: **Timeout period of each load job > Amount of data to be loaded/Average loading speed** For example, if the size of the data file that you want to load is 10 GB and the average loading speed of your StarRocks cluster is 100 MB/s, set the timeout period to more than 100 seconds. note **Average loading speed** in the preceding formula is the average loading speed of your StarRocks cluster. It varies depending on the disk I/O and the number of BEs or CNs in your StarRocks cluster. Stream Load also provides the `timeout` parameter, which allows you to specify the timeout period of an individual load job. For more information, see [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md). ##### Usage notes[​](#usage-notes "Direct link to Usage notes") If a field is missing for a record in the data file you want to load and the column onto which the field is mapped in your StarRocks table is defined as `NOT NULL`, StarRocks automatically fills a `NULL` value in the mapping column of your StarRocks table during the load of the record. You can also use the `ifnull()` function to specify the default value that you want to fill. For example, if the field that represents city ID in the preceding `example2.json` file is missing and you want to fill an `x` value in the mapping column of `table2`, you can specify `"columns: city, tmp_id, id = ifnull(tmp_id, 'x')"`. #### Loading from a local file system via Broker Load[​](#loading-from-a-local-file-system-via-broker-load "Direct link to Loading from a local file system via Broker Load") In addition to Stream Load, you can also use Broker Load to load data from a local file system. This feature is supported from v2.5 onwards. Broker Load is an asynchronous loading method. After you submit a load job, StarRocks asynchronously runs the job and does not immediately return the job result. You need to query the job result by hand. See [Check Broker Load progress](#check-broker-load-progress). ##### Limits[​](#limits-1 "Direct link to Limits") * Currently Broker Load supports loading from a local file system only through a single broker whose version is v2.5 or later. * Highly concurrent queries against a single broker may cause issues such as timeout and OOM. To mitigate the impact, you can use the `pipeline_dop` variable (see [System variable](https://docs.starrocks.io/docs/sql-reference/System_variable.md#pipeline_dop)) to set the query parallelism for Broker Load. For queries against a single broker, we recommend that you set `pipeline_dop` to a value smaller than `16`. ##### Typical example[​](#typical-example-1 "Direct link to Typical example") Broker Load supports loading from a single data file to a single table, loading from multiple data files to a single table, and loading from multiple data files to multiple tables. This section uses loading from multiple data files to a single table as an example. Note that in StarRocks some literals are used as reserved keywords by the SQL language. Do not directly use these keywords in SQL statements. If you want to use such a keyword in an SQL statement, enclose it in a pair of backticks (\`). See [Keywords](https://docs.starrocks.io/docs/sql-reference/sql-statements/keywords.md). ###### Prepare datasets[​](#prepare-datasets-2 "Direct link to Prepare datasets") Use the CSV file format as an example. Log in to your local file system, and create two CSV files, `file1.csv` and `file2.csv`, in a specific storage location (for example, `/home/disk1/business/`). Both files consist of three columns, which represent the user ID, user name, and user score in sequence. * `file1.csv` ```plain 1,Lily,21 2,Rose,22 3,Alice,23 4,Julia,24 ``` * `file2.csv` ```plain 5,Tony,25 6,Adam,26 7,Allen,27 8,Jacky,28 ``` ###### Create a database and a table[​](#create-a-database-and-a-table-2 "Direct link to Create a database and a table") Create a database and switch to it: ```sql CREATE DATABASE IF NOT EXISTS mydatabase; USE mydatabase; ``` Create a Primary Key table named `mytable`. The table consists of three columns: `id`, `name`, and `score`, of which `id` is the primary key. ```sql CREATE TABLE `mytable` ( `id` int(11) NOT NULL COMMENT "User ID", `name` varchar(65533) NULL DEFAULT "" COMMENT "User name", `score` int(11) NOT NULL DEFAULT "0" COMMENT "User score" ) ENGINE=OLAP PRIMARY KEY(`id`) DISTRIBUTED BY HASH(`id`) PROPERTIES("replication_num"="1"); ``` ###### Start a Broker Load[​](#start-a-broker-load "Direct link to Start a Broker Load") Run the following command to start a Broker Load job that loads data from all data files (`file1.csv` and `file2.csv`) stored in the `/home/disk1/business/` path of your local file system to the StarRocks table `mytable`: ```sql LOAD LABEL mydatabase.label_local ( DATA INFILE("file:///home/disk1/business/csv/*") INTO TABLE mytable COLUMNS TERMINATED BY "," (id, name, score) ) WITH BROKER "sole_broker" PROPERTIES ( "timeout" = "3600" ); ``` This job has four main sections: * `LABEL`: A string used when querying the state of the load job. * `LOAD` declaration: The source URI, source data format, and destination table name. * `PROPERTIES`: The timeout value and any other properties to apply to the load job. For detailed syntax and parameter descriptions, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). ###### Check Broker Load progress[​](#check-broker-load-progress "Direct link to Check Broker Load progress") In v3.0 and earlier, use the [SHOW LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SHOW_LOAD.md) statement or the curl command to view the progress of Broker Load jobs. In v3.1 and later, you can view the progress of Broker Load jobs from the [`information_schema.loads`](https://docs.starrocks.io/docs/sql-reference/information_schema/loads.md) view: ```sql SELECT * FROM information_schema.loads; ``` If you have submitted multiple load jobs, you can filter on the `LABEL` associated with the job. Example: ```sql SELECT * FROM information_schema.loads WHERE LABEL = 'label_local'; ``` After you confirm that the load job has finished, you can query table to see if the data has been successfully loaded. Example: ```sql SELECT * FROM mytable; +------+-------+-------+ | id | name | score | +------+-------+-------+ | 3 | Alice | 23 | | 5 | Tony | 25 | | 6 | Adam | 26 | | 1 | Lily | 21 | | 2 | Rose | 22 | | 4 | Julia | 24 | | 7 | Allen | 27 | | 8 | Jacky | 28 | +------+-------+-------+ 8 rows in set (0.07 sec) ``` ###### Cancel a Broker Load job[​](#cancel-a-broker-load-job "Direct link to Cancel a Broker Load job") When a load job is not in the **CANCELLED** or **FINISHED** stage, you can use the [CANCEL LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/CANCEL_LOAD.md) statement to cancel the job. For example, you can execute the following statement to cancel a load job, whose label is `label_local`, in the database `mydatabase`: ```sql CANCEL LOAD FROM mydatabase WHERE LABEL = "label_local"; ``` #### Loading from NAS via Broker Load[​](#loading-from-nas-via-broker-load "Direct link to Loading from NAS via Broker Load") There are two ways to load data from NAS by using Broker Load: * Consider NAS as a local file system, and run a load job with a broker. See the previous section "[Loading from a local system via Broker Load](#loading-from-a-local-file-system-via-broker-load)". * (Recommended) Consider NAS as a cloud storage system, and run a load job without a broker. This section introduces the second way. Detailed operations are as follows: 1. Mount your NAS device to the same path on all the BE or CN nodes and FE nodes of your StarRocks cluster. As such, all BEs or CNs can access the NAS device like they access their own locally stored files. 2. Use Broker Load to load data from the NAS device to the destination StarRocks table. Example: ```sql LOAD LABEL test_db.label_nas ( DATA INFILE("file:///home/disk1/sr/*") INTO TABLE mytable COLUMNS TERMINATED BY "," ) WITH BROKER PROPERTIES ( "timeout" = "3600" ); ``` This job has four main sections: * `LABEL`: A string used when querying the state of the load job. * `LOAD` declaration: The source URI, source data format, and destination table name. Note that `DATA INFILE` in the declaration is used to specify the mount point folder path of the NAS device, as shown in the above example in which `file:///` is the prefix and `/home/disk1/sr` is the mount point folder path. * `BROKER`: You do not need to specify the broker name. * `PROPERTIES`: The timeout value and any other properties to apply to the load job. For detailed syntax and parameter descriptions, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md). After you submit a job, you can view the load progress or cancel the job as needed. For detailed operations, see "[Check Broker Load progress](#check-broker-load-progress)" and "[Cancel a Broker Load job](#cancel-a-broker-load-job) in this topic. --- ### tencent --- ## Project_help ### Community Chat (via Slack) The StarRocks project uses Slack for community chat. You can find our community at [starrocks.slack.com](https://docs.starrocks.io/join/). --- ## Quick_start ### Quick Start These Quick Start guides will help you get going with a small StarRocks environment. The clusters that you will launch will be suitable for learning how StarRocks works, but are not meant for analyzing large datasets or performance testing. For scalable deployments please see [deploying StarRocks](https://docs.starrocks.io/docs/deployment/deployment_overview.md) #### [📄️ Claude + StarRocks MCP](https://docs.starrocks.io/docs/quick_start/MCP.md) [Analyze complex data with Claude and the StarRocks MCP server on a shared-data cluster.](https://docs.starrocks.io/docs/quick_start/MCP.md) #### [📄️ Deploy StarRocks with Docker](https://docs.starrocks.io/docs/quick_start/shared-nothing.md) [StarRocks in Docker: Query real data with JOINs](https://docs.starrocks.io/docs/quick_start/shared-nothing.md) #### [📄️ Separate storage and compute](https://docs.starrocks.io/docs/quick_start/shared-data.md) [Quick start guide for deploying StarRocks in shared-data mode with separate compute and storage.](https://docs.starrocks.io/docs/quick_start/shared-data.md) #### [📄️ Apache Iceberg Lakehouse](https://docs.starrocks.io/docs/quick_start/iceberg.md) [Quick start guide for querying Apache Iceberg data lakes with StarRocks.](https://docs.starrocks.io/docs/quick_start/iceberg.md) #### [📄️ Apache Hudi Lakehouse](https://docs.starrocks.io/docs/quick_start/hudi.md) [Quick start guide for querying Apache Hudi data lakes with StarRocks.](https://docs.starrocks.io/docs/quick_start/hudi.md) #### [📄️ Kafka routine load StarRocks using shared-data storage](https://docs.starrocks.io/docs/quick_start/routine-load.md) [Quick start guide for streaming data from Kafka into StarRocks using Routine Load with shared-data storage.](https://docs.starrocks.io/docs/quick_start/routine-load.md) #### [📄️ StarRocks with Helm](https://docs.starrocks.io/docs/quick_start/helm.md) [Quick start guide for deploying StarRocks on Kubernetes using Helm.](https://docs.starrocks.io/docs/quick_start/helm.md) --- ### StarRocks with Helm #### Goals[​](#goals "Direct link to Goals") The goals of this quickstart are: * Deploy the StarRocks Kubernetes Operator and a StarRocks cluster with Helm * Configure a password for the StarRocks database user `root` * Provide for high-availability with three FEs and three BEs * Store metadata in persistent storage * Store data in persistent storage * Allow MySQL clients to connect from outside the Kubernetes cluster * Allow loading data from outside the Kubernetes cluster using Stream Load * Load some public datasets * Query the data tip The datasets and queries are the same as the ones used in the Basic Quick Start. The main difference here is deploying with Helm and the StarRocks Operator. The data used is provided by NYC OpenData and the National Centers for Environmental Information. Both of these datasets are large, and because this tutorial is intended to help you get exposed to working with StarRocks we are not going to load data for the past 120 years. You can run this with a GKE Kubernetes cluster built on three e2-standard-4 machines (or similar) with 80GB disk. For larger deployments, we have other documentation and will provide that later. There is a lot of information in this document, and it is presented with step-by-step content at the beginning, and the technical details at the end. This is done to serve these purposes in this order: 1. Get the system deployed with Helm. 2. Allow the reader to load data in StarRocks and analyze that data. 3. Explain the basics of data transformation during loading. *** #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") ##### Kubernetes environment[​](#kubernetes-environment "Direct link to Kubernetes environment") The Kubernetes environment used while writing this guide consists of three nodes with four vCPUS, and 16GB RAM each (GCP `e2-standard-4` machines). The Kubernetes cluster was deployed with this `gcloud` command: tip This command is for your reference, if you are using AWS, Azure, or any other Kubernetes provider you will need to modify this for your environment. In Google Cloud you will need to specify your own project and an appropriate location. ```bash gcloud container --project enterprise-demo-422514 \ clusters create ee-docs \ --location=southamerica-west1-b \ --machine-type e2-standard-4 --disk-size 80 --num-nodes 3 ``` ##### Helm[​](#helm "Direct link to Helm") Helm is a package manager for Kubernetes that simplifies the deployment and management of applications. In this lab you will use Helm to deploy the CelerData Enterprise Edition Kubernetes operator and the sample StarRocks cluster. [Install helm](https://helm.sh/docs/intro/quickstart/) ##### SQL client[​](#sql-client "Direct link to SQL client") You can use the SQL client provided in the Kubernetes environment, or use one on your system. This guide uses the `mysql CLI` Many MySQL-compatible clients will work. ##### curl[​](#curl "Direct link to curl") `curl` is used to issue the data load job to StarRocks, and to download the datasets. Check to see if you have it installed by running `curl` or `curl.exe` at your OS prompt. If curl is not installed, [get curl here](https://curl.se/). *** #### Terminology[​](#terminology "Direct link to Terminology") ##### FE[​](#fe "Direct link to FE") Frontend nodes are responsible for metadata management, client connection management, query planning, and query scheduling. Each FE stores and maintains a complete copy of metadata in its memory, which guarantees indiscriminate services among the FEs. ##### BE[​](#be "Direct link to BE") Backend nodes are responsible for both data storage and executing query plans. *** #### Add the StarRocks Helm chart repo[​](#add-the-starrocks-helm-chart-repo "Direct link to Add the StarRocks Helm chart repo") The Helm Chart contains the definitions of the StarRocks Operator and the custom resource StarRocksCluster. 1. Add the Helm Chart Repo. ```bash helm repo add starrocks https://starrocks.github.io/starrocks-kubernetes-operator ``` 2. Update the Helm Chart Repo to the latest version. ```bash helm repo update ``` 3. View the Helm Chart Repo that you added. ```bash helm search repo starrocks ``` ```text NAME CHART VERSION APP VERSION DESCRIPTION starrocks/kube-starrocks 1.9.7 3.2-latest kube-starrocks includes two subcharts, operator... starrocks/operator 1.9.7 1.9.7 A Helm chart for StarRocks operator starrocks/starrocks 1.9.7 3.2-latest A Helm chart for StarRocks cluster starrocks/warehouse 1.9.7 3.2-latest Warehouse is currently a feature of the StarRoc... ``` *** #### Download the data[​](#download-the-data "Direct link to Download the data") Download these two datasets to your machine. ##### New York City crash data[​](#new-york-city-crash-data "Direct link to New York City crash data") ```bash curl -O https://raw.githubusercontent.com/StarRocks/demo/master/documentation-samples/quickstart/datasets/NYPD_Crash_Data.csv ``` ##### Weather data[​](#weather-data "Direct link to Weather data") ```bash curl -O https://raw.githubusercontent.com/StarRocks/demo/master/documentation-samples/quickstart/datasets/72505394728.csv ``` *** #### Create a Helm values file[​](#create-a-helm-values-file "Direct link to Create a Helm values file") The goals for this quick start are: 1. Configure a password for the StarRocks database user `root` 2. Provide for high-availability with three FEs and three BEs 3. Store metadata in persistent storage 4. Store data in persistent storage 5. Allow MySQL clients to connect from outside the Kubernetes cluster 6. Allow loading data from outside the Kubernetes cluster using Stream Load The Helm chart provides options to satisfy all of these goals, but they are not configured by default. The rest of this section covers the configuration needed to meet all of these goals. A complete values spec will be provided, but first read the details for each of the six sections and then copy the full spec. ##### 1. Password for the database user[​](#1-password-for-the-database-user "Direct link to 1. Password for the database user") This bit of YAML instructs the StarRocks operator to set the password for the database user `root` to the value of the `password` key of the Kubernetes secret \`starrocks-root-pass. ```yaml starrocks: initPassword: enabled: true # Set a password secret, for example: # kubectl create secret generic starrocks-root-pass --from-literal=password='g()()dpa$$word' passwordSecret: starrocks-root-pass ``` * Task: Create the Kubernetes secret ```bash kubectl create secret generic starrocks-root-pass --from-literal=password='g()()dpa$$word' ``` ##### 2. High Availability with 3 FEs and 3 BEs[​](#2-high-availability-with-3-fes-and-3-bes "Direct link to 2. High Availability with 3 FEs and 3 BEs") By setting `starrocks.starrockFESpec.replicas` to 3, and `starrocks.starrockBeSpec.replicas` to 3 you will have enough FEs and BEs for high availability. Setting the CPU and memory requests low allows the pods to be created in a small Kubernetes environment. ```yaml starrocks: starrocksFESpec: replicas: 3 resources: requests: cpu: 1 memory: 1Gi starrocksBeSpec: replicas: 3 resources: requests: cpu: 1 memory: 2Gi ``` ##### 3. Store metadata in persistent storage[​](#3-store-metadata-in-persistent-storage "Direct link to 3. Store metadata in persistent storage") Setting a value for `starrocks.starrocksFESpec.storageSpec.name` to anything other than `""` causes: * Persistent storage to be used * the value of `starrocks.starrocksFESpec.storageSpec.name` to be used as the prefix for all storage volumes for the service. By setting the value to `fe` these PVs will be created for FE 0: * `fe-meta-kube-starrocks-fe-0` * `fe-log-kube-starrocks-fe-0` ```yaml starrocks: starrocksFESpec: storageSpec: name: fe ``` ##### 4. Store data in persistent storage[​](#4-store-data-in-persistent-storage "Direct link to 4. Store data in persistent storage") Setting a value for `starrocks.starrocksBeSpec.storageSpec.name` to anything other than `""` causes: * Persistent storage to be used * the value of `starrocks.starrocksBeSpec.storageSpec.name` to be used as the prefix for all storage volumes for the service. By setting the value to `be` these PVs will be created for BE 0: * `be-data-kube-starrocks-be-0` * `be-log-kube-starrocks-be-0` Setting the `storageSize` to 15Gi reduces the storage from the default of 1Ti to fit smaller quotas for storage. ```yaml starrocks: starrocksBeSpec: storageSpec: name: be storageSize: 15Gi ``` ##### 5. LoadBalancer for MySQL clients[​](#5-loadbalancer-for-mysql-clients "Direct link to 5. LoadBalancer for MySQL clients") By default, access to the FE service is through cluster IPs. To allow external access, `service.type` is set to `LoadBalancer` ```yaml starrocks: starrocksFESpec: service: type: LoadBalancer ``` ##### 6. LoadBalancer for external data loading[​](#6-loadbalancer-for-external-data-loading "Direct link to 6. LoadBalancer for external data loading") Stream Load requires external access to both FEs and BEs. The requests are sent to the FE and then the FE assigns a BE to process the upload. To allow the `curl` command to be redirected to the BE the `starroclFeProxySpec` needs to be enabled and set to type `LoadBalancer`. ```yaml starrocks: starrocksFeProxySpec: enabled: true service: type: LoadBalancer ``` ##### The complete values file[​](#the-complete-values-file "Direct link to The complete values file") The above snippets combined provide a full values file. Save this to `my-values.yaml`: ```yaml starrocks: initPassword: enabled: true # Set a password secret, for example: # kubectl create secret generic starrocks-root-pass --from-literal=password='g()()dpa$$word' passwordSecret: starrocks-root-pass starrocksFESpec: replicas: 3 service: type: LoadBalancer resources: requests: cpu: 1 memory: 1Gi storageSpec: name: fe starrocksBeSpec: replicas: 3 resources: requests: cpu: 1 memory: 2Gi storageSpec: name: be storageSize: 15Gi starrocksFeProxySpec: enabled: true service: type: LoadBalancer ``` #### Set the StarRocks root database user password[​](#set-the-starrocks-root-database-user-password "Direct link to Set the StarRocks root database user password") To load data from outside of the Kubernetes cluster the StarRocks database will be exposed externally. You should set a password for the StarRocks database user `root`. The operator will apply the password to the FE and BE nodes. ```bash kubectl create secret generic starrocks-root-pass --from-literal=password='g()()dpa$$word' ``` ```text secret/starrocks-root-pass created ``` *** #### Deploy the operator and StarRocks cluster[​](#deploy-the-operator-and-starrocks-cluster "Direct link to Deploy the operator and StarRocks cluster") ```bash helm install -f my-values.yaml starrocks starrocks/kube-starrocks ``` ```text NAME: starrocks LAST DEPLOYED: Wed Jun 26 20:25:09 2024 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: Thank you for installing kube-starrocks-1.9.7 kube-starrocks chart. It will install both operator and starrocks cluster, please wait for a few minutes for the cluster to be ready. Please see the values.yaml for more operation information: https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/helm-charts/charts/kube-starrocks/values.yaml ``` #### Check the status of the StarRocks cluster[​](#check-the-status-of-the-starrocks-cluster "Direct link to Check the status of the StarRocks cluster") You can check the progress with these commands: ```bash kubectl --namespace default get starrockscluster -l "cluster=kube-starrocks" ``` ```text NAME PHASE FESTATUS BESTATUS CNSTATUS FEPROXYSTATUS kube-starrocks reconciling reconciling reconciling reconciling ``` ```bash kubectl get pods ``` note The `kube-starrocks-initpwd` pod will go through `error` and `CrashLoopBackOff` states as it attempts to connect to the FE and BE pods to set the StarRocks root password. You should ignore these errors and wait for a status of `Completed` for this pod. ```text NAME READY STATUS RESTARTS AGE kube-starrocks-be-0 0/1 Running 0 20s kube-starrocks-be-1 0/1 Running 0 20s kube-starrocks-be-2 0/1 Running 0 20s kube-starrocks-fe-0 1/1 Running 0 66s kube-starrocks-fe-1 0/1 Running 0 65s kube-starrocks-fe-2 0/1 Running 0 66s kube-starrocks-fe-proxy-56f8998799-d4qmt 1/1 Running 0 20s kube-starrocks-initpwd-m84br 0/1 CrashLoopBackOff 3 (50s ago) 92s kube-starrocks-operator-54ffcf8c5c-xsjc8 1/1 Running 0 92s ``` ```bash kubectl get pvc ``` ```text NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE be-data-kube-starrocks-be-0 Bound pvc-4ae0c9d8-7f9a-4147-ad74-b22569165448 15Gi RWO standard-rwo 82s be-data-kube-starrocks-be-1 Bound pvc-28b4dbd1-0c8f-4b06-87e8-edec616cabbc 15Gi RWO standard-rwo 82s be-data-kube-starrocks-be-2 Bound pvc-c7232ea6-d3d9-42f1-bfc1-024205a17656 15Gi RWO standard-rwo 82s be-log-kube-starrocks-be-0 Bound pvc-6193c43d-c74f-4d12-afcc-c41ace3d5408 1Gi RWO standard-rwo 82s be-log-kube-starrocks-be-1 Bound pvc-c01f124a-014a-439a-99a6-6afe95215bf0 1Gi RWO standard-rwo 82s be-log-kube-starrocks-be-2 Bound pvc-136df15f-4d2e-43bc-a1c0-17227ce3fe6b 1Gi RWO standard-rwo 82s fe-log-kube-starrocks-fe-0 Bound pvc-7eac524e-d286-4760-b21c-d9b6261d976f 5Gi RWO standard-rwo 2m23s fe-log-kube-starrocks-fe-1 Bound pvc-38076b78-71e8-4659-b8e7-6751bec663f6 5Gi RWO standard-rwo 2m23s fe-log-kube-starrocks-fe-2 Bound pvc-4ccfee60-02b7-40ba-a22e-861ea29dac74 5Gi RWO standard-rwo 2m23s fe-meta-kube-starrocks-fe-0 Bound pvc-5130c9ff-b797-4f79-a1d2-4214af860d70 10Gi RWO standard-rwo 2m23s fe-meta-kube-starrocks-fe-1 Bound pvc-13545330-63be-42cf-b1ca-3ed6f96a8c98 10Gi RWO standard-rwo 2m23s fe-meta-kube-starrocks-fe-2 Bound pvc-609cadd4-c7b7-4cf9-84b0-a75678bb3c4d 10Gi RWO standard-rwo 2m23s ``` ##### Verify that the cluster is healthy[​](#verify-that-the-cluster-is-healthy "Direct link to Verify that the cluster is healthy") tip These are the same commands as above, but show the desired state. ```bash kubectl --namespace default get starrockscluster -l "cluster=kube-starrocks" ``` ```text NAME PHASE FESTATUS BESTATUS CNSTATUS FEPROXYSTATUS kube-starrocks running running running running ``` ```bash kubectl get pods ``` tip The system is ready when all of the pods except for `kube-starrocks-initpwd` show `1/1` in the `READY` column. The `kube-starrocks-initpwd` pod should show `0/1` and a `STATUS` of `Completed`. ```text NAME READY STATUS RESTARTS AGE kube-starrocks-be-0 1/1 Running 0 57s kube-starrocks-be-1 1/1 Running 0 57s kube-starrocks-be-2 1/1 Running 0 57s kube-starrocks-fe-0 1/1 Running 0 103s kube-starrocks-fe-1 1/1 Running 0 102s kube-starrocks-fe-2 1/1 Running 0 103s kube-starrocks-fe-proxy-56f8998799-d4qmt 1/1 Running 0 57s kube-starrocks-initpwd-m84br 0/1 Completed 4 2m9s kube-starrocks-operator-54ffcf8c5c-xsjc8 1/1 Running 0 2m9s ``` The `EXTERNAL-IP` addresses in the highlighted lines will be used to provide SQL client and Stream Load access from outside the Kubernetes cluster. ```bash kubectl get services ``` ```bash NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-starrocks-be-search ClusterIP None 9050/TCP 78s kube-starrocks-be-service ClusterIP 34.118.228.231 9060/TCP,8040/TCP,9050/TCP,8060/TCP 78s kube-starrocks-fe-proxy-service LoadBalancer 34.118.230.176 34.176.12.205 8080:30241/TCP 78s kube-starrocks-fe-search ClusterIP None 9030/TCP 2m4s kube-starrocks-fe-service LoadBalancer 34.118.226.82 34.176.215.97 8030:30620/TCP,9020:32461/TCP,9030:32749/TCP,9010:30911/TCP 2m4s kubernetes ClusterIP 34.118.224.1 443/TCP 8h ``` tip Store the `EXTERNAL-IP` addresses from the highlighted lines in environment variables so that you have them handy: ```text export MYSQL_IP=`kubectl get services kube-starrocks-fe-service --output jsonpath='{.status.loadBalancer.ingress[0].ip}'` ``` ```text export FE_PROXY=`kubectl get services kube-starrocks-fe-proxy-service --output jsonpath='{.status.loadBalancer.ingress[0].ip}'`:8080 ``` *** ##### Connect to StarRocks with a SQL client[​](#connect-to-starrocks-with-a-sql-client "Direct link to Connect to StarRocks with a SQL client") tip If you are using a client other than the mysql CLI, open that now. This command will run the `mysql` command in a Kubernetes pod: ```sql kubectl exec --stdin --tty kube-starrocks-fe-0 -- \ mysql -P9030 -h127.0.0.1 -u root --prompt="StarRocks > " ``` If you have the mysql CLI installed locally, you can use it instead of the one in the Kubernetes cluster: ```sql mysql -P9030 -h $MYSQL_IP -u root --prompt="StarRocks > " -p ``` *** #### Create some tables[​](#create-some-tables "Direct link to Create some tables") ```bash mysql -P9030 -h $MYSQL_IP -u root --prompt="StarRocks > " -p ``` ##### Create a database[​](#create-a-database "Direct link to Create a database") Type these two lines in at the `StarRocks > `prompt and press enter after each: ```sql CREATE DATABASE IF NOT EXISTS quickstart; USE quickstart; ``` ##### Create two tables[​](#create-two-tables "Direct link to Create two tables") ###### Crashdata[​](#crashdata "Direct link to Crashdata") The crash dataset contains many more fields than these, the schema has been trimmed down to include only the fields that might be useful to answer questions about the impact weather has on driving conditions. ```sql CREATE TABLE IF NOT EXISTS crashdata ( CRASH_DATE DATETIME, BOROUGH STRING, ZIP_CODE STRING, LATITUDE INT, LONGITUDE INT, LOCATION STRING, ON_STREET_NAME STRING, CROSS_STREET_NAME STRING, OFF_STREET_NAME STRING, CONTRIBUTING_FACTOR_VEHICLE_1 STRING, CONTRIBUTING_FACTOR_VEHICLE_2 STRING, COLLISION_ID INT, VEHICLE_TYPE_CODE_1 STRING, VEHICLE_TYPE_CODE_2 STRING ); ``` ###### Weatherdata[​](#weatherdata "Direct link to Weatherdata") Similar to the crash data, the weather dataset has many more columns (a total of 125 columns) and only the ones that are expected to answer the questions are included in the database. ```sql CREATE TABLE IF NOT EXISTS weatherdata ( DATE DATETIME, NAME STRING, HourlyDewPointTemperature STRING, HourlyDryBulbTemperature STRING, HourlyPrecipitation STRING, HourlyPresentWeatherType STRING, HourlyPressureChange STRING, HourlyPressureTendency STRING, HourlyRelativeHumidity STRING, HourlySkyConditions STRING, HourlyVisibility STRING, HourlyWetBulbTemperature STRING, HourlyWindDirection STRING, HourlyWindGustSpeed STRING, HourlyWindSpeed STRING ); ``` Exit from the MySQL client, or open a new shell to run commands at the command line to upload data. ```sql exit ``` #### Upload data[​](#upload-data "Direct link to Upload data") There are many ways to load data into StarRocks. For this tutorial, the simplest way is to use curl and StarRocks Stream Load. Upload the two datasets that you downloaded earlier. tip Open a new shell as these curl commands are run at the operating system prompt, not in the `mysql` client. The commands refer to the datasets that you downloaded, so run them from the directory where you downloaded the files. Since this is a new shell, run the export commands again: ```bash export MYSQL_IP=`kubectl get services kube-starrocks-fe-service --output jsonpath='{.status.loadBalancer.ingress[0].ip}'` export FE_PROXY=`kubectl get services kube-starrocks-fe-proxy-service --output jsonpath='{.status.loadBalancer.ingress[0].ip}'`:8080 ``` You will be prompted for a password. Use the password that you added to the Kubernetes secret `starrocks-root-pass`. If you used the command provided, the password is `g()()dpa$$word`. The `curl` commands look complex, but they are explained in detail at the end of the tutorial. For now, we recommend running the commands and running some SQL to analyze the data, and then reading about the data loading details at the end. ```bash curl --location-trusted -u root \ -T ./NYPD_Crash_Data.csv \ -H "label:crashdata-0" \ -H "column_separator:," \ -H "skip_header:1" \ -H "enclose:\"" \ -H "max_filter_ratio:1" \ -H "columns:tmp_CRASH_DATE, tmp_CRASH_TIME, CRASH_DATE=str_to_date(concat_ws(' ', tmp_CRASH_DATE, tmp_CRASH_TIME), '%m/%d/%Y %H:%i'),BOROUGH,ZIP_CODE,LATITUDE,LONGITUDE,LOCATION,ON_STREET_NAME,CROSS_STREET_NAME,OFF_STREET_NAME,NUMBER_OF_PERSONS_INJURED,NUMBER_OF_PERSONS_KILLED,NUMBER_OF_PEDESTRIANS_INJURED,NUMBER_OF_PEDESTRIANS_KILLED,NUMBER_OF_CYCLIST_INJURED,NUMBER_OF_CYCLIST_KILLED,NUMBER_OF_MOTORIST_INJURED,NUMBER_OF_MOTORIST_KILLED,CONTRIBUTING_FACTOR_VEHICLE_1,CONTRIBUTING_FACTOR_VEHICLE_2,CONTRIBUTING_FACTOR_VEHICLE_3,CONTRIBUTING_FACTOR_VEHICLE_4,CONTRIBUTING_FACTOR_VEHICLE_5,COLLISION_ID,VEHICLE_TYPE_CODE_1,VEHICLE_TYPE_CODE_2,VEHICLE_TYPE_CODE_3,VEHICLE_TYPE_CODE_4,VEHICLE_TYPE_CODE_5" \ -XPUT http://$FE_PROXY/api/quickstart/crashdata/_stream_load ``` ```text Enter host password for user 'root': { "TxnId": 2, "Label": "crashdata-0", "Status": "Success", "Message": "OK", "NumberTotalRows": 423726, "NumberLoadedRows": 423725, "NumberFilteredRows": 1, "NumberUnselectedRows": 0, "LoadBytes": 96227746, "LoadTimeMs": 2483, "BeginTxnTimeMs": 42, "StreamLoadPlanTimeMs": 122, "ReadDataTimeMs": 1610, "WriteDataTimeMs": 2253, "CommitAndPublishTimeMs": 65, "ErrorURL": "http://kube-starrocks-be-2.kube-starrocks-be-search.default.svc.cluster.local:8040/api/_load_error_log?file=error_log_5149e6f80de42bcb_eab2ea77276de4ba" } ``` ```bash curl --location-trusted -u root \ -T ./72505394728.csv \ -H "label:weather-0" \ -H "column_separator:," \ -H "skip_header:1" \ -H "enclose:\"" \ -H "max_filter_ratio:1" \ -H "columns: STATION, DATE, LATITUDE, LONGITUDE, ELEVATION, NAME, REPORT_TYPE, SOURCE, HourlyAltimeterSetting, HourlyDewPointTemperature, HourlyDryBulbTemperature, HourlyPrecipitation, HourlyPresentWeatherType, HourlyPressureChange, HourlyPressureTendency, HourlyRelativeHumidity, HourlySkyConditions, HourlySeaLevelPressure, HourlyStationPressure, HourlyVisibility, HourlyWetBulbTemperature, HourlyWindDirection, HourlyWindGustSpeed, HourlyWindSpeed, Sunrise, Sunset, DailyAverageDewPointTemperature, DailyAverageDryBulbTemperature, DailyAverageRelativeHumidity, DailyAverageSeaLevelPressure, DailyAverageStationPressure, DailyAverageWetBulbTemperature, DailyAverageWindSpeed, DailyCoolingDegreeDays, DailyDepartureFromNormalAverageTemperature, DailyHeatingDegreeDays, DailyMaximumDryBulbTemperature, DailyMinimumDryBulbTemperature, DailyPeakWindDirection, DailyPeakWindSpeed, DailyPrecipitation, DailySnowDepth, DailySnowfall, DailySustainedWindDirection, DailySustainedWindSpeed, DailyWeather, MonthlyAverageRH, MonthlyDaysWithGT001Precip, MonthlyDaysWithGT010Precip, MonthlyDaysWithGT32Temp, MonthlyDaysWithGT90Temp, MonthlyDaysWithLT0Temp, MonthlyDaysWithLT32Temp, MonthlyDepartureFromNormalAverageTemperature, MonthlyDepartureFromNormalCoolingDegreeDays, MonthlyDepartureFromNormalHeatingDegreeDays, MonthlyDepartureFromNormalMaximumTemperature, MonthlyDepartureFromNormalMinimumTemperature, MonthlyDepartureFromNormalPrecipitation, MonthlyDewpointTemperature, MonthlyGreatestPrecip, MonthlyGreatestPrecipDate, MonthlyGreatestSnowDepth, MonthlyGreatestSnowDepthDate, MonthlyGreatestSnowfall, MonthlyGreatestSnowfallDate, MonthlyMaxSeaLevelPressureValue, MonthlyMaxSeaLevelPressureValueDate, MonthlyMaxSeaLevelPressureValueTime, MonthlyMaximumTemperature, MonthlyMeanTemperature, MonthlyMinSeaLevelPressureValue, MonthlyMinSeaLevelPressureValueDate, MonthlyMinSeaLevelPressureValueTime, MonthlyMinimumTemperature, MonthlySeaLevelPressure, MonthlyStationPressure, MonthlyTotalLiquidPrecipitation, MonthlyTotalSnowfall, MonthlyWetBulb, AWND, CDSD, CLDD, DSNW, HDSD, HTDD, NormalsCoolingDegreeDay, NormalsHeatingDegreeDay, ShortDurationEndDate005, ShortDurationEndDate010, ShortDurationEndDate015, ShortDurationEndDate020, ShortDurationEndDate030, ShortDurationEndDate045, ShortDurationEndDate060, ShortDurationEndDate080, ShortDurationEndDate100, ShortDurationEndDate120, ShortDurationEndDate150, ShortDurationEndDate180, ShortDurationPrecipitationValue005, ShortDurationPrecipitationValue010, ShortDurationPrecipitationValue015, ShortDurationPrecipitationValue020, ShortDurationPrecipitationValue030, ShortDurationPrecipitationValue045, ShortDurationPrecipitationValue060, ShortDurationPrecipitationValue080, ShortDurationPrecipitationValue100, ShortDurationPrecipitationValue120, ShortDurationPrecipitationValue150, ShortDurationPrecipitationValue180, REM, BackupDirection, BackupDistance, BackupDistanceUnit, BackupElements, BackupElevation, BackupEquipment, BackupLatitude, BackupLongitude, BackupName, WindEquipmentChangeDate" \ -XPUT http://$FE_PROXY/api/quickstart/weatherdata/_stream_load ``` ```text Enter host password for user 'root': { "TxnId": 4, "Label": "weather-0", "Status": "Success", "Message": "OK", "NumberTotalRows": 22931, "NumberLoadedRows": 22931, "NumberFilteredRows": 0, "NumberUnselectedRows": 0, "LoadBytes": 15558550, "LoadTimeMs": 404, "BeginTxnTimeMs": 1, "StreamLoadPlanTimeMs": 7, "ReadDataTimeMs": 157, "WriteDataTimeMs": 372, "CommitAndPublishTimeMs": 23 } ``` #### Connect with a MySQL client[​](#connect-with-a-mysql-client "Direct link to Connect with a MySQL client") Connect with a MySQL client if you are not connected. Remember to use the external IP address of the `kube-starrocks-fe-service` service and the password that you configured in the Kubernetes secret `starrocks-root-pass`. ```bash mysql -P9030 -h $MYSQL_IP -u root --prompt="StarRocks > " -p ``` #### Answer some questions[​](#answer-some-questions "Direct link to Answer some questions") These queries can be run in your SQL client. All of the queries use the `quickstart` database. ```sql USE quickstart; ``` ###### How many crashes are there per hour in NYC?[​](#how-many-crashes-are-there-per-hour-in-nyc "Direct link to How many crashes are there per hour in NYC?") ```sql SELECT COUNT(*), date_trunc("hour", crashdata.CRASH_DATE) AS Time FROM crashdata GROUP BY Time ORDER BY Time ASC LIMIT 200; ``` Here is part of the output. Note that I am looking closer at January 6th and 7th as this is Monday and Tuesday of a non-holiday week. Looking at New Years Day is probably not indicative of a normal morning during rush-hour traffic. ```plaintext | 14 | 2014-01-06 06:00:00 | | 16 | 2014-01-06 07:00:00 | | 43 | 2014-01-06 08:00:00 | | 44 | 2014-01-06 09:00:00 | | 21 | 2014-01-06 10:00:00 | | 28 | 2014-01-06 11:00:00 | | 34 | 2014-01-06 12:00:00 | | 31 | 2014-01-06 13:00:00 | | 35 | 2014-01-06 14:00:00 | | 36 | 2014-01-06 15:00:00 | | 33 | 2014-01-06 16:00:00 | | 40 | 2014-01-06 17:00:00 | | 35 | 2014-01-06 18:00:00 | | 23 | 2014-01-06 19:00:00 | | 16 | 2014-01-06 20:00:00 | | 12 | 2014-01-06 21:00:00 | | 17 | 2014-01-06 22:00:00 | | 14 | 2014-01-06 23:00:00 | | 10 | 2014-01-07 00:00:00 | | 4 | 2014-01-07 01:00:00 | | 1 | 2014-01-07 02:00:00 | | 3 | 2014-01-07 03:00:00 | | 2 | 2014-01-07 04:00:00 | | 6 | 2014-01-07 06:00:00 | | 16 | 2014-01-07 07:00:00 | | 41 | 2014-01-07 08:00:00 | | 37 | 2014-01-07 09:00:00 | | 33 | 2014-01-07 10:00:00 | ``` It looks like about 40 accidents on a Monday or Tuesday morning during rush hour traffic, and around the same at 17:00 hours. ###### What is the average temperature in NYC?[​](#what-is-the-average-temperature-in-nyc "Direct link to What is the average temperature in NYC?") ```sql SELECT avg(HourlyDryBulbTemperature), date_trunc("hour", weatherdata.DATE) AS Time FROM weatherdata GROUP BY Time ORDER BY Time ASC LIMIT 100; ``` Output: Note that this is data from 2014, NYC has not been this cold lately. ```plaintext +-------------------------------+---------------------+ | avg(HourlyDryBulbTemperature) | Time | +-------------------------------+---------------------+ | 25 | 2014-01-01 00:00:00 | | 25 | 2014-01-01 01:00:00 | | 24 | 2014-01-01 02:00:00 | | 24 | 2014-01-01 03:00:00 | | 24 | 2014-01-01 04:00:00 | | 24 | 2014-01-01 05:00:00 | | 25 | 2014-01-01 06:00:00 | | 26 | 2014-01-01 07:00:00 | ``` ###### Is it safe to drive in NYC when visibility is poor?[​](#is-it-safe-to-drive-in-nyc-when-visibility-is-poor "Direct link to Is it safe to drive in NYC when visibility is poor?") Let's look at the number of crashes when visibility is poor (between 0 and 1.0 miles). To answer this question use a JOIN across the two tables on the DATETIME column. ```sql SELECT COUNT(DISTINCT c.COLLISION_ID) AS Crashes, truncate(avg(w.HourlyDryBulbTemperature), 1) AS Temp_F, truncate(avg(w.HourlyVisibility), 2) AS Visibility, max(w.HourlyPrecipitation) AS Precipitation, date_format((date_trunc("hour", c.CRASH_DATE)), '%d %b %Y %H:%i') AS Hour FROM crashdata c LEFT JOIN weatherdata w ON date_trunc("hour", c.CRASH_DATE)=date_trunc("hour", w.DATE) WHERE w.HourlyVisibility BETWEEN 0.0 AND 1.0 GROUP BY Hour ORDER BY Crashes DESC LIMIT 100; ``` The highest number of crashes in a single hour during low visibility is 129. There are multiple things to consider: * February 3rd 2014 was a Monday * 8AM is rush hour * It was raining (0.12 inches or precipitation that hour) * The temperature is 32 degrees F (the freezing point for water) * Visibility is bad at 0.25 miles, normal for NYC is 10 miles ```plaintext +---------+--------+------------+---------------+-------------------+ | Crashes | Temp_F | Visibility | Precipitation | Hour | +---------+--------+------------+---------------+-------------------+ | 129 | 32 | 0.25 | 0.12 | 03 Feb 2014 08:00 | | 114 | 32 | 0.25 | 0.12 | 03 Feb 2014 09:00 | | 104 | 23 | 0.33 | 0.03 | 09 Jan 2015 08:00 | | 96 | 26.3 | 0.33 | 0.07 | 01 Mar 2015 14:00 | | 95 | 26 | 0.37 | 0.12 | 01 Mar 2015 15:00 | | 93 | 35 | 0.75 | 0.09 | 18 Jan 2015 09:00 | | 92 | 31 | 0.25 | 0.12 | 03 Feb 2014 10:00 | | 87 | 26.8 | 0.5 | 0.09 | 01 Mar 2015 16:00 | | 85 | 55 | 0.75 | 0.20 | 23 Dec 2015 17:00 | | 85 | 20 | 0.62 | 0.01 | 06 Jan 2015 11:00 | | 83 | 19.6 | 0.41 | 0.04 | 05 Mar 2015 13:00 | | 80 | 20 | 0.37 | 0.02 | 06 Jan 2015 10:00 | | 76 | 26.5 | 0.25 | 0.06 | 05 Mar 2015 09:00 | | 71 | 26 | 0.25 | 0.09 | 05 Mar 2015 10:00 | | 71 | 24.2 | 0.25 | 0.04 | 05 Mar 2015 11:00 | ``` ###### What about driving in icy conditions?[​](#what-about-driving-in-icy-conditions "Direct link to What about driving in icy conditions?") Water vapor can desublimate to ice at 40 degrees F; this query looks at temps between 0 and 40 degrees F. ```sql SELECT COUNT(DISTINCT c.COLLISION_ID) AS Crashes, truncate(avg(w.HourlyDryBulbTemperature), 1) AS Temp_F, truncate(avg(w.HourlyVisibility), 2) AS Visibility, max(w.HourlyPrecipitation) AS Precipitation, date_format((date_trunc("hour", c.CRASH_DATE)), '%d %b %Y %H:%i') AS Hour FROM crashdata c LEFT JOIN weatherdata w ON date_trunc("hour", c.CRASH_DATE)=date_trunc("hour", w.DATE) WHERE w.HourlyDryBulbTemperature BETWEEN 0.0 AND 40.5 GROUP BY Hour ORDER BY Crashes DESC LIMIT 100; ``` The results for freezing temperatures suprised me a little, I did not expect too much traffic on a Sunday morning in the city on a cold January day.A quick look at [weather.com](https://weather.com/storms/winter/news/northeast-storm-rain-snow-wind) showed that there was a big storm with many crashes that day, just like what can be seen in the data. ```plaintext +---------+--------+------------+---------------+-------------------+ | Crashes | Temp_F | Visibility | Precipitation | Hour | +---------+--------+------------+---------------+-------------------+ | 192 | 34 | 1.5 | 0.09 | 18 Jan 2015 08:00 | | 170 | 21 | NULL | | 21 Jan 2014 10:00 | | 145 | 19 | NULL | | 21 Jan 2014 11:00 | | 138 | 33.5 | 5 | 0.02 | 18 Jan 2015 07:00 | | 137 | 21 | NULL | | 21 Jan 2014 09:00 | | 129 | 32 | 0.25 | 0.12 | 03 Feb 2014 08:00 | | 114 | 32 | 0.25 | 0.12 | 03 Feb 2014 09:00 | | 104 | 23 | 0.7 | 0.04 | 09 Jan 2015 08:00 | | 98 | 16 | 8 | 0.00 | 06 Mar 2015 08:00 | | 96 | 26.3 | 0.33 | 0.07 | 01 Mar 2015 14:00 | ``` Drive carefully! ```sql exit ``` #### Cleanup[​](#cleanup "Direct link to Cleanup") Run this command if you are finished and would like to remove the StarRocks cluster and the StarRocks operator. ```bash helm delete starrocks ``` *** #### Summary[​](#summary "Direct link to Summary") In this tutorial you: * Deployed StarRocks with Helm and the StarRocks Operator * Loaded crash data provided by New York City and weather data provided by NOAA * Analyzed the data using SQL JOINs to find out that driving in low visibility or icy streets is a bad idea There is more to learn; we intentionally glossed over the data transformation done during the Stream Load. The details on that are in the notes on the curl commands below. *** #### Notes on the curl commands[​](#notes-on-the-curl-commands "Direct link to Notes on the curl commands") StarRocks Stream Load and curl take many arguments. Only the ones used in this tutorial are described here, the rest will be linked to in the more information section. ###### `--location-trusted`[​](#--location-trusted "Direct link to --location-trusted") This configures curl to pass credentials to any redirected URLs. ###### `-u root`[​](#-u-root "Direct link to -u-root") The username used to log in to StarRocks ###### `-T filename`[​](#-t-filename "Direct link to -t-filename") T is for transfer, the filename to transfer. ###### `label:name-num`[​](#labelname-num "Direct link to labelname-num") The label to associate with this Stream Load job. The label must be unique, so if you run the job multiple times you can add a number and keep incrementing that. ###### `column_separator:,`[​](#column_separator "Direct link to column_separator") If you load a file that uses a single `,` then set it as shown above, if you use a different delimiter then set that delimiter here. Common choices are `\t`, `,`, and `|`. ###### `skip_header:1`[​](#skip_header1 "Direct link to skip_header1") Some CSV files have a single header row with all of the column names listed, and some add a second line with datatypes. Set skip\_header to `1` or `2` if you have one or two header lines, and set it to `0` if you have none. ###### `enclose:\"`[​](#enclose "Direct link to enclose") It is common to enclose strings that contain embedded commas with double-quotes. The sample datasets used in this tutorial have geo locations that contain commas and so the enclose setting is set to `\"`. Remember to escape the `"` with a `\`. ###### `max_filter_ratio:1`[​](#max_filter_ratio1 "Direct link to max_filter_ratio1") This allows some errors in the data. Ideally this would be set to `0` and the job would fail with any errors. It is set to `1` to allow all rows to fail during debugging. ###### `columns:`[​](#columns "Direct link to columns") The mapping of CSV file columns to StarRocks table columns. You will notice that there are many more columns in the CSV files than columns in the table. Any columns that are not included in the table are skipped. You will also notice that there is some transformation of data included in the `columns:` line for the crash dataset. It is very common to find dates and times in CSV files that do not conform to standards. This is the logic for converting the CSV data for the time and date of the crash to a DATETIME type: ###### The columns line[​](#the-columns-line "Direct link to The columns line") This is the beginning of one data record. The date is in `MM/DD/YYYY` format, and the time is `HH:MI`. Since DATETIME is generally `YYYY-MM-DD HH:MI:SS` we need to transform this data. ```plaintext 08/05/2014,9:10,BRONX,10469,40.8733019,-73.8536375,"(40.8733019, -73.8536375)", ``` This is the beginning of the `columns:` parameter: ```bash -H "columns:tmp_CRASH_DATE, tmp_CRASH_TIME, CRASH_DATE=str_to_date(concat_ws(' ', tmp_CRASH_DATE, tmp_CRASH_TIME), '%m/%d/%Y %H:%i') ``` This instructs StarRocks to: * Assign the content of the first column of the CSV file to `tmp_CRASH_DATE` * Assign the content of the second column of the CSV file to `tmp_CRASH_TIME` * `concat_ws()` concatenates `tmp_CRASH_DATE` and `tmp_CRASH_TIME` together with a space between them * `str_to_date()` creates a DATETIME from the concatenated string * store the resulting DATETIME in the column `CRASH_DATE` *** #### More information[​](#more-information "Direct link to More information") Default [`values.yaml`](https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/helm-charts/charts/kube-starrocks/values.yaml) [Stream Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md) The [Motor Vehicle Collisions - Crashes](https://data.cityofnewyork.us/Public-Safety/Motor-Vehicle-Collisions-Crashes/h9gi-nx95) dataset is provided by New York City subject to these [terms of use](https://www.nyc.gov/home/terms-of-use.page) and [privacy policy](https://www.nyc.gov/home/privacy-policy.page). The [Local Climatological Data](https://www.ncdc.noaa.gov/cdo-web/datatools/lcd)(LCD) is provided by NOAA with this [disclaimer](https://www.noaa.gov/disclaimer) and this [privacy policy](https://www.noaa.gov/protecting-your-privacy). [Helm](https://helm.sh/) is a package manager for Kubernetes. A [Helm Chart](https://helm.sh/docs/topics/charts/) is a Helm package and contains all of the resource definitions necessary to run an application on a Kubernetes cluster. [`starrocks-kubernetes-operator` and `kube-starrocks` Helm Chart](https://github.com/StarRocks/starrocks-kubernetes-operator). --- ### Apache Hudi Lakehouse #### Overview[​](#overview "Direct link to Overview") * Deploy Object Storage, Apache Spark, Hudi, and StarRocks using Docker compose * Load a tiny dataset into Hudi with Apache Spark * Configure StarRocks to access the Hive Metastore using an external catalog * Query the data with StarRocks where the data sits ![DLA](/assets/images/1.1-8-dla-c67d601d709b092317fa11eb64ac1783.png) In addition to efficient analytics of local data, StarRocks can work as the compute engine to analyze data stored in data lakes such as Apache Hudi, Apache Iceberg, and Delta Lake. One of the key features of StarRocks is its external catalog, which acts as the linkage to an externally maintained metastore. This functionality provides users with the capability to query external data sources seamlessly, eliminating the need for data migration. As such, users can analyze data from different systems such as HDFS and Amazon S3, in various file formats such as Parquet, ORC, and CSV, etc. The preceding figure shows a data lake analytics scenario where StarRocks is responsible for data computing and analysis, and the data lake is responsible for data storage, organization, and maintenance. Data lakes allow users to store data in open storage formats and use flexible schemas to produce reports on "single source of truth" for various BI, AI, ad-hoc, and reporting use cases. StarRocks fully leverages the advantages of its vectorization engine and CBO, significantly improving the performance of data lake analytics. #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") ##### StarRocks `demo` repository[​](#starrocks-demo-repository "Direct link to starrocks-demo-repository") Clone the [StarRocks demo repository](https://github.com/StarRocks/demo/) to your local machine. All the steps in this guide will be run from the `demo/documentation-samples/hudi/` directory in the directory where you cloned the `demo` GitHub repo. ##### Docker[​](#docker "Direct link to Docker") * Docker Setup: For Mac, Please follow the steps as defined in [Install Docker Desktop on Mac](https://docs.docker.com/desktop/setup/install/mac-install/). For running Spark-SQL queries, please ensure at least 5 GB memory and 4 CPUs are allocated to Docker (See Docker → Preferences → Advanced). Otherwise, spark-SQL queries could be killed because of memory issues. * 20 GB free disk space assigned to Docker ##### SQL client[​](#sql-client "Direct link to SQL client") You can use the SQL client provided in the Docker environment, or use one on your system. Many MySQL compatible clients will work. #### Configuration[​](#configuration "Direct link to Configuration") Change directory into `demo/documentation-samples/hudi` and look at the files. This is not a tutorial on Hudi, so not every configuration file will be described; but it is important for the reader to know where to look to see how things are configured. In the `hudi/` directory you will find the `docker-compose.yml` file which is used to launch and configure the services in Docker. Here is a list of those services and a brief description: ##### Docker services[​](#docker-services "Direct link to Docker services") | Service | Responsibilities | | ------------------------ | ------------------------------------------------------------------- | | **`starrocks-fe`** | Metadata management, client connections, query plans and scheduling | | **`starrocks-be`** | Running query plans | | **`metastore_db`** | Postgres DB used to store the Hive metadata | | **`hive_metastore`** | Provides the Apache Hive metastore | | **`minio`** and **`mc`** | MinIO Object Storage and MinIO command line client | | **`spark-hudi`** | Distributed computing and Transactional data lake platform | ##### Configuration files[​](#configuration-files "Direct link to Configuration files") In the `hudi/conf/` directory you will find configuration files that get mounted in the `spark-hudi` container. ###### `core-site.xml`[​](#core-sitexml "Direct link to core-sitexml") This file contains the object storage related settings. Links for this and other items in More information at the end of this document. ###### `spark-defaults.conf`[​](#spark-defaultsconf "Direct link to spark-defaultsconf") Settings for Hive, MinIO, and Spark SQL. ###### `hudi-defaults.conf`[​](#hudi-defaultsconf "Direct link to hudi-defaultsconf") Default file used to silence warnings in the `spark-shell`. ###### `hadoop-metrics2-hbase.properties`[​](#hadoop-metrics2-hbaseproperties "Direct link to hadoop-metrics2-hbaseproperties") Empty file used to silence warnings in the `spark-shell`. ###### `hadoop-metrics2-s3a-file-system.properties`[​](#hadoop-metrics2-s3a-file-systemproperties "Direct link to hadoop-metrics2-s3a-file-systemproperties") Empty file used to silence warnings in the `spark-shell`. #### Bringing up Demo Cluster[​](#bringing-up-demo-cluster "Direct link to Bringing up Demo Cluster") This demo system consists of StarRocks, Hudi, MinIO, and Spark services. Run Docker compose to bring up the cluster: ```bash docker compose up --detach --wait --wait-timeout 60 ``` ```plaintext [+] Running 8/8 ✔ Network hudi Created 0.0s ✔ Container hudi-starrocks-fe-1 Healthy 0.1s ✔ Container hudi-minio-1 Healthy 0.1s ✔ Container hudi-metastore_db-1 Healthy 0.1s ✔ Container hudi-starrocks-be-1 Healthy 0.0s ✔ Container hudi-mc-1 Healthy 0.0s ✔ Container hudi-hive-metastore-1 Healthy 0.0s ✔ Container hudi-spark-hudi-1 Healthy 0.1s ``` tip With many containers running, `docker compose ps` output is easier to read if you pipe it to `jq`: ```bash docker compose ps --format json | \ jq '{Service: .Service, State: .State, Status: .Status}' ``` ```json { "Service": "hive-metastore", "State": "running", "Status": "Up About a minute (healthy)" } { "Service": "mc", "State": "running", "Status": "Up About a minute" } { "Service": "metastore_db", "State": "running", "Status": "Up About a minute" } { "Service": "minio", "State": "running", "Status": "Up About a minute" } { "Service": "spark-hudi", "State": "running", "Status": "Up 33 seconds (healthy)" } { "Service": "starrocks-be", "State": "running", "Status": "Up About a minute (healthy)" } { "Service": "starrocks-fe", "State": "running", "Status": "Up About a minute (healthy)" } ``` #### Configure MinIO[​](#configure-minio "Direct link to Configure MinIO") When you run the Spark commands you will set the basepath for the table being created to an `s3a` URI: ```java val basePath = "s3a://huditest/hudi_coders" ``` In this step you will create the bucket `huditest` in MinIO. The MinIO console is running on port `9000`. ##### Authenticate to MinIO[​](#authenticate-to-minio "Direct link to Authenticate to MinIO") Open a browser to and authenticate. The username and password are specified in `docker-compose.yml`; they are `admin` and `password`. ##### Create a bucket[​](#create-a-bucket "Direct link to Create a bucket") In the left navigation select **Buckets**, and then **Create Bucket +**. Name the bucket `huditest` and select **Create Bucket** ![Create bucket huditest](/assets/images/hudi-test-bucket-7e936058aca4001f97f613ee96f6c7d7.png) #### Create and populate a table, then sync it to Hive[​](#create-and-populate-a-table-then-sync-it-to-hive "Direct link to Create and populate a table, then sync it to Hive") tip Run this command, and any other `docker compose` commands, from the directory containing the `docker-compose.yml` file. Open `spark-shell` in the `spark-hudi` service ```bash docker compose exec spark-hudi spark-shell ``` note There will be warnings when `spark-shell` starts about illegal reflective access. You can ignore these warnings. Run these commands at the `scala>` prompt to: * Configure this Spark session to load, process, and write data * Create a dataframe and write that to a Hudi table * Sync to the Hive Metastore ```scala import org.apache.spark.sql.functions._ import org.apache.spark.sql.types._ import org.apache.spark.sql.Row import org.apache.spark.sql.SaveMode._ import org.apache.hudi.DataSourceReadOptions._ import org.apache.hudi.DataSourceWriteOptions._ import org.apache.hudi.config.HoodieWriteConfig._ import scala.collection.JavaConversions._ val schema = StructType( Array( StructField("language", StringType, true), StructField("users", StringType, true), StructField("id", StringType, true) )) val rowData= Seq(Row("Java", "20000", "a"), Row("Python", "100000", "b"), Row("Scala", "3000", "c")) val df = spark.createDataFrame(rowData,schema) val databaseName = "hudi_sample" val tableName = "hudi_coders_hive" val basePath = "s3a://huditest/hudi_coders" df.write.format("hudi"). option(org.apache.hudi.config.HoodieWriteConfig.TABLE_NAME, tableName). option(RECORDKEY_FIELD_OPT_KEY, "id"). option(PARTITIONPATH_FIELD_OPT_KEY, "language"). option(PRECOMBINE_FIELD_OPT_KEY, "users"). option("hoodie.datasource.write.hive_style_partitioning", "true"). option("hoodie.datasource.hive_sync.enable", "true"). option("hoodie.datasource.hive_sync.mode", "hms"). option("hoodie.datasource.hive_sync.database", databaseName). option("hoodie.datasource.hive_sync.table", tableName). option("hoodie.datasource.hive_sync.partition_fields", "language"). option("hoodie.datasource.hive_sync.partition_extractor_class", "org.apache.hudi.hive.MultiPartKeysValueExtractor"). option("hoodie.datasource.hive_sync.metastore.uris", "thrift://hive-metastore:9083"). mode(Overwrite). save(basePath) System.exit(0) ``` note You will see a warning: ```java WARN org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not found at path s3a://huditest/hudi_coders/.hoodie/metadata ``` This can be ignored, the file will be created automatically during this `spark-shell` session. There will also be a warning: ```bash 78184 [main] WARN org.apache.hadoop.fs.s3a.S3ABlockOutputStream - Application invoked the Syncable API against stream writing to hudi_coders/.hoodie/metadata/files/.files-0000_00000000000000.log.1_0-0-0. This is unsupported ``` This warning informs you that syncing a log file that is open for writes is not supported when using object storage. The file will only be synced when it is closed. See [Stack Overflow](https://stackoverflow.com/a/74886836/10424890). The final command in the above spark-shell session should exit the container, if it doesn't press enter and it will exit. #### Configure StarRocks[​](#configure-starrocks "Direct link to Configure StarRocks") ##### Connect to StarRocks[​](#connect-to-starrocks "Direct link to Connect to StarRocks") Connect to StarRocks with the provided MySQL client provided by the `starrocks-fe` service, or use your favorite SQL client and configure it to connect using the MySQL protocol on `localhost:9030`. ```bash docker compose exec starrocks-fe \ mysql -P 9030 -h 127.0.0.1 -u root --prompt="StarRocks > " ``` ##### Create the linkage between StarRocks and Hudi[​](#create-the-linkage-between-starrocks-and-hudi "Direct link to Create the linkage between StarRocks and Hudi") There is a link at the end of this guide with more information on external catalogs. The external catalog created in this step acts as the linkage to the Hive Metastore (HMS) running in Docker. ```sql CREATE EXTERNAL CATALOG hudi_catalog_hms PROPERTIES ( "type" = "hudi", "hive.metastore.type" = "hive", "hive.metastore.uris" = "thrift://hive-metastore:9083", "aws.s3.use_instance_profile" = "false", "aws.s3.access_key" = "admin", "aws.s3.secret_key" = "password", "aws.s3.enable_ssl" = "false", "aws.s3.enable_path_style_access" = "true", "aws.s3.endpoint" = "http://minio:9000" ); ``` ```plaintext Query OK, 0 rows affected (0.59 sec) ``` ##### Use the new catalog[​](#use-the-new-catalog "Direct link to Use the new catalog") ```sql SET CATALOG hudi_catalog_hms; ``` ```plaintext Query OK, 0 rows affected (0.01 sec) ``` ##### Navigate to the data inserted with Spark[​](#navigate-to-the-data-inserted-with-spark "Direct link to Navigate to the data inserted with Spark") ```sql SHOW DATABASES; ``` ```plaintext +--------------------+ | Database | +--------------------+ | default | | hudi_sample | | information_schema | +--------------------+ 2 rows in set (0.40 sec) ``` ```sql USE hudi_sample; ``` ```plaintext Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed ``` ```sql SHOW TABLES; ``` ```plaintext +-----------------------+ | Tables_in_hudi_sample | +-----------------------+ | hudi_coders_hive | +-----------------------+ 1 row in set (0.07 sec) ``` ##### Query the data in Hudi with StarRocks[​](#query-the-data-in-hudi-with-starrocks "Direct link to Query the data in Hudi with StarRocks") Run this query twice, the first time may take around five seconds to complete as data is not yet cached in StarRocks. The second query will be very quick. ```sql SELECT * from hudi_coders_hive\G ``` tip Some of the SQL queries in the StarRocks documentation end with `\G` instead of a semicolon. The `\G` causes the mysql CLI to render the query results vertically. Many SQL clients do not interpret vertical formatting output, so you should replace `\G` with `;` if you are not using the mysql CLI. ```plaintext *************************** 1. row *************************** _hoodie_commit_time: 20240208165522561 _hoodie_commit_seqno: 20240208165522561_0_0 _hoodie_record_key: c _hoodie_partition_path: language=Scala _hoodie_file_name: bb29249a-b69d-4c32-843b-b7142d8dc51c-0_0-27-1221_20240208165522561.parquet language: Scala users: 3000 id: c *************************** 2. row *************************** _hoodie_commit_time: 20240208165522561 _hoodie_commit_seqno: 20240208165522561_2_0 _hoodie_record_key: a _hoodie_partition_path: language=Java _hoodie_file_name: 12fc14aa-7dc4-454c-b710-1ad0556c9386-0_2-27-1223_20240208165522561.parquet language: Java users: 20000 id: a *************************** 3. row *************************** _hoodie_commit_time: 20240208165522561 _hoodie_commit_seqno: 20240208165522561_1_0 _hoodie_record_key: b _hoodie_partition_path: language=Python _hoodie_file_name: 51977039-d71e-4dd6-90d4-0c93656dafcf-0_1-27-1222_20240208165522561.parquet language: Python users: 100000 id: b 3 rows in set (0.15 sec) ``` #### Summary[​](#summary "Direct link to Summary") This tutorial exposed you to the use of a StarRocks external catalog to show you that you can query your data where it sits using the Hudi external catalog. Many other integrations are available using Iceberg, Delta Lake, and JDBC catalogs. In this tutorial you: * Deployed StarRocks and a Hudi/Spark/MinIO environment in Docker * Loaded a tiny dataset into Hudi with Apache Spark * Configured a StarRocks external catalog to provide access to the Hudi catalog * Queried the data with SQL in StarRocks without copying the data from the data lake #### More information[​](#more-information "Direct link to More information") [StarRocks Catalogs](https://docs.starrocks.io/docs/data_source/catalog/catalog_overview.md) [Apache Hudi quickstart](https://hudi.apache.org/docs/quick-start-guide/) (includes Spark) [Apache Hudi S3 configuration](https://hudi.apache.org/docs/s3_hoodie/) [Apache Spark configuration docs](https://spark.apache.org/docs/latest/configuration.html) --- ### Apache Iceberg Lakehouse This guide will get you up and running with Apache Iceberg™ using StarRocks™, including sample code to highlight some powerful features. ##### Docker-Compose[​](#docker-compose "Direct link to Docker-Compose") The fastest way to get started is to use a docker-compose file that uses the `starrocks/fe-ubuntu` and `starrocks/be-ubuntu` images which contain a local StarRocks cluster with a configured Iceberg catalog. To use this, you'll need to install the Docker CLI. Once you have Docker installed, save the yaml below into a file named docker-compose.yml: ```yml services: starrocks-fe: image: starrocks/fe-ubuntu:4.0-latest hostname: starrocks-fe container_name: starrocks-fe user: root command: | bash /opt/starrocks/fe/bin/start_fe.sh --host_type FQDN ports: - 8030:8030 - 9020:9020 - 9030:9030 networks: iceberg_net: environment: - AWS_ACCESS_KEY_ID=admin - AWS_SECRET_ACCESS_KEY=password - AWS_REGION=us-east-1 healthcheck: test: 'mysql -u root -h starrocks-fe -P 9030 -e "SHOW FRONTENDS\G" |grep "Alive: true"' interval: 10s timeout: 5s retries: 3 starrocks-be: image: starrocks/be-ubuntu:4.0-latest command: - /bin/bash - -c - | ulimit -n 65535; echo "# Enable data cache" >> /opt/starrocks/be/conf/be.conf echo "block_cache_enable = true" >> /opt/starrocks/be/conf/be.conf echo "block_cache_mem_size = 536870912" >> /opt/starrocks/be/conf/be.conf echo "block_cache_disk_size = 1073741824" >> /opt/starrocks/be/conf/be.conf sleep 15s mysql --connect-timeout 2 -h starrocks-fe -P 9030 -u root -e "ALTER SYSTEM ADD BACKEND \"starrocks-be:9050\";" bash /opt/starrocks/be/bin/start_be.sh ports: - 8040:8040 hostname: starrocks-be container_name: starrocks-be user: root depends_on: starrocks-fe: condition: service_healthy healthcheck: test: 'mysql -u root -h starrocks-fe -P 9030 -e "SHOW BACKENDS\G" |grep "Alive: true"' interval: 10s timeout: 5s retries: 3 networks: iceberg_net: environment: - HOST_TYPE=FQDN - AWS_EC2_METADATA_DISABLED=true rest: image: apache/iceberg-rest-fixture container_name: iceberg-rest networks: iceberg_net: aliases: - iceberg-rest.minio ports: - 8181:8181 environment: - AWS_ACCESS_KEY_ID=admin - AWS_SECRET_ACCESS_KEY=password - AWS_REGION=us-east-1 - CATALOG_WAREHOUSE=s3://warehouse/ - CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO - CATALOG_S3_ENDPOINT=http://minio:9000 minio: image: minio/minio:RELEASE.2024-10-29T16-01-48Z container_name: minio environment: - MINIO_ROOT_USER=admin - MINIO_ROOT_PASSWORD=password - MINIO_DOMAIN=minio networks: iceberg_net: aliases: - warehouse.minio ports: - 9001:9001 - 9000:9000 command: ["server", "/data", "--console-address", ":9001"] mc: depends_on: - minio image: minio/mc:RELEASE.2024-10-29T15-34-59Z container_name: mc networks: iceberg_net: environment: - AWS_ACCESS_KEY_ID=admin - AWS_SECRET_ACCESS_KEY=password - AWS_REGION=us-east-1 entrypoint: > /bin/sh -c " until (/usr/bin/mc config host add minio http://minio:9000 admin password) do echo '...waiting...' && sleep 1; done; /usr/bin/mc rm -r --force minio/warehouse; /usr/bin/mc mb minio/warehouse; /usr/bin/mc policy set public minio/warehouse; tail -f /dev/null " networks: iceberg_net: ``` Next, start up the docker containers with this command: ```plain docker compose up --detach --wait --wait-timeout 400 ``` You can then run any of the following commands to start a StarRocks session. ```bash docker exec -it starrocks-fe \ mysql -P 9030 -h 127.0.0.1 -u root --prompt="StarRocks > " ``` ##### Adding and Using a Catalog[​](#adding-and-using-a-catalog "Direct link to Adding and Using a Catalog") ```sql CREATE EXTERNAL CATALOG 'demo' COMMENT "External catalog to Apache Iceberg on MinIO" PROPERTIES ( "type"="iceberg", "iceberg.catalog.type"="rest", "iceberg.catalog.uri"="http://iceberg-rest:8181", "iceberg.catalog.warehouse"="warehouse", "aws.s3.access_key"="admin", "aws.s3.secret_key"="password", "aws.s3.endpoint"="http://minio:9000", "aws.s3.enable_path_style_access"="true" ); ``` ```sql SHOW CATALOGS\G ``` ```sql *************************** 1. row *************************** Catalog: default_catalog Type: Internal Comment: An internal catalog contains this cluster's self-managed tables. *************************** 2. row *************************** Catalog: demo Type: Iceberg Comment: External catalog to Apache Iceberg on MinIO 2 rows in set (0.00 sec) ``` ```sql SET CATALOG demo; ``` ##### Creating and using a database[​](#creating-and-using-a-database "Direct link to Creating and using a database") ```sql CREATE DATABASE nyc; ``` ```sql USE nyc; ``` ##### Creating a table[​](#creating-a-table "Direct link to Creating a table") ```sql CREATE TABLE demo.nyc.taxis ( trip_id bigint, trip_distance float, fare_amount double, store_and_fwd_flag string, vendor_id bigint ) PARTITION BY (vendor_id); ``` ##### Writing Data to a Table[​](#writing-data-to-a-table "Direct link to Writing Data to a Table") ```sql INSERT INTO demo.nyc.taxis VALUES (1000371, 1.8, 15.32, 'N', 1), (1000372, 2.5, 22.15, 'N', 2), (1000373, 0.9, 9.01, 'N', 2), (1000374, 8.4, 42.13, 'Y', 1); ``` ##### Reading Data from a Table[​](#reading-data-from-a-table "Direct link to Reading Data from a Table") ```sql SELECT * FROM demo.nyc.taxis; ``` ##### Verify that the data is stored in object storage[​](#verify-that-the-data-is-stored-in-object-storage "Direct link to Verify that the data is stored in object storage") When you added and used the external catalog, Starrocks started using MinIO as the object store for the `demo.nyc.taxis` table. If you navigate to and then navigate through the Object Browser menu to `warehouse/nyc/taxis/` you can confirm that StarRocks is using MinIO for the storage. tip The username and password for MinIO are in the docker-compose.yml file. You will be prompted to change the password to something better, just ignore this advice for the tutorial. ![img](/assets/images/MinIO-Iceberg-data-8ade61c31be69444bd02b00acafe263c.png) ##### Next Steps[​](#next-steps "Direct link to Next Steps") ###### Adding Iceberg to StarRocks[​](#adding-iceberg-to-starrocks "Direct link to Adding Iceberg to StarRocks") If you already have a StarRocks 3.2.0, or later, environment, it comes with the Iceberg 1.6.0 included. No additional downloads or jars are needed. ###### Learn More[​](#learn-more "Direct link to Learn More") Now that you're up and running with Iceberg and StarRocks, check out the [StarRocks-Iceberg docs](https://docs.starrocks.io/docs/data_source/catalog/iceberg/iceberg_catalog.md) to learn more! --- ### Claude + StarRocks MCP Stand up a small StarRocks cluster on object storage, load a real public dataset, and ask **Claude** plain-English questions about it through the **StarRocks MCP server**. Claude discovers the schema, writes the SQL (including multi-table joins), and renders charts. All data lives in object storage on **MinIO**. This tutorial covers: * Running StarRocks in shared-data mode (1 FE + 1 CN) and MinIO in Docker * Creating an S3 (MinIO) storage volume for separate storage and compute * Loading the Olist Brazilian E-Commerce dataset (8 related tables) * Wiring the StarRocks and AIStor MCP servers to Claude * Asking Claude plain-English questions and rendering charts The dataset is the **Olist Brazilian E-Commerce** set — 8 related tables, good for genuinely complex joins. Everything runs on **1 FE + 1 CN**, on a laptop or a free cloud tier. There is a lot of information in this document, and it is presented with the step by step content at the beginning and the reference material at the end. This is done so that you can stand up the environment and start asking questions first, and read the supporting details afterward. *** #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") ##### Docker[​](#docker "Direct link to Docker") * [Docker](https://www.docker.com/get-started/) (Docker Desktop or engine + compose) * \~4 GB RAM assigned to Docker ##### MySQL client[​](#mysql-client "Direct link to MySQL client") A MySQL client (e.g. `mysql`) to run the SQL files. This is provided by the StarRocks FE server, so all of the SQL commands in this guide are run with `docker compose exec`. ##### uv[​](#uv "Direct link to uv") [`uv`](https://docs.astral.sh/uv/) runs the StarRocks MCP server. ##### Claude Code or Claude Desktop[​](#claude-code-or-claude-desktop "Direct link to Claude Code or Claude Desktop") [Claude Code](https://claude.com/product/claude-code) or Claude Desktop connects the MCP servers. tip This demo includes steps using Claude Code. Other LLMs work with MCP servers, but this demo has only been tested with Claude Code. Please open an issue and let us know what you experience with other LLMs. ##### The Olist dataset[​](#the-olist-dataset "Direct link to The Olist dataset") The **Olist dataset** is downloaded automatically when you load the data via `kagglehub` (no Kaggle account needed): . note Apple Silicon: StarRocks leans on AVX2 (x86); use an ARM build or expect slower emulation. *** #### Terminology[​](#terminology "Direct link to Terminology") ##### MCP[​](#mcp "Direct link to MCP") The Model Context Protocol (MCP) is an open protocol that lets an AI assistant such as Claude discover and call external tools. This tutorial wires up two MCP servers: * The **StarRocks MCP server** (`mcp-server-starrocks`) exposes tools that let Claude read the schema and run SQL against StarRocks. * The **AIStor MCP server** (`aistor`) exposes tools that let Claude work with the MinIO object storage where the data lives — for example, browsing buckets and inspecting the objects StarRocks writes. ##### FE[​](#fe "Direct link to FE") Frontend nodes are responsible for metadata management, client connection management, query planning, and query scheduling. ##### CN[​](#cn "Direct link to CN") Compute Nodes are responsible for executing query plans in shared-data deployments. *** #### Install the prerequisites[​](#install-the-prerequisites "Direct link to Install the prerequisites") ##### `uv`[​](#uv-1 "Direct link to uv-1") ```bash # macOS / Linux curl -LsSf https://astral.sh/uv/install.sh | sh # Windows (PowerShell) powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex" # Or via Homebrew / pipx / pip brew install uv # pipx install uv # pip install uv ``` See the [official `uv` installation guide](https://docs.astral.sh/uv/getting-started/installation/) for other options. After installing, verify it is on your `PATH`: ```bash uv --version ``` ##### Docker[​](#docker-1 "Direct link to Docker") ##### Claude Code[​](#claude-code "Direct link to Claude Code") *** #### Clone the demo repo[​](#clone-the-demo-repo "Direct link to Clone the demo repo") Clone : ```bash gh repo clone StarRocks/demo ``` or ```bash git clone git@github.com:StarRocks/demo.git ``` note This directory contains just enough to get you going. It deliberately contains *no* canned queries, no expected answers, and no walkthrough — so that everything Claude shows is reasoned from the live schema, not recalled from material in the checkout. *** #### Bring up StarRocks and MinIO[​](#bring-up-starrocks-and-minio "Direct link to Bring up StarRocks and MinIO") ```bash cd demo/documentation-samples/MCP docker compose up --detach --wait --wait-timeout 120 ``` A frontend (FE), a compute node (CN), and MinIO come up locally. Check for healthy status on the MinIO, FE, and CN services: ```bash docker compose ps -a --format "table {{.Service}}\t{{.Status}}" ``` tip If the CN is not reporting healthy just wait a few seconds and check again — it is the last service to start. *** #### Create a bucket in MinIO[​](#create-a-bucket-in-minio "Direct link to Create a bucket in MinIO") Open the MinIO console at (login `miniouser` / `M!n10R0cks`), click **Create Bucket**, and create the bucket `my-starrocks-bucket`. *** #### Create the storage volume[​](#create-the-storage-volume "Direct link to Create the storage volume") The file `storage_volume.sql` creates a StarRocks storage volume in the bucket you created in the previous step and sets it as the default. The contents of the file are explained in detail at the end of this tutorial; for now, run it: ```bash docker compose exec -T starrocks-fe \ mysql -P9030 -h127.0.0.1 -uroot < storage_volume.sql ``` This must succeed (and the volume must be the default) before any `CREATE TABLE`. ```sql *************************** 1. row *************************** Name: s3_volume Type: S3 IsDefault: true Location: s3://my-starrocks-bucket/ Params: {"aws.s3.access_key":"******","aws.s3.secret_key":"******","aws.s3.endpoint":"minio:9000","aws.s3.region":"us-east-1","aws.s3.use_instance_profile":"false","aws.s3.use_web_identity_token_file":"false","aws.s3.use_aws_sdk_default_behavior":"false"} Enabled: true Comment: ``` *** #### Create the tables[​](#create-the-tables "Direct link to Create the tables") ```bash docker compose exec -T starrocks-fe \ mysql -h127.0.0.1 -P9030 -uroot < olist_schema.sql ``` tip If you see an error `The specified bucket does not exist` you may have skipped part of the previous step. Open the MinIO UI and create the bucket specified above. tip This is a good time to try out the MCP server. Start Claude Code from the current directory and allow the two MCP servers (`aistor` and `mcp-server-starrocks`), then ask Claude to list the databases and describe the schema in the `olist` DB. *** #### Load the data[​](#load-the-data "Direct link to Load the data") Download the Olist CSVs from Kaggle with **kagglehub** — anonymous, no Kaggle account or API token required. It caches the files locally and prints the folder they landed in. Download the data and capture the folder it landed in as `CSV_DIR`. kagglehub prints a version warning to stdout, so take the last line (the path) with `tail -n1`: ```bash export CSV_DIR="$(uv run --with "kagglehub==0.3.12" python \ -c "import kagglehub; print(kagglehub.dataset_download('olistbr/brazilian-ecommerce'))" \ | tail -n1)" echo "$CSV_DIR" # sanity check: should be a .../brazilian-ecommerce/versions/N path ``` note The `kagglehub==0.3.12` pin is deliberate: newer releases (1.0.x) pull a `kagglesdk` build that fails to import (`ModuleNotFoundError: kagglesdk.competitions.legacy`). 0.3.12 downloads public datasets anonymously and works fine. kagglehub caches the files, so re-running is cheap. Then run the loader (`CSV_DIR` is already exported): ```bash FE_HTTP_PORT=8040 bash load_olist.sh ``` `FE_HTTP_PORT=8040` posts Stream Load straight to the CN, avoiding a `starrocks-cn` hostname redirect (no `/etc/hosts` edit / no sudo needed). The script prints a row-count check at the end. *** #### Wire the MCP servers to Claude[​](#wire-the-mcp-servers-to-claude "Direct link to Wire the MCP servers to Claude") ```bash cp .env.example .env ``` `.mcp.json` defines two MCP servers — `mcp-server-starrocks` (run from GitHub via `uv`, so there's **nothing to install by hand**) and `aistor` (run via Docker) — both reading `.env`. Launch Claude Code from this directory (or add the same block to Claude Desktop's config). note **Approve the servers on first launch.** Project-scoped servers from `.mcp.json` are *not* trusted automatically — Claude Code prompts *"New MCP servers found — approve?"* the first time you launch here. Accept it. (The pre-approval lives in `.claude/settings.local.json`, which is gitignored, so it is **absent on a fresh clone** — that's expected; approving via the prompt recreates it.) **Confirm the connection with `/mcp`.** Run `/mcp` inside Claude Code and check that both `mcp-server-starrocks` and `aistor` show **connected**. This is the authoritative status check — if a server isn't connected, its tools won't load no matter how you ask. **First launch is slow.** The first `uv run` downloads the server's dependencies (pyarrow, kaleido, …, ~115 MB), so the StarRocks server can take a couple of minutes to come up the first time. Subsequent launches are fast. Once `/mcp` shows both servers connected, confirm the tools are available (e.g. ask *"What tools does the StarRocks MCP server provide?"*). The StarRocks cluster is reachable over the MySQL protocol as `root` (no password) at `localhost:9030`, database `olist`. *** #### Ask questions[​](#ask-questions "Direct link to Ask questions") Ask Claude plain-English questions about the data — start by having it orient itself, e.g. *"What tables are in this database, and how do they relate to each other?"*, then explore from there. Claude inspects the live schema and writes the SQL itself. ##### Suggested questions[​](#suggested-questions "Direct link to Suggested questions") These are the questions asked in the demo video, in order. Each one builds on the last — start with the first to let Claude orient itself, then work down: * *What databases and tables are in here?* * *How do these tables fit together?* * *Where does this data actually live?* * *Plot orders by customer state.* * *Does it cost more to ship things farther?* * *Add a second line — freight per kilo — to separate weight from distance.* * *How did you build that query?* * *Do late deliveries hurt our reviews?* * *Are you using canned answers, or did every number come from the live tables?* tip **Charts:** ask for an interactive chart in **HTML** format and the StarRocks MCP server writes it to `DEMO_Output/` (set by `STARROCKS_CHART_OUTPUT_DIR` in `.env`), with a static preview shown inline — open the HTML file in a browser to hover / zoom / pan. *** #### Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting") * **StarRocks MCP tools never load / server "still connecting":** the project servers were never approved. Run `/mcp` — if they aren't **connected**, relaunch Claude Code from this directory and accept the *"New MCP servers found — approve?"* prompt (or add `"enabledMcpjsonServers": ["mcp-server-starrocks", "aistor"]` to `.claude/settings.local.json`). You do **not** need to install the server separately — `.mcp.json` runs it from GitHub via `uv`. * **First launch takes a couple of minutes:** the first `uv run` downloads the server's deps (~115 MB). This is expected once, not a failure. * **`mysql` client fails with `Authentication plugin 'mysql_native_password' cannot be loaded`:** a Homebrew `mysql`-client quirk, not a StarRocks problem. Query through the container instead: `docker compose exec -T starrocks-fe mysql -h127.0.0.1 -P9030 -uroot -e "SHOW DATABASES;"`. * **`CREATE TABLE` hangs / times out:** the CN's AWS SDK is probing the EC2 metadata service. The compose file already sets `AWS_EC2_METADATA_DISABLED=true` on `starrocks-cn`; if you use your own compose file, add it. * **Stream Load redirect errors:** use `FE_HTTP_PORT=8040` (as above) to post to the CN directly, or add `127.0.0.1 starrocks-cn` to `/etc/hosts`. * **`order_reviews` rejects rows:** the free-text comment columns contain embedded newlines; use the LEAN variant noted in `load_olist.sh`. *** #### Summary[​](#summary "Direct link to Summary") In this tutorial you: * Deployed StarRocks shared-data (1 FE + 1 CN) and MinIO in Docker * Created an S3 (MinIO) storage volume and set it as the default * Loaded the Olist Brazilian E-Commerce dataset (8 related tables) * Wired the StarRocks and AIStor MCP servers to Claude * Asked Claude plain-English questions and let it discover the schema, write the SQL, and render charts The build/setup material — narrated walkthrough and the video script — is maintained separately so this environment stays free of canned answers: * **StarRocks MCP server:** *** #### What's in here[​](#whats-in-here "Direct link to What's in here") The demo files live in the `documentation-samples/MCP` directory of the [StarRocks/demo](https://github.com/StarRocks/demo) repo: | File | What it is | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `CLAUDE.md` | Instructions to Claude Code to show its work, rely on the schema and not outside information, etc. You should read this file. | | `.claude/settings.json` | Content is `"autoMemoryEnabled": false` to prevent Claude from remembering the questions (and answers) asked in previous sessions. You can remove this if you like; it is in here to prevent the author's questions from biasing your experimentation. | | `docker-compose.yml` | StarRocks shared-data quick start (1 FE + 1 CN + MinIO), patched for laptop/Docker use. | | `storage_volume.sql` | Creates the S3 (MinIO) storage volume and sets it as default. | | `olist_schema.sql` | DDL for the 8 (+1 optional) Olist tables. | | `load_olist.sh` | Stream Load for the Olist CSVs + a row-count check. | | `.mcp.json` | Wires the StarRocks and AIStor MCP servers to Claude. | | `.env.example` | Credentials/endpoints template — `cp` to `.env` and edit if needed. | *** #### Notes on `storage_volume.sql`[​](#notes-on-storage_volumesql "Direct link to notes-on-storage_volumesql") `storage_volume.sql` does two things: it raises a timeout, then creates and activates the storage volume. First it increases the timeout for tablet creation and creates the storage volume: ```sql -- default is 10 s, too tight for object-store tablet creation ADMIN SET FRONTEND CONFIG ('tablet_create_timeout_second'='60'); CREATE STORAGE VOLUME s3_volume TYPE = S3 LOCATIONS = ("s3://my-starrocks-bucket/") PROPERTIES ( "enabled" = "true", "aws.s3.endpoint" = "minio:9000", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "aws.s3.use_instance_profile" = "false", "aws.s3.use_aws_sdk_default_behavior" = "false" ); ``` It then sets the new storage volume as the default and shows its details: ```sql SET s3_volume AS DEFAULT STORAGE VOLUME; -- REQUIRED before CREATE DATABASE/TABLE DESC STORAGE VOLUME s3_volume\G -- verify: enabled=true, IsDefault=true ``` *** #### More information[​](#more-information "Direct link to More information") * StarRocks shared-data quick start — [shared-data](https://docs.starrocks.io/docs/quick_start/shared-data.md) * StarRocks MCP server — * AIStor MCP server — * Olist dataset — *** #### Credits & license[​](#credits--license "Direct link to Credits & license") * **Olist Brazilian E-Commerce** dataset via Kaggle, licensed **CC BY-NC-SA 4.0 (non-commercial)** — used here for educational demonstration; credit retained to Olist. Confirm terms before any commercial reuse. * StarRocks and the StarRocks MCP server are open source (Apache-2.0 / project license). * MinIO and `mcp-server-aistor` © MinIO, Inc. --- ### Kafka routine load StarRocks using shared-data storage #### About Routine Load[​](#about-routine-load "Direct link to About Routine Load") Routine load is a method using Apache Kafka, or in this lab, Redpanda, to continuously stream data into StarRocks. The data is streamed into a Kafka topic, and a Routine Load job consumes the data into StarRocks. More details on Routine Load are provided at the end of the lab. #### About shared-data[​](#about-shared-data "Direct link to About shared-data") In systems that separate storage from compute, data is stored in low-cost reliable remote storage systems such as Amazon S3, Google Cloud Storage, Azure Blob Storage, and other S3-compatible storage like MinIO. Hot data is cached locally and when the cache is hit, the query performance is comparable to that of storage-compute coupled architecture. Compute nodes (CN) can be added or removed on demand within seconds. This architecture reduces storage costs, ensures better resource isolation, and provides elasticity and scalability. This tutorial covers: * Running StarRocks, Redpanda, and MinIO with Docker Compose * Using MinIO as the StarRocks storage layer * Configuring StarRocks for shared-data * Adding a Routine Load job to consume data from Redpanda The data used is synthetic. There is a lot of information in this document, and it is presented with step-by-step content at the beginning, and the technical details at the end. This is done to serve these purposes in this order: 1. Configure Routine Load. 2. Allow the reader to load data in a shared-data deployment and analyze that data. 3. Provide the configuration details for shared-data deployments. *** #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") ##### Docker[​](#docker "Direct link to Docker") * [Docker](https://docs.docker.com/engine/install/) * 4 GB RAM assigned to Docker * 10 GB free disk space assigned to Docker ##### SQL client[​](#sql-client "Direct link to SQL client") You can use the SQL client provided in the Docker environment, or use one on your system. Many MySQL-compatible clients will work, and this guide covers the configuration of DBeaver and MySQL Workbench. ##### curl[​](#curl "Direct link to curl") `curl` is used to download the Compose file and the script to generate the data. Check to see if you have it installed by running `curl` or `curl.exe` at your OS prompt. If curl is not installed, [get curl here](https://curl.se/). ##### Python[​](#python "Direct link to Python") Python 3 and the Python client for Apache Kafka, `kafka-python`, are required. * [Python](https://www.python.org/) * [`kafka-python`](https://pypi.org/project/kafka-python/) *** #### Terminology[​](#terminology "Direct link to Terminology") ##### FE[​](#fe "Direct link to FE") Frontend nodes are responsible for metadata management, client connection management, query planning, and query scheduling. Each FE stores and maintains a complete copy of metadata in its memory, which guarantees indiscriminate services among the FEs. ##### CN[​](#cn "Direct link to CN") Compute Nodes are responsible for executing query plans in shared-data deployments. ##### BE[​](#be "Direct link to BE") Backend nodes are responsible for both data storage and executing query plans in shared-nothing deployments. note This guide does not use BEs, this information is included here so that you understand the difference between BEs and CNs. *** #### Launch StarRocks[​](#launch-starrocks "Direct link to Launch StarRocks") To run StarRocks with shared-data using Object Storage you need: * A frontend engine (FE) * A compute node (CN) * Object Storage This guide uses MinIO, which is S3 compatible Object Storage provider. MinIO is provided under the GNU Affero General Public License. ##### Download the lab files[​](#download-the-lab-files "Direct link to Download the lab files") ###### `docker-compose.yml`[​](#docker-composeyml "Direct link to docker-composeyml") ```bash mkdir routineload cd routineload curl -O https://raw.githubusercontent.com/StarRocks/demo/master/documentation-samples/routine-load-shared-data/docker-compose.yml ``` ###### `gen.py`[​](#genpy "Direct link to genpy") `gen.py` is a script that uses the Python client for Apache Kafka to publish (produce) data to a Kafka topic. The script has been written with the address and port of the Redpanda container. ```bash curl -O https://raw.githubusercontent.com/StarRocks/demo/master/documentation-samples/routine-load-shared-data/gen.py ``` #### Start StarRocks, MinIO, and Redpanda[​](#start-starrocks-minio-and-redpanda "Direct link to Start StarRocks, MinIO, and Redpanda") ```bash docker compose up --detach --wait --wait-timeout 120 ``` Check the progress of the services. It should take 30 seconds or more for the containers to become healthy. The `routineload-minio_mc-1` container will not show a health indicator, and it will exit once it is done configuring MinIO with the access key that StarRocks will use. Wait for `routineload-minio_mc-1` to exit with a `0` code and the rest of the services to be `Healthy`. Run `docker compose ps` until the services are healthy: ```bash docker compose ps ``` ```plaintext WARN[0000] /Users/droscign/routineload/docker-compose.yml: `version` is obsolete [+] Running 6/7 ✔ Network routineload_default Crea... 0.0s ✔ Container minio Healthy 5.6s ✔ Container redpanda Healthy 3.6s ✔ Container redpanda-console Healt... 1.1s ⠧ Container routineload-minio_mc-1 Waiting 23.1s ✔ Container starrocks-fe Healthy 11.1s ✔ Container starrocks-cn Healthy 23.0s container routineload-minio_mc-1 exited (0) ``` *** #### Examine MinIO credentials[​](#examine-minio-credentials "Direct link to Examine MinIO credentials") In order to use MinIO for Object Storage with StarRocks, StarRocks needs a MinIO access key. The access key was generated during the startup of the Docker services. To help you better understand the way that StarRocks connects to MinIO you should verify that the key exists. ##### Open the MinIO web UI[​](#open-the-minio-web-ui "Direct link to Open the MinIO web UI") Browse to The username and password are specified in the Docker compose file, and are `miniouser` and `miniopassword`. You should see that there is one access key. The Key is `AAAAAAAAAAAAAAAAAAAA`, you cannot see the secret in the MinIO Console, but it is in the Docker compose file and is `BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB`: ![View the MinIO access key](/assets/images/MinIO-view-key-9df9610b4c42313f682f7ac6ecb1df07.png) *** ##### Create a bucket for your data[​](#create-a-bucket-for-your-data "Direct link to Create a bucket for your data") When you create a storage volume in StarRocks you will specify the `LOCATION` for the data: ```sh LOCATIONS = ("s3://my-starrocks-bucket/") ``` Open and add a bucket for the storage volume. Name the bucket `my-starrocks-bucket`. Accept the defaults for the three listed options. *** #### SQL Clients[​](#sql-clients "Direct link to SQL Clients") These three clients are tested with this tutorial, you only need one: * mysql CLI: You can run this from the Docker environment or your machine. * [DBeaver](https://dbeaver.io/download/) is available as a community version and a Pro version. * [MySQL Workbench](https://dev.mysql.com/downloads/workbench/) ###### Configuring the client[​](#configuring-the-client "Direct link to Configuring the client") * mysql CLI * DBeaver * MySQL Workbench The easiest way to use the mysql CLI is to run it from the StarRocks container `starrocks-fe`: ```bash docker compose exec starrocks-fe \ mysql -P 9030 -h 127.0.0.1 -u root --prompt="StarRocks > " ``` tip All `docker compose` commands must be run from the directory containing the `docker-compose.yml` file. If you would like to install the mysql CLI expand **mysql client install** below: mysql client install * macOS: If you use Homebrew and do not need MySQL Server run `brew install mysql-client@8.0` to install the CLI. * Linux: Check your repository system for the `mysql` client. For example, `yum install mariadb`. * Microsoft Windows: Install the [MySQL Community Server](https://dev.mysql.com/downloads/mysql/) and run the provided client, or run `mysql` from WSL. - Install [DBeaver](https://dbeaver.io/download/), and add a connection: ![Add a connection](/assets/images/DBeaver-1-58907bd9df44bba1e40282214e4a056c.png) - Configure the port, IP, and username. Test the connection, and click Finish if the test succeeds: ![Configure and test](/assets/images/DBeaver-2-8e4bdc09135b4738944d7016bfb1af94.png) * Install the [MySQL Workbench](https://dev.mysql.com/downloads/workbench/), and add a connection. * Configure the port, IP, and username and then test the connection: ![Connection settings](/assets/images/Workbench-1-9fd3a3126d5f83478e51891287e412b9.png) * You will see warnings from the Workbench as it is checking for a specific MySQL version. You can ignore the warnings and when prompted, you can configure Workbench to stop displaying the warnings: ![Ignore warning](/assets/images/Workbench-2-efba18763e56101091746837ae666b5e.png) *** #### StarRocks configuration for shared-data[​](#starrocks-configuration-for-shared-data "Direct link to StarRocks configuration for shared-data") At this point you have StarRocks running, and you have MinIO running. The MinIO access key is used to connect StarRocks and Minio. This is the part of the `FE` configuration that specifies that the StarRocks deployment will use shared data. This was added to the file `fe.conf` when Docker Compose created the deployment. ```sh # enable the shared data run mode run_mode = shared_data cloud_native_storage_type = S3 ``` info You can verify these settings by running this command from the `quickstart` directory and looking at the end of the file: ```sh docker compose exec starrocks-fe \ cat /opt/starrocks/fe/conf/fe.conf ``` ::: ##### Connect to StarRocks with a SQL client[​](#connect-to-starrocks-with-a-sql-client "Direct link to Connect to StarRocks with a SQL client") tip Run this command from the directory containing the `docker-compose.yml` file. If you are using a client other than the mysql CLI, open that now. ```sql docker compose exec starrocks-fe \ mysql -P9030 -h127.0.0.1 -uroot --prompt="StarRocks > " ``` ###### Examine the storage volumes[​](#examine-the-storage-volumes "Direct link to Examine the storage volumes") ```sql SHOW STORAGE VOLUMES; ``` tip There should be no storage volumes, you will create one next. ```sh Empty set (0.04 sec) ``` ###### Create a shared-data storage volume[​](#create-a-shared-data-storage-volume "Direct link to Create a shared-data storage volume") Earlier you created a bucket in MinIO named `my-starrocks-volume`, and you verified that MinIO has an access key named `AAAAAAAAAAAAAAAAAAAA`. The following SQL will create a storage volume in the MionIO bucket using the access key and secret. ```sql CREATE STORAGE VOLUME s3_volume TYPE = S3 LOCATIONS = ("s3://my-starrocks-bucket/") PROPERTIES ( "enabled" = "true", "aws.s3.endpoint" = "minio:9000", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "aws.s3.use_instance_profile" = "false", "aws.s3.use_aws_sdk_default_behavior" = "false" ); ``` Now you should see a storage volume listed, earlier it was an empty set: ```text SHOW STORAGE VOLUMES; ``` ```text +----------------+ | Storage Volume | +----------------+ | s3_volume | +----------------+ 1 row in set (0.02 sec) ``` View the details of the storage volume and note that this is nott yet the default volume, and that it is configured to use your bucket: ```text DESC STORAGE VOLUME s3_volume\G ``` tip Some of the SQL in this document, and many other documents in the StarRocks documentation, and with `\G` instead of a semicolon. The `\G` causes the mysql CLI to render the query results vertically. Many SQL clients do not interpret vertical formatting output, so you should replace `\G` with `;`. ```sh *************************** 1. row *************************** Name: s3_volume Type: S3 IsDefault: false Location: s3://my-starrocks-bucket/ Params: {"aws.s3.access_key":"******","aws.s3.secret_key":"******","aws.s3.endpoint":"minio:9000","aws.s3.region":"us-east-1","aws.s3.use_instance_profile":"false","aws.s3.use_web_identity_token_file":"false","aws.s3.use_aws_sdk_default_behavior":"false"} Enabled: true Comment: 1 row in set (0.02 sec) ``` #### Set the default storage volume[​](#set-the-default-storage-volume "Direct link to Set the default storage volume") ```text SET s3_volume AS DEFAULT STORAGE VOLUME; ``` ```text DESC STORAGE VOLUME s3_volume\G ``` ```sh *************************** 1. row *************************** Name: s3_volume Type: S3 IsDefault: true Location: s3://my-starrocks-bucket/ Params: {"aws.s3.access_key":"******","aws.s3.secret_key":"******","aws.s3.endpoint":"minio:9000","aws.s3.region":"us-east-1","aws.s3.use_instance_profile":"false","aws.s3.use_web_identity_token_file":"false","aws.s3.use_aws_sdk_default_behavior":"false"} Enabled: true Comment: 1 row in set (0.02 sec) ``` *** #### Create a table[​](#create-a-table "Direct link to Create a table") These SQL commands are run in your SQL client. ```sql CREATE DATABASE IF NOT EXISTS quickstart; ``` Verify that the database `quickstart` is using the storage volume `s3_volume`: ```text SHOW CREATE DATABASE quickstart \G ``` ```sh *************************** 1. row *************************** Database: quickstart Create Database: CREATE DATABASE `quickstart` PROPERTIES ("storage_volume" = "s3_volume") ``` ```sql USE quickstart; ``` ```sql CREATE TABLE site_clicks ( `uid` bigint NOT NULL COMMENT "uid", `site` string NOT NULL COMMENT "site url", `vtime` bigint NOT NULL COMMENT "vtime" ) DISTRIBUTED BY HASH(`uid`) PROPERTIES("replication_num"="1"); ``` *** ##### Open the Redpanda Console[​](#open-the-redpanda-console "Direct link to Open the Redpanda Console") There will be no topics yet, a topic will be created in the next step. ##### Publish data to a Redpanda topic[​](#publish-data-to-a-redpanda-topic "Direct link to Publish data to a Redpanda topic") From a command shell in the `routineload/` folder run this command to generate data: ```python python gen.py 5 ``` tip On your system, you might need to use `python3` in place of `python` in the command. If you are missing `kafka-python` try: ```text pip install kafka-python ``` or ```text pip3 install kafka-python ``` ```plaintext b'{ "uid": 6926, "site": "https://docs.starrocks.io/", "vtime": 1718034793 } ' b'{ "uid": 3303, "site": "https://www.starrocks.io/product/community", "vtime": 1718034793 } ' b'{ "uid": 227, "site": "https://docs.starrocks.io/", "vtime": 1718034243 } ' b'{ "uid": 7273, "site": "https://docs.starrocks.io/", "vtime": 1718034794 } ' b'{ "uid": 4666, "site": "https://www.starrocks.io/", "vtime": 1718034794 } ' ``` ##### Verify in the Redpanda Console[​](#verify-in-the-redpanda-console "Direct link to Verify in the Redpanda Console") Navigate to in the Redpanda Console, and you will see one topic named `test2`. Select that topic and then the **Messages** tab and you will see five messages matching the output of `gen.py`. #### Consume the messages[​](#consume-the-messages "Direct link to Consume the messages") In StarRocks you will create a Routine Load job to: 1. Consume the messages from the Redpanda topic `test2` 2. Load those messages into the table `site_clicks` StarRocks is configured to use MinIO for storage, so the data inserted into the `site_clicks` table will be stored in MinIO. ##### Create a Routine Load job[​](#create-a-routine-load-job "Direct link to Create a Routine Load job") Run this command in the SQL client to create the Routine Load job, the command will be explained in detail at the end of the lab. ```sql CREATE ROUTINE LOAD quickstart.clicks ON site_clicks PROPERTIES ( "format" = "JSON", "jsonpaths" ="[\"$.uid\",\"$.site\",\"$.vtime\"]" ) FROM KAFKA ( "kafka_broker_list" = "redpanda:29092", "kafka_topic" = "test2", "kafka_partitions" = "0", "kafka_offsets" = "OFFSET_BEGINNING" ); ``` ##### Verify the Routine Load job[​](#verify-the-routine-load-job "Direct link to Verify the Routine Load job") ```sql SHOW ROUTINE LOAD\G ``` Verify the three highlighted lines: 1. The state should be `RUNNING` 2. The topic should be `test2` and the broker should be `redpanda:2092` 3. The statistics should show either 0 or 5 loaded rows depending on how soon you ran the `SHOW ROUTINE LOAD` command. If there are 0 loaded rows run it again. ```sql *************************** 1. row *************************** Id: 10078 Name: clicks CreateTime: 2024-06-12 15:51:12 PauseTime: NULL EndTime: NULL DbName: quickstart TableName: site_clicks State: RUNNING DataSourceType: KAFKA CurrentTaskNum: 1 JobProperties: {"partitions":"*","partial_update":"false","columnToColumnExpr":"*","maxBatchIntervalS":"10","partial_update_mode":"null","whereExpr":"*","dataFormat":"json","timezone":"Etc/UTC","format":"json","log_rejected_record_num":"0","taskTimeoutSecond":"60","json_root":"","maxFilterRatio":"1.0","strict_mode":"false","jsonpaths":"[\"$.uid\",\"$.site\",\"$.vtime\"]","taskConsumeSecond":"15","desireTaskConcurrentNum":"5","maxErrorNum":"0","strip_outer_array":"false","currentTaskConcurrentNum":"1","maxBatchRows":"200000"} DataSourceProperties: {"topic":"test2","currentKafkaPartitions":"0","brokerList":"redpanda:29092"} CustomProperties: {"group.id":"clicks_ea38a713-5a0f-4abe-9b11-ff4a241ccbbd"} Statistic: {"receivedBytes":0,"errorRows":0,"committedTaskNum":0,"loadedRows":0,"loadRowsRate":0,"abortedTaskNum":0,"totalRows":0,"unselectedRows":0,"receivedBytesRate":0,"taskExecuteTimeMs":1} Progress: {"0":"OFFSET_ZERO"} TimestampProgress: {} ReasonOfStateChanged: ErrorLogUrls: TrackingSQL: OtherMsg: LatestSourcePosition: {} 1 row in set (0.00 sec) ``` ```sql SHOW ROUTINE LOAD\G ``` ```sql *************************** 1. row *************************** Id: 10076 Name: clicks CreateTime: 2024-06-12 18:40:53 PauseTime: NULL EndTime: NULL DbName: quickstart TableName: site_clicks State: RUNNING DataSourceType: KAFKA CurrentTaskNum: 1 JobProperties: {"partitions":"*","partial_update":"false","columnToColumnExpr":"*","maxBatchIntervalS":"10","partial_update_mode":"null","whereExpr":"*","dataFormat":"json","timezone":"Etc/UTC","format":"json","log_rejected_record_num":"0","taskTimeoutSecond":"60","json_root":"","maxFilterRatio":"1.0","strict_mode":"false","jsonpaths":"[\"$.uid\",\"$.site\",\"$.vtime\"]","taskConsumeSecond":"15","desireTaskConcurrentNum":"5","maxErrorNum":"0","strip_outer_array":"false","currentTaskConcurrentNum":"1","maxBatchRows":"200000"} DataSourceProperties: {"topic":"test2","currentKafkaPartitions":"0","brokerList":"redpanda:29092"} CustomProperties: {"group.id":"clicks_a9426fee-45bb-403a-a1a3-b3bc6c7aa685"} Statistic: {"receivedBytes":372,"errorRows":0,"committedTaskNum":1,"loadedRows":5,"loadRowsRate":0,"abortedTaskNum":0,"totalRows":5,"unselectedRows":0,"receivedBytesRate":0,"taskExecuteTimeMs":519} Progress: {"0":"4"} TimestampProgress: {"0":"1718217035111"} ReasonOfStateChanged: ErrorLogUrls: TrackingSQL: OtherMsg: LatestSourcePosition: {"0":"5"} 1 row in set (0.00 sec) ``` *** #### Verify that data is stored in MinIO[​](#verify-that-data-is-stored-in-minio "Direct link to Verify that data is stored in MinIO") Open MinIO and verify that there are objects stored under `my-starrocks-bucket`. *** #### Query the data from StarRocks[​](#query-the-data-from-starrocks "Direct link to Query the data from StarRocks") ```sql USE quickstart; SELECT * FROM site_clicks; ``` ```sql +------+--------------------------------------------+------------+ | uid | site | vtime | +------+--------------------------------------------+------------+ | 4607 | https://www.starrocks.io/blog | 1718031441 | | 1575 | https://www.starrocks.io/ | 1718031523 | | 2398 | https://docs.starrocks.io/ | 1718033630 | | 3741 | https://www.starrocks.io/product/community | 1718030845 | | 4792 | https://www.starrocks.io/ | 1718033413 | +------+--------------------------------------------+------------+ 5 rows in set (0.07 sec) ``` #### Publish additional data[​](#publish-additional-data "Direct link to Publish additional data") Running `gen.py` again will publish another five records to Redpanda. ```bash python gen.py 5 ``` ##### Verify that data is added[​](#verify-that-data-is-added "Direct link to Verify that data is added") Since the Routine Load job runs on a schedule (every 10 seconds by default), the data will be loaded within a few seconds. ```sql SELECT * FROM site_clicks; ``` ```text +------+--------------------------------------------+------------+ | uid | site | vtime | +------+--------------------------------------------+------------+ | 6648 | https://www.starrocks.io/blog | 1718205970 | | 7914 | https://www.starrocks.io/ | 1718206760 | | 9854 | https://www.starrocks.io/blog | 1718205676 | | 1186 | https://www.starrocks.io/ | 1718209083 | | 3305 | https://docs.starrocks.io/ | 1718209083 | | 2288 | https://www.starrocks.io/blog | 1718206759 | | 7879 | https://www.starrocks.io/product/community | 1718204280 | | 2666 | https://www.starrocks.io/ | 1718208842 | | 5801 | https://www.starrocks.io/ | 1718208783 | | 8409 | https://www.starrocks.io/ | 1718206889 | +------+--------------------------------------------+------------+ 10 rows in set (0.02 sec) ``` *** #### Configuration details[​](#configuration-details "Direct link to Configuration details") Now that you have experienced using StarRocks with shared-data it is important to understand the configuration. ##### CN configuration[​](#cn-configuration "Direct link to CN configuration") The CN configuration used here is the default, as the CN is designed for shared-data use. The default configuration is shown below. You do not need to make any changes. ```bash sys_log_level = INFO # ports for admin, web, heartbeat service be_port = 9060 be_http_port = 8040 heartbeat_service_port = 9050 brpc_port = 8060 starlet_port = 9070 ``` ##### FE configuration[​](#fe-configuration "Direct link to FE configuration") The FE configuration is slightly different from the default as the FE must be configured to expect that data is stored in Object Storage rather than on local disks on BE nodes. The `docker-compose.yml` file generates the FE configuration in the `command`. ```plaintext # enable shared data, set storage type, set endpoint run_mode = shared_data cloud_native_storage_type = S3 ``` note This config file does not contain the default entries for an FE, only the shared-data configuration is shown. The non-default FE configuration settings: note Many configuration parameters are prefixed with `s3_`. This prefix is used for all Amazon S3 compatible storage types (for example: S3, GCS, and MinIO). When using Azure Blob Storage the prefix is `azure_`. ###### `run_mode=shared_data`[​](#run_modeshared_data "Direct link to run_modeshared_data") This enables shared-data use. ###### `cloud_native_storage_type=S3`[​](#cloud_native_storage_types3 "Direct link to cloud_native_storage_types3") This specifies whether S3 compatible storage or Azure Blob Storage is used. For MinIO this is always S3. ##### Details of `CREATE storage volume`[​](#details-of-create-storage-volume "Direct link to details-of-create-storage-volume") ```sql CREATE STORAGE VOLUME s3_volume TYPE = S3 LOCATIONS = ("s3://my-starrocks-bucket/") PROPERTIES ( "enabled" = "true", "aws.s3.endpoint" = "minio:9000", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "aws.s3.use_instance_profile" = "false", "aws.s3.use_aws_sdk_default_behavior" = "false" ); ``` ###### `aws_s3_endpoint=minio:9000`[​](#aws_s3_endpointminio9000 "Direct link to aws_s3_endpointminio9000") The MinIO endpoint, including port number. ###### `aws_s3_path=starrocks`[​](#aws_s3_pathstarrocks "Direct link to aws_s3_pathstarrocks") The bucket name. ###### `aws_s3_access_key=AAAAAAAAAAAAAAAAAAAA`[​](#aws_s3_access_keyaaaaaaaaaaaaaaaaaaaa "Direct link to aws_s3_access_keyaaaaaaaaaaaaaaaaaaaa") The MinIO access key. ###### `aws_s3_secret_key=BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB`[​](#aws_s3_secret_keybbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb "Direct link to aws_s3_secret_keybbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb") The MinIO access key secret. ###### `aws_s3_use_instance_profile=false`[​](#aws_s3_use_instance_profilefalse "Direct link to aws_s3_use_instance_profilefalse") When using MinIO an access key is used, and so instance profiles are not used with MinIO. ###### `aws_s3_use_aws_sdk_default_behavior=false`[​](#aws_s3_use_aws_sdk_default_behaviorfalse "Direct link to aws_s3_use_aws_sdk_default_behaviorfalse") When using MinIO this parameter is always set to false. *** #### Notes on the Routine Load command[​](#notes-on-the-routine-load-command "Direct link to Notes on the Routine Load command") StarRocks Routine Load takes many arguments. Only the ones used in this tutorial are described here, the rest will be linked to in the more information section. ```sql CREATE ROUTINE LOAD quickstart.clicks ON site_clicks PROPERTIES ( "format" = "JSON", "jsonpaths" ="[\"$.uid\",\"$.site\",\"$.vtime\"]" ) FROM KAFKA ( "kafka_broker_list" = "redpanda:29092", "kafka_topic" = "test2", "kafka_partitions" = "0", "kafka_offsets" = "OFFSET_BEGINNING" ); ``` ##### Parameters[​](#parameters "Direct link to Parameters") ```text CREATE ROUTINE LOAD quickstart.clicks ON site_clicks ``` The parameters for `CREATE ROUTINE LOAD ON` are: * database\_name.job\_name * table\_name `database_name` is optional. In this lab, it is `quickstart` and is specified. `job_name` is required, and is `clicks` `table_name` is required, and is `site_clicks` ##### Job properties[​](#job-properties "Direct link to Job properties") ###### Property `format`[​](#property-format "Direct link to property-format") ```text "format" = "JSON", ``` In this case, the data is in JSON format, so the property is set to `JSON`. The other valid formats are: `CSV`, `JSON`, and `Avro`. `CSV` is the default. ###### Property `jsonpaths`[​](#property-jsonpaths "Direct link to property-jsonpaths") ```text "jsonpaths" ="[\"$.uid\",\"$.site\",\"$.vtime\"]" ``` The names of the fields that you want to load from JSON-formatted data. The value of this parameter is a valid JsonPath expression. More information is available at the end of this page. ##### Data source properties[​](#data-source-properties "Direct link to Data source properties") ###### `kafka_broker_list`[​](#kafka_broker_list "Direct link to kafka_broker_list") ```text "kafka_broker_list" = "redpanda:29092", ``` Kafka's broker connection information. The format is `:`. Multiple brokers are separated by commas. ###### `kafka_topic`[​](#kafka_topic "Direct link to kafka_topic") ```text "kafka_topic" = "test2", ``` The Kafka topic to consume from. ###### `kafka_partitions` and `kafka_offsets`[​](#kafka_partitions-and-kafka_offsets "Direct link to kafka_partitions-and-kafka_offsets") ```text "kafka_partitions" = "0", "kafka_offsets" = "OFFSET_BEGINNING" ``` These properties are presented together as there is one `kafka_offset` required for each `kafka_partitions` entry. `kafka_partitions` is a list of one or more partitions to consume. If this property is not set, then all partitions are consumed. `kafka_offsets` is a list of offsets, one for each partition listed in `kafka_partitions`. In this case the value is `OFFSET_BEGINNING` which causes all of the data to be consumed. The default is to only consume new data. *** #### Summary[​](#summary "Direct link to Summary") In this tutorial you: * Deployed StarRocks, Reedpanda, and Minio in Docker * Created a Routine Load job to consume data from a Kafka topic * Learned how to configure a StarRocks Storage Volume that uses MinIO #### More information[​](#more-information "Direct link to More information") [StarRocks Architecture](https://docs.starrocks.io/docs/introduction/Architecture.md) The sample used for this lab is very simple. Routine Load has many more options and capabilities. [learn more](https://docs.starrocks.io/docs/loading/RoutineLoad.md). [JSONPath](https://goessner.net/articles/JsonPath/) --- ### Separate storage and compute In systems that separate storage from compute data is stored in low-cost reliable remote storage systems such as Amazon S3, Google Cloud Storage, Azure Blob Storage, and other S3-compatible storage like MinIO. Hot data is cached locally and When the cache is hit, the query performance is comparable to that of storage-compute coupled architecture. Compute nodes (CN) can be added or removed on demand within seconds. This architecture reduces storage cost, ensures better resource isolation, and provides elasticity and scalability. This tutorial covers: * Running StarRocks in Docker containers * Using MinIO for Object Storage * Configuring StarRocks for shared-data * Loading two public datasets * Analyzing the data with SELECT and JOIN * Basic data transformation (the **T** in ETL) The data used is provided by NYC OpenData and the National Centers for Environmental Information at NOAA. Both of these datasets are very large, and because this tutorial is intended to help you get exposed to working with StarRocks we are not going to load data for the past 120 years. You can run the Docker image and load this data on a machine with 4 GB RAM assigned to Docker. For larger fault-tolerant and scalable deployments we have other documentation and will provide that later. There is a lot of information in this document, and it is presented with the step by step content at the beginning, and the technical details at the end. This is done to serve these purposes in this order: 1. Allow the reader to load data in a shared-data deployment and analyze that data. 2. Provide the configuration details for shared-data deployments. 3. Explain the basics of data transformation during loading. *** #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") ##### Docker[​](#docker "Direct link to Docker") * [Docker](https://docs.docker.com/engine/install/) * 4 GB RAM assigned to Docker * 10 GB free disk space assigned to Docker ##### SQL client[​](#sql-client "Direct link to SQL client") You can use the SQL client provided in the Docker environment, or use one on your system. Many MySQL compatible clients will work, and this guide covers the configuration of DBeaver and MySQL Workbench. ##### curl[​](#curl "Direct link to curl") `curl` is used to issue the data load job to StarRocks, and to download the datasets. Check to see if you have it installed by running `curl` or `curl.exe` at your OS prompt. If curl is not installed, [get curl here](https://curl.se/). ##### `/etc/hosts`[​](#etchosts "Direct link to etchosts") The ingest method used in this guide is Stream Load. Stream Load connects to the FE service to start the ingest job. The FE then assigns the job to a backend node, the CN in this guide. In order for the ingest job to connect to the CN the name of the CN must be available to your operating system. Add this line to `/etc/hosts`: ```bash 127.0.0.1 starrocks-cn ``` *** #### Terminology[​](#terminology "Direct link to Terminology") ##### FE[​](#fe "Direct link to FE") Frontend nodes are responsible for metadata management, client connection management, query planning, and query scheduling. Each FE stores and maintains a complete copy of metadata in its memory, which guarantees indiscriminate services among the FEs. ##### CN[​](#cn "Direct link to CN") Compute Nodes are responsible for executing query plans in shared-data deployments. ##### BE[​](#be "Direct link to BE") Backend nodes are responsible for both data storage and executing query plans in shared-nothing deployments. note This guide does not use BEs, this information is included here so that you understand the difference between BEs and CNs. *** #### Edit your hosts file[​](#edit-your-hosts-file "Direct link to Edit your hosts file") The ingest method used in this guide is Stream Load. Stream Load connects to the FE service to start the ingest job. The FE then assigns the job to a backend node—the CN in this guide. In order for the ingest job to connect to the CN, the name of the CN must be available to your operating system. Add this line to `/etc/hosts`: ```bash 127.0.0.1 starrocks-cn ``` #### Download the lab files[​](#download-the-lab-files "Direct link to Download the lab files") There are three files to download: * The Docker Compose file that deploys the StarRocks and MinIO environment * New York City crash data * Weather data This guide uses MinIO, which is S3 compatible Object Storage provided under the GNU Affero General Public License. ##### Create a directory to store the lab files[​](#create-a-directory-to-store-the-lab-files "Direct link to Create a directory to store the lab files") ```bash mkdir quickstart cd quickstart ``` ##### Download the Docker Compose file[​](#download-the-docker-compose-file "Direct link to Download the Docker Compose file") ```bash curl -O https://raw.githubusercontent.com/StarRocks/demo/master/documentation-samples/quickstart/docker-compose.yml ``` ##### Download the data[​](#download-the-data "Direct link to Download the data") Download these two datasets: ###### New York City crash data[​](#new-york-city-crash-data "Direct link to New York City crash data") ```bash curl -O https://raw.githubusercontent.com/StarRocks/demo/master/documentation-samples/quickstart/datasets/NYPD_Crash_Data.csv ``` ###### Weather data[​](#weather-data "Direct link to Weather data") ```bash curl -O https://raw.githubusercontent.com/StarRocks/demo/master/documentation-samples/quickstart/datasets/72505394728.csv ``` *** #### Deploy StarRocks and MinIO[​](#deploy-starrocks-and-minio "Direct link to Deploy StarRocks and MinIO") ```bash docker compose up --detach --wait --wait-timeout 120 ``` It should take around 30 seconds for the FE, CN, and MinIO services to become healthy. The `quickstart-minio_mc-1` container will show a status of `Waiting` and also an exit code. An exit code of `0` indicates success. ```bash [+] Running 4/5 ✔ Network quickstart_default Created 0.0s ✔ Container minio Healthy 6.8s ✔ Container starrocks-fe Healthy 29.3s ⠼ Container quickstart-minio_mc-1 Waiting 29.3s ✔ Container starrocks-cn Healthy 29.2s container quickstart-minio_mc-1 exited (0) ``` *** #### MinIO[​](#minio "Direct link to MinIO") This quick start uses MinIO for shared storage. ##### Verify the MinIO credentials[​](#verify-the-minio-credentials "Direct link to Verify the MinIO credentials") To use MinIO for Object Storage with StarRocks, StarRocks needs a MinIO access key. The access key was generated during the startup of the Docker services. To help you better understand the way that StarRocks connects to MinIO you should verify that the key exists. Browse to The username and password are specified in the Docker compose file, and are `miniouser` and `miniopassword`. You should see that there is one access key. The Key is `AAAAAAAAAAAAAAAAAAAA`, you cannot see the secret in the MinIO Console, but it is in the Docker compose file and is `BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB`: ![View the MinIO access key](/assets/images/MinIO-view-key-9df9610b4c42313f682f7ac6ecb1df07.png) tip If there are no access keys showing in the MinIO web UI, check the logs of the `minio_mc` service: ```bash docker compose logs minio_mc ``` Try rerunning the `minio_mc` pod: ```bash docker compose run minio_mc ``` ##### Create a bucket for your data[​](#create-a-bucket-for-your-data "Direct link to Create a bucket for your data") When you create a storage volume in StarRocks you will specify the `LOCATION` for the data: ```sh LOCATIONS = ("s3://my-starrocks-bucket/") ``` Open and add a bucket for the storage volume. Name the bucket `my-starrocks-bucket`. Accept the defaults for the three listed options. *** #### SQL Clients[​](#sql-clients "Direct link to SQL Clients") These three clients are tested with this tutorial, you only need one: * mysql CLI: You can run this from the Docker environment or your machine. * [DBeaver](https://dbeaver.io/download/) is available as a community version and a Pro version. * [MySQL Workbench](https://dev.mysql.com/downloads/workbench/) ###### Configuring the client[​](#configuring-the-client "Direct link to Configuring the client") * mysql CLI * DBeaver * MySQL Workbench The easiest way to use the mysql CLI is to run it from the StarRocks container `starrocks-fe`: ```bash docker compose exec starrocks-fe \ mysql -P 9030 -h 127.0.0.1 -u root --prompt="StarRocks > " ``` tip All `docker compose` commands must be run from the directory containing the `docker-compose.yml` file. If you would like to install the mysql CLI expand **mysql client install** below: mysql client install * macOS: If you use Homebrew and do not need MySQL Server run `brew install mysql-client@8.0` to install the CLI. * Linux: Check your repository system for the `mysql` client. For example, `yum install mariadb`. * Microsoft Windows: Install the [MySQL Community Server](https://dev.mysql.com/downloads/mysql/) and run the provided client, or run `mysql` from WSL. - Install [DBeaver](https://dbeaver.io/download/), and add a connection: ![Add a connection](/assets/images/DBeaver-1-58907bd9df44bba1e40282214e4a056c.png) - Configure the port, IP, and username. Test the connection, and click Finish if the test succeeds: ![Configure and test](/assets/images/DBeaver-2-8e4bdc09135b4738944d7016bfb1af94.png) * Install the [MySQL Workbench](https://dev.mysql.com/downloads/workbench/), and add a connection. * Configure the port, IP, and username and then test the connection: ![Connection settings](/assets/images/Workbench-1-9fd3a3126d5f83478e51891287e412b9.png) * You will see warnings from the Workbench as it is checking for a specific MySQL version. You can ignore the warnings and when prompted, you can configure Workbench to stop displaying the warnings: ![Ignore warning](/assets/images/Workbench-2-efba18763e56101091746837ae666b5e.png) *** #### StarRocks configuration for shared-data[​](#starrocks-configuration-for-shared-data "Direct link to StarRocks configuration for shared-data") At this point you have StarRocks running, and you have MinIO running. The MinIO access key is used to connect StarRocks and Minio. This is the part of the `FE` configuration that specifies that the StarRocks deployment will use shared data. This was added to the file `fe.conf` when Docker Compose created the deployment. ```sh # enable the shared data run mode run_mode = shared_data cloud_native_storage_type = S3 ``` info You can verify these settings by running this command from the `quickstart` directory and looking at the end of the file: ```sh docker compose exec starrocks-fe \ cat /opt/starrocks/fe/conf/fe.conf ``` ::: ##### Connect to StarRocks with a SQL client[​](#connect-to-starrocks-with-a-sql-client "Direct link to Connect to StarRocks with a SQL client") tip Run this command from the directory containing the `docker-compose.yml` file. If you are using a client other than the MySQL Command-Line Client, open that now. ```sql docker compose exec starrocks-fe \ mysql -P9030 -h127.0.0.1 -uroot --prompt="StarRocks > " ``` ###### Examine the storage volumes[​](#examine-the-storage-volumes "Direct link to Examine the storage volumes") ```sql SHOW STORAGE VOLUMES; ``` tip There should be no storage volumes, you will create one next. ```sh Empty set (0.04 sec) ``` ###### Create a shared-data storage volume[​](#create-a-shared-data-storage-volume "Direct link to Create a shared-data storage volume") Earlier you created a bucket in MinIO named `my-starrocks-volume`, and you verified that MinIO has an access key named `AAAAAAAAAAAAAAAAAAAA`. The following SQL will create a storage volume in the MionIO bucket using the access key and secret. ```sql CREATE STORAGE VOLUME s3_volume TYPE = S3 LOCATIONS = ("s3://my-starrocks-bucket/") PROPERTIES ( "enabled" = "true", "aws.s3.endpoint" = "minio:9000", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "aws.s3.use_instance_profile" = "false", "aws.s3.use_aws_sdk_default_behavior" = "false" ); ``` Now you should see a storage volume listed, earlier it was an empty set: ```text SHOW STORAGE VOLUMES; ``` ```text +----------------+ | Storage Volume | +----------------+ | s3_volume | +----------------+ 1 row in set (0.02 sec) ``` View the details of the storage volume and note that this is nott yet the default volume, and that it is configured to use your bucket: ```text DESC STORAGE VOLUME s3_volume\G ``` tip Some of the SQL in this document, and many other documents in the StarRocks documentation, and with `\G` instead of a semicolon. The `\G` causes the mysql CLI to render the query results vertically. Many SQL clients do not interpret vertical formatting output, so you should replace `\G` with `;`. ```sh *************************** 1. row *************************** Name: s3_volume Type: S3 IsDefault: false Location: s3://my-starrocks-bucket/ Params: {"aws.s3.access_key":"******","aws.s3.secret_key":"******","aws.s3.endpoint":"minio:9000","aws.s3.region":"us-east-1","aws.s3.use_instance_profile":"false","aws.s3.use_web_identity_token_file":"false","aws.s3.use_aws_sdk_default_behavior":"false"} Enabled: true Comment: 1 row in set (0.02 sec) ``` #### Set the default storage volume[​](#set-the-default-storage-volume "Direct link to Set the default storage volume") ```text SET s3_volume AS DEFAULT STORAGE VOLUME; ``` ```text DESC STORAGE VOLUME s3_volume\G ``` ```sh *************************** 1. row *************************** Name: s3_volume Type: S3 IsDefault: true Location: s3://my-starrocks-bucket/ Params: {"aws.s3.access_key":"******","aws.s3.secret_key":"******","aws.s3.endpoint":"minio:9000","aws.s3.region":"us-east-1","aws.s3.use_instance_profile":"false","aws.s3.use_web_identity_token_file":"false","aws.s3.use_aws_sdk_default_behavior":"false"} Enabled: true Comment: 1 row in set (0.02 sec) ``` #### Create a database[​](#create-a-database "Direct link to Create a database") ```text CREATE DATABASE IF NOT EXISTS quickstart; ``` Verify that the database `quickstart` is using the storage volume `s3_volume`: ```text SHOW CREATE DATABASE quickstart \G ``` ```sh *************************** 1. row *************************** Database: quickstart Create Database: CREATE DATABASE `quickstart` PROPERTIES ("storage_volume" = "s3_volume") ``` *** #### Create some tables[​](#create-some-tables "Direct link to Create some tables") ##### Create a database[​](#create-a-database "Direct link to Create a database") Type these two lines in at the `StarRocks > `prompt and press enter after each: ```sql CREATE DATABASE IF NOT EXISTS quickstart; USE quickstart; ``` ##### Create two tables[​](#create-two-tables "Direct link to Create two tables") ###### Crashdata[​](#crashdata "Direct link to Crashdata") The crash dataset contains many more fields than these, the schema has been trimmed down to include only the fields that might be useful to answer questions about the impact weather has on driving conditions. ```sql CREATE TABLE IF NOT EXISTS crashdata ( CRASH_DATE DATETIME, BOROUGH STRING, ZIP_CODE STRING, LATITUDE INT, LONGITUDE INT, LOCATION STRING, ON_STREET_NAME STRING, CROSS_STREET_NAME STRING, OFF_STREET_NAME STRING, CONTRIBUTING_FACTOR_VEHICLE_1 STRING, CONTRIBUTING_FACTOR_VEHICLE_2 STRING, COLLISION_ID INT, VEHICLE_TYPE_CODE_1 STRING, VEHICLE_TYPE_CODE_2 STRING ); ``` ###### Weatherdata[​](#weatherdata "Direct link to Weatherdata") Similar to the crash data, the weather dataset has many more columns (a total of 125 columns) and only the ones that are expected to answer the questions are included in the database. ```sql CREATE TABLE IF NOT EXISTS weatherdata ( DATE DATETIME, NAME STRING, HourlyDewPointTemperature STRING, HourlyDryBulbTemperature STRING, HourlyPrecipitation STRING, HourlyPresentWeatherType STRING, HourlyPressureChange STRING, HourlyPressureTendency STRING, HourlyRelativeHumidity STRING, HourlySkyConditions STRING, HourlyVisibility STRING, HourlyWetBulbTemperature STRING, HourlyWindDirection STRING, HourlyWindGustSpeed STRING, HourlyWindSpeed STRING ); ``` *** #### Load two datasets[​](#load-two-datasets "Direct link to Load two datasets") There are many ways to load data into StarRocks. For this tutorial the simplest way is to use curl and StarRocks Stream Load. tip Run these curl commands from the directory where you downloaded the dataset. You will be prompted for a password. You probably have not assigned a password to the MySQL `root` user, so just hit enter. The `curl` commands look complex, but they are explained in detail at the end of the tutorial. For now, we recommend running the commands and running some SQL to analyze the data, and then reading about the data loading details at the end. ##### New York City collision data - Crashes[​](#new-york-city-collision-data---crashes "Direct link to New York City collision data - Crashes") ```bash curl --location-trusted -u root \ -T ./NYPD_Crash_Data.csv \ -H "label:crashdata-0" \ -H "column_separator:," \ -H "skip_header:1" \ -H "enclose:\"" \ -H "max_filter_ratio:1" \ -H "columns:tmp_CRASH_DATE, tmp_CRASH_TIME, CRASH_DATE=str_to_date(concat_ws(' ', tmp_CRASH_DATE, tmp_CRASH_TIME), '%m/%d/%Y %H:%i'),BOROUGH,ZIP_CODE,LATITUDE,LONGITUDE,LOCATION,ON_STREET_NAME,CROSS_STREET_NAME,OFF_STREET_NAME,NUMBER_OF_PERSONS_INJURED,NUMBER_OF_PERSONS_KILLED,NUMBER_OF_PEDESTRIANS_INJURED,NUMBER_OF_PEDESTRIANS_KILLED,NUMBER_OF_CYCLIST_INJURED,NUMBER_OF_CYCLIST_KILLED,NUMBER_OF_MOTORIST_INJURED,NUMBER_OF_MOTORIST_KILLED,CONTRIBUTING_FACTOR_VEHICLE_1,CONTRIBUTING_FACTOR_VEHICLE_2,CONTRIBUTING_FACTOR_VEHICLE_3,CONTRIBUTING_FACTOR_VEHICLE_4,CONTRIBUTING_FACTOR_VEHICLE_5,COLLISION_ID,VEHICLE_TYPE_CODE_1,VEHICLE_TYPE_CODE_2,VEHICLE_TYPE_CODE_3,VEHICLE_TYPE_CODE_4,VEHICLE_TYPE_CODE_5" \ -XPUT http://localhost:8030/api/quickstart/crashdata/_stream_load ``` Here is the output of the above command. The first highlighted section shown what you should expect to see (OK and all but one row inserted). One row was filtered out because it does not contain the correct number of columns. ```bash Enter host password for user 'root': { "TxnId": 2, "Label": "crashdata-0", "Status": "Success", "Message": "OK", "NumberTotalRows": 423726, "NumberLoadedRows": 423725, "NumberFilteredRows": 1, "NumberUnselectedRows": 0, "LoadBytes": 96227746, "LoadTimeMs": 1013, "BeginTxnTimeMs": 21, "StreamLoadPlanTimeMs": 63, "ReadDataTimeMs": 563, "WriteDataTimeMs": 870, "CommitAndPublishTimeMs": 57, "ErrorURL": "http://starrocks-cn:8040/api/_load_error_log?file=error_log_da41dd88276a7bfc_739087c94262ae9f" }% ``` If there was an error the output provides a URL to see the error messages. The error message also contains the backend node that the Stream Load job was assigned to (`starrocks-cn`). Because you added an entry for `starrocks-cn` to the `/etc/hosts` file, you should be able to navigate to it and read the error message. Expand the summary for the content seen while developing this tutorial: Reading error messages in the browser ```bash Error: Value count does not match column count. Expect 29, but got 32. Column delimiter: 44,Row delimiter: 10.. Row: 09/06/2015,14:15,,,40.6722269,-74.0110059,"(40.6722269, -74.0110059)",,,"R/O 1 BEARD ST. ( IKEA'S 09/14/2015,5:30,BRONX,10473,40.814551,-73.8490955,"(40.814551, -73.8490955)",TORRY AVENUE ,NORTON AVENUE ,,0,0,0,0,0,0,0,0,Driver Inattention/Distraction,Unspecified,,,,3297457,PASSENGER VEHICLE,PASSENGER VEHICLE,,, ``` ##### Weather data[​](#weather-data-1 "Direct link to Weather data") Load the weather dataset in the same manner as you loaded the crash data. ```bash curl --location-trusted -u root \ -T ./72505394728.csv \ -H "label:weather-0" \ -H "column_separator:," \ -H "skip_header:1" \ -H "enclose:\"" \ -H "max_filter_ratio:1" \ -H "columns: STATION, DATE, LATITUDE, LONGITUDE, ELEVATION, NAME, REPORT_TYPE, SOURCE, HourlyAltimeterSetting, HourlyDewPointTemperature, HourlyDryBulbTemperature, HourlyPrecipitation, HourlyPresentWeatherType, HourlyPressureChange, HourlyPressureTendency, HourlyRelativeHumidity, HourlySkyConditions, HourlySeaLevelPressure, HourlyStationPressure, HourlyVisibility, HourlyWetBulbTemperature, HourlyWindDirection, HourlyWindGustSpeed, HourlyWindSpeed, Sunrise, Sunset, DailyAverageDewPointTemperature, DailyAverageDryBulbTemperature, DailyAverageRelativeHumidity, DailyAverageSeaLevelPressure, DailyAverageStationPressure, DailyAverageWetBulbTemperature, DailyAverageWindSpeed, DailyCoolingDegreeDays, DailyDepartureFromNormalAverageTemperature, DailyHeatingDegreeDays, DailyMaximumDryBulbTemperature, DailyMinimumDryBulbTemperature, DailyPeakWindDirection, DailyPeakWindSpeed, DailyPrecipitation, DailySnowDepth, DailySnowfall, DailySustainedWindDirection, DailySustainedWindSpeed, DailyWeather, MonthlyAverageRH, MonthlyDaysWithGT001Precip, MonthlyDaysWithGT010Precip, MonthlyDaysWithGT32Temp, MonthlyDaysWithGT90Temp, MonthlyDaysWithLT0Temp, MonthlyDaysWithLT32Temp, MonthlyDepartureFromNormalAverageTemperature, MonthlyDepartureFromNormalCoolingDegreeDays, MonthlyDepartureFromNormalHeatingDegreeDays, MonthlyDepartureFromNormalMaximumTemperature, MonthlyDepartureFromNormalMinimumTemperature, MonthlyDepartureFromNormalPrecipitation, MonthlyDewpointTemperature, MonthlyGreatestPrecip, MonthlyGreatestPrecipDate, MonthlyGreatestSnowDepth, MonthlyGreatestSnowDepthDate, MonthlyGreatestSnowfall, MonthlyGreatestSnowfallDate, MonthlyMaxSeaLevelPressureValue, MonthlyMaxSeaLevelPressureValueDate, MonthlyMaxSeaLevelPressureValueTime, MonthlyMaximumTemperature, MonthlyMeanTemperature, MonthlyMinSeaLevelPressureValue, MonthlyMinSeaLevelPressureValueDate, MonthlyMinSeaLevelPressureValueTime, MonthlyMinimumTemperature, MonthlySeaLevelPressure, MonthlyStationPressure, MonthlyTotalLiquidPrecipitation, MonthlyTotalSnowfall, MonthlyWetBulb, AWND, CDSD, CLDD, DSNW, HDSD, HTDD, NormalsCoolingDegreeDay, NormalsHeatingDegreeDay, ShortDurationEndDate005, ShortDurationEndDate010, ShortDurationEndDate015, ShortDurationEndDate020, ShortDurationEndDate030, ShortDurationEndDate045, ShortDurationEndDate060, ShortDurationEndDate080, ShortDurationEndDate100, ShortDurationEndDate120, ShortDurationEndDate150, ShortDurationEndDate180, ShortDurationPrecipitationValue005, ShortDurationPrecipitationValue010, ShortDurationPrecipitationValue015, ShortDurationPrecipitationValue020, ShortDurationPrecipitationValue030, ShortDurationPrecipitationValue045, ShortDurationPrecipitationValue060, ShortDurationPrecipitationValue080, ShortDurationPrecipitationValue100, ShortDurationPrecipitationValue120, ShortDurationPrecipitationValue150, ShortDurationPrecipitationValue180, REM, BackupDirection, BackupDistance, BackupDistanceUnit, BackupElements, BackupElevation, BackupEquipment, BackupLatitude, BackupLongitude, BackupName, WindEquipmentChangeDate" \ -XPUT http://localhost:8030/api/quickstart/weatherdata/_stream_load ``` *** #### Verify that data is stored in MinIO[​](#verify-that-data-is-stored-in-minio "Direct link to Verify that data is stored in MinIO") Open MinIO and verify that you have entries below `my-starrocks-bucket/` tip The folder names below `my-starrocks-bucket/` are generated when you load the data. You should see a single directory below `my-starrocks-bucket`, and then two more below that. In those directories you will find the data, metadata, or schema entries. ![MinIO object browser](/assets/images/MinIO-data-1c9d4bdab9cdda10f4526ba5c41673c9.png) *** #### Answer some questions[​](#answer-some-questions "Direct link to Answer some questions") These queries can be run in your SQL client. All of the queries use the `quickstart` database. ```sql USE quickstart; ``` ###### How many crashes are there per hour in NYC?[​](#how-many-crashes-are-there-per-hour-in-nyc "Direct link to How many crashes are there per hour in NYC?") ```sql SELECT COUNT(*), date_trunc("hour", crashdata.CRASH_DATE) AS Time FROM crashdata GROUP BY Time ORDER BY Time ASC LIMIT 200; ``` Here is part of the output. Note that I am looking closer at January 6th and 7th as this is Monday and Tuesday of a non-holiday week. Looking at New Years Day is probably not indicative of a normal morning during rush-hour traffic. ```plaintext | 14 | 2014-01-06 06:00:00 | | 16 | 2014-01-06 07:00:00 | | 43 | 2014-01-06 08:00:00 | | 44 | 2014-01-06 09:00:00 | | 21 | 2014-01-06 10:00:00 | | 28 | 2014-01-06 11:00:00 | | 34 | 2014-01-06 12:00:00 | | 31 | 2014-01-06 13:00:00 | | 35 | 2014-01-06 14:00:00 | | 36 | 2014-01-06 15:00:00 | | 33 | 2014-01-06 16:00:00 | | 40 | 2014-01-06 17:00:00 | | 35 | 2014-01-06 18:00:00 | | 23 | 2014-01-06 19:00:00 | | 16 | 2014-01-06 20:00:00 | | 12 | 2014-01-06 21:00:00 | | 17 | 2014-01-06 22:00:00 | | 14 | 2014-01-06 23:00:00 | | 10 | 2014-01-07 00:00:00 | | 4 | 2014-01-07 01:00:00 | | 1 | 2014-01-07 02:00:00 | | 3 | 2014-01-07 03:00:00 | | 2 | 2014-01-07 04:00:00 | | 6 | 2014-01-07 06:00:00 | | 16 | 2014-01-07 07:00:00 | | 41 | 2014-01-07 08:00:00 | | 37 | 2014-01-07 09:00:00 | | 33 | 2014-01-07 10:00:00 | ``` It looks like about 40 accidents on a Monday or Tuesday morning during rush hour traffic, and around the same at 17:00 hours. ###### What is the average temperature in NYC?[​](#what-is-the-average-temperature-in-nyc "Direct link to What is the average temperature in NYC?") ```sql SELECT avg(HourlyDryBulbTemperature), date_trunc("hour", weatherdata.DATE) AS Time FROM weatherdata GROUP BY Time ORDER BY Time ASC LIMIT 100; ``` Output: Note that this is data from 2014, NYC has not been this cold lately. ```plaintext +-------------------------------+---------------------+ | avg(HourlyDryBulbTemperature) | Time | +-------------------------------+---------------------+ | 25 | 2014-01-01 00:00:00 | | 25 | 2014-01-01 01:00:00 | | 24 | 2014-01-01 02:00:00 | | 24 | 2014-01-01 03:00:00 | | 24 | 2014-01-01 04:00:00 | | 24 | 2014-01-01 05:00:00 | | 25 | 2014-01-01 06:00:00 | | 26 | 2014-01-01 07:00:00 | ``` ###### Is it safe to drive in NYC when visibility is poor?[​](#is-it-safe-to-drive-in-nyc-when-visibility-is-poor "Direct link to Is it safe to drive in NYC when visibility is poor?") Let's look at the number of crashes when visibility is poor (between 0 and 1.0 miles). To answer this question use a JOIN across the two tables on the DATETIME column. ```sql SELECT COUNT(DISTINCT c.COLLISION_ID) AS Crashes, truncate(avg(w.HourlyDryBulbTemperature), 1) AS Temp_F, truncate(avg(w.HourlyVisibility), 2) AS Visibility, max(w.HourlyPrecipitation) AS Precipitation, date_format((date_trunc("hour", c.CRASH_DATE)), '%d %b %Y %H:%i') AS Hour FROM crashdata c LEFT JOIN weatherdata w ON date_trunc("hour", c.CRASH_DATE)=date_trunc("hour", w.DATE) WHERE w.HourlyVisibility BETWEEN 0.0 AND 1.0 GROUP BY Hour ORDER BY Crashes DESC LIMIT 100; ``` The highest number of crashes in a single hour during low visibility is 129. There are multiple things to consider: * February 3rd 2014 was a Monday * 8AM is rush hour * It was raining (0.12 inches or precipitation that hour) * The temperature is 32 degrees F (the freezing point for water) * Visibility is bad at 0.25 miles, normal for NYC is 10 miles ```plaintext +---------+--------+------------+---------------+-------------------+ | Crashes | Temp_F | Visibility | Precipitation | Hour | +---------+--------+------------+---------------+-------------------+ | 129 | 32 | 0.25 | 0.12 | 03 Feb 2014 08:00 | | 114 | 32 | 0.25 | 0.12 | 03 Feb 2014 09:00 | | 104 | 23 | 0.33 | 0.03 | 09 Jan 2015 08:00 | | 96 | 26.3 | 0.33 | 0.07 | 01 Mar 2015 14:00 | | 95 | 26 | 0.37 | 0.12 | 01 Mar 2015 15:00 | | 93 | 35 | 0.75 | 0.09 | 18 Jan 2015 09:00 | | 92 | 31 | 0.25 | 0.12 | 03 Feb 2014 10:00 | | 87 | 26.8 | 0.5 | 0.09 | 01 Mar 2015 16:00 | | 85 | 55 | 0.75 | 0.20 | 23 Dec 2015 17:00 | | 85 | 20 | 0.62 | 0.01 | 06 Jan 2015 11:00 | | 83 | 19.6 | 0.41 | 0.04 | 05 Mar 2015 13:00 | | 80 | 20 | 0.37 | 0.02 | 06 Jan 2015 10:00 | | 76 | 26.5 | 0.25 | 0.06 | 05 Mar 2015 09:00 | | 71 | 26 | 0.25 | 0.09 | 05 Mar 2015 10:00 | | 71 | 24.2 | 0.25 | 0.04 | 05 Mar 2015 11:00 | ``` ###### What about driving in icy conditions?[​](#what-about-driving-in-icy-conditions "Direct link to What about driving in icy conditions?") Water vapor can desublimate to ice at 40 degrees F; this query looks at temps between 0 and 40 degrees F. ```sql SELECT COUNT(DISTINCT c.COLLISION_ID) AS Crashes, truncate(avg(w.HourlyDryBulbTemperature), 1) AS Temp_F, truncate(avg(w.HourlyVisibility), 2) AS Visibility, max(w.HourlyPrecipitation) AS Precipitation, date_format((date_trunc("hour", c.CRASH_DATE)), '%d %b %Y %H:%i') AS Hour FROM crashdata c LEFT JOIN weatherdata w ON date_trunc("hour", c.CRASH_DATE)=date_trunc("hour", w.DATE) WHERE w.HourlyDryBulbTemperature BETWEEN 0.0 AND 40.5 GROUP BY Hour ORDER BY Crashes DESC LIMIT 100; ``` The results for freezing temperatures suprised me a little, I did not expect too much traffic on a Sunday morning in the city on a cold January day.A quick look at [weather.com](https://weather.com/storms/winter/news/northeast-storm-rain-snow-wind) showed that there was a big storm with many crashes that day, just like what can be seen in the data. ```plaintext +---------+--------+------------+---------------+-------------------+ | Crashes | Temp_F | Visibility | Precipitation | Hour | +---------+--------+------------+---------------+-------------------+ | 192 | 34 | 1.5 | 0.09 | 18 Jan 2015 08:00 | | 170 | 21 | NULL | | 21 Jan 2014 10:00 | | 145 | 19 | NULL | | 21 Jan 2014 11:00 | | 138 | 33.5 | 5 | 0.02 | 18 Jan 2015 07:00 | | 137 | 21 | NULL | | 21 Jan 2014 09:00 | | 129 | 32 | 0.25 | 0.12 | 03 Feb 2014 08:00 | | 114 | 32 | 0.25 | 0.12 | 03 Feb 2014 09:00 | | 104 | 23 | 0.7 | 0.04 | 09 Jan 2015 08:00 | | 98 | 16 | 8 | 0.00 | 06 Mar 2015 08:00 | | 96 | 26.3 | 0.33 | 0.07 | 01 Mar 2015 14:00 | ``` Drive carefully! *** #### Configuring StarRocks for shared-data[​](#configuring-starrocks-for-shared-data "Direct link to Configuring StarRocks for shared-data") Now that you have experienced using StarRocks with shared-data it is important to understand the configuration. ##### CN configuration[​](#cn-configuration "Direct link to CN configuration") The CN configuration used here is the default, as the CN is designed for shared-data use. The default configuration is shown below. You do not need to make any changes. ```bash sys_log_level = INFO # ports for admin, web, heartbeat service be_port = 9060 be_http_port = 8040 heartbeat_service_port = 9050 brpc_port = 8060 starlet_port = 9070 ``` ##### FE configuration[​](#fe-configuration "Direct link to FE configuration") The FE configuration is slightly different from the default as the FE must be configured to expect that data is stored in Object Storage rather than on local disks on BE nodes. The `docker-compose.yml` file generates the FE configuration in the `command`. ```plaintext # enable shared data, set storage type, set endpoint run_mode = shared_data cloud_native_storage_type = S3 ``` note This config file does not contain the default entries for an FE, only the shared-data configuration is shown. The non-default FE configuration settings: note Many configuration parameters are prefixed with `s3_`. This prefix is used for all Amazon S3 compatible storage types (for example: S3, GCS, and MinIO). When using Azure Blob Storage the prefix is `azure_`. ###### `run_mode=shared_data`[​](#run_modeshared_data "Direct link to run_modeshared_data") This enables shared-data use. ###### `cloud_native_storage_type=S3`[​](#cloud_native_storage_types3 "Direct link to cloud_native_storage_types3") This specifies whether S3 compatible storage or Azure Blob Storage is used. For MinIO this is always S3. ##### Details of `CREATE storage volume`[​](#details-of-create-storage-volume "Direct link to details-of-create-storage-volume") ```sql CREATE STORAGE VOLUME s3_volume TYPE = S3 LOCATIONS = ("s3://my-starrocks-bucket/") PROPERTIES ( "enabled" = "true", "aws.s3.endpoint" = "minio:9000", "aws.s3.access_key" = "AAAAAAAAAAAAAAAAAAAA", "aws.s3.secret_key" = "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB", "aws.s3.use_instance_profile" = "false", "aws.s3.use_aws_sdk_default_behavior" = "false" ); ``` ###### `aws_s3_endpoint=minio:9000`[​](#aws_s3_endpointminio9000 "Direct link to aws_s3_endpointminio9000") The MinIO endpoint, including port number. ###### `aws_s3_path=starrocks`[​](#aws_s3_pathstarrocks "Direct link to aws_s3_pathstarrocks") The bucket name. ###### `aws_s3_access_key=AAAAAAAAAAAAAAAAAAAA`[​](#aws_s3_access_keyaaaaaaaaaaaaaaaaaaaa "Direct link to aws_s3_access_keyaaaaaaaaaaaaaaaaaaaa") The MinIO access key. ###### `aws_s3_secret_key=BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB`[​](#aws_s3_secret_keybbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb "Direct link to aws_s3_secret_keybbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb") The MinIO access key secret. ###### `aws_s3_use_instance_profile=false`[​](#aws_s3_use_instance_profilefalse "Direct link to aws_s3_use_instance_profilefalse") When using MinIO an access key is used, and so instance profiles are not used with MinIO. ###### `aws_s3_use_aws_sdk_default_behavior=false`[​](#aws_s3_use_aws_sdk_default_behaviorfalse "Direct link to aws_s3_use_aws_sdk_default_behaviorfalse") When using MinIO this parameter is always set to false. ##### Configuring FQDN mode[​](#configuring-fqdn-mode "Direct link to Configuring FQDN mode") The command to start the FE is also changed. The FE service command in the Docker Compose file has the option `--host_type FQDN` added. By setting `host_type` to `FQDN` the Stream Load job is forwarded to the fully qualified domain name of the CN pod, rather than the IP address. This is done because the IP address is in a range assigned to the Docker environment, and is not typically available from the host machine. These three changes allow traffic between the host network and the CN: * setting `--host_type` to `FQDN` * exposing the CN port 8040 to the host network * adding an entry to the hosts file for `starrocks-cn` pointing to `127.0.0.1` *** #### Summary[​](#summary "Direct link to Summary") In this tutorial you: * Deployed StarRocks and Minio in Docker * Created a MinIO access key * Configured a StarRocks Storage Volume that uses MinIO * Loaded crash data provided by New York City and weather data provided by NOAA * Analyzed the data using SQL JOINs to find out that driving in low visibility or icy streets is a bad idea There is more to learn; we intentionally glossed over the data transform done during the Stream Load. The details on that are in the notes on the curl commands below. #### Notes on the curl commands[​](#notes-on-the-curl-commands "Direct link to Notes on the curl commands") StarRocks Stream Load and curl take many arguments. Only the ones used in this tutorial are described here, the rest will be linked to in the more information section. ###### `--location-trusted`[​](#--location-trusted "Direct link to --location-trusted") This configures curl to pass credentials to any redirected URLs. ###### `-u root`[​](#-u-root "Direct link to -u-root") The username used to log in to StarRocks ###### `-T filename`[​](#-t-filename "Direct link to -t-filename") T is for transfer, the filename to transfer. ###### `label:name-num`[​](#labelname-num "Direct link to labelname-num") The label to associate with this Stream Load job. The label must be unique, so if you run the job multiple times you can add a number and keep incrementing that. ###### `column_separator:,`[​](#column_separator "Direct link to column_separator") If you load a file that uses a single `,` then set it as shown above, if you use a different delimiter then set that delimiter here. Common choices are `\t`, `,`, and `|`. ###### `skip_header:1`[​](#skip_header1 "Direct link to skip_header1") Some CSV files have a single header row with all of the column names listed, and some add a second line with datatypes. Set skip\_header to `1` or `2` if you have one or two header lines, and set it to `0` if you have none. ###### `enclose:\"`[​](#enclose "Direct link to enclose") It is common to enclose strings that contain embedded commas with double-quotes. The sample datasets used in this tutorial have geo locations that contain commas and so the enclose setting is set to `\"`. Remember to escape the `"` with a `\`. ###### `max_filter_ratio:1`[​](#max_filter_ratio1 "Direct link to max_filter_ratio1") This allows some errors in the data. Ideally this would be set to `0` and the job would fail with any errors. It is set to `1` to allow all rows to fail during debugging. ###### `columns:`[​](#columns "Direct link to columns") The mapping of CSV file columns to StarRocks table columns. You will notice that there are many more columns in the CSV files than columns in the table. Any columns that are not included in the table are skipped. You will also notice that there is some transformation of data included in the `columns:` line for the crash dataset. It is very common to find dates and times in CSV files that do not conform to standards. This is the logic for converting the CSV data for the time and date of the crash to a DATETIME type: ###### The columns line[​](#the-columns-line "Direct link to The columns line") This is the beginning of one data record. The date is in `MM/DD/YYYY` format, and the time is `HH:MI`. Since DATETIME is generally `YYYY-MM-DD HH:MI:SS` we need to transform this data. ```plaintext 08/05/2014,9:10,BRONX,10469,40.8733019,-73.8536375,"(40.8733019, -73.8536375)", ``` This is the beginning of the `columns:` parameter: ```bash -H "columns:tmp_CRASH_DATE, tmp_CRASH_TIME, CRASH_DATE=str_to_date(concat_ws(' ', tmp_CRASH_DATE, tmp_CRASH_TIME), '%m/%d/%Y %H:%i') ``` This instructs StarRocks to: * Assign the content of the first column of the CSV file to `tmp_CRASH_DATE` * Assign the content of the second column of the CSV file to `tmp_CRASH_TIME` * `concat_ws()` concatenates `tmp_CRASH_DATE` and `tmp_CRASH_TIME` together with a space between them * `str_to_date()` creates a DATETIME from the concatenated string * store the resulting DATETIME in the column `CRASH_DATE` #### More information[​](#more-information "Direct link to More information") [StarRocks table design](https://docs.starrocks.io/docs/table_design/StarRocks_table_design.md) [Stream Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md) The [Motor Vehicle Collisions - Crashes](https://data.cityofnewyork.us/Public-Safety/Motor-Vehicle-Collisions-Crashes/h9gi-nx95) dataset is provided by New York City subject to these [terms of use](https://www.nyc.gov/home/terms-of-use.page) and [privacy policy](https://www.nyc.gov/home/privacy-policy.page). The [Local Climatological Data](https://www.ncdc.noaa.gov/cdo-web/datatools/lcd)(LCD) is provided by NOAA with this [disclaimer](https://www.noaa.gov/disclaimer) and this [privacy policy](https://www.noaa.gov/protecting-your-privacy). --- ### Deploy StarRocks with Docker This tutorial covers: * Running StarRocks in a single Docker container * Loading two public datasets including basic transformation of the data * Analyzing the data with SELECT and JOIN * Basic data transformation (the **T** in ETL) #### Follow along with the video if you prefer[​](#follow-along-with-the-video-if-you-prefer "Direct link to Follow along with the video if you prefer") [StarRocks in 5 - Getting Started With StarRocks on Docker](https://www.youtube.com/embed/h7F4U6xEA5M) The data used is provided by NYC OpenData and the National Centers for Environmental Information. Both of these datasets are very large, and because this tutorial is intended to help you get exposed to working with StarRocks we are not going to load data for the past 120 years. You can run the Docker image and load this data on a machine with 4 GB RAM assigned to Docker. For larger fault-tolerant and scalable deployments we have other documentation and will provide that later. There is a lot of information in this document, and it is presented with the step by step content at the beginning, and the technical details at the end. This is done to serve these purposes in this order: 1. Allow the reader to load data in StarRocks and analyze that data. 2. Explain the basics of data transformation during loading. *** #### Prerequisites[​](#prerequisites "Direct link to Prerequisites") ##### Docker[​](#docker "Direct link to Docker") * [Docker](https://docs.docker.com/engine/install/) * 4 GB RAM assigned to Docker * 10 GB free disk space assigned to Docker ##### SQL client[​](#sql-client "Direct link to SQL client") You can use the SQL client provided in the Docker environment, or use one on your system. Many MySQL compatible clients will work, and this guide covers the configuration of DBeaver and MySQL Workbench. ##### curl[​](#curl "Direct link to curl") `curl` is used to issue the data load job to StarRocks, and to download the datasets. Check to see if you have it installed by running `curl` or `curl.exe` at your OS prompt. If curl is not installed, [get curl here](https://curl.se/download.html). *** #### Terminology[​](#terminology "Direct link to Terminology") ##### FE[​](#fe "Direct link to FE") Frontend nodes are responsible for metadata management, client connection management, query planning, and query scheduling. Each FE stores and maintains a complete copy of metadata in its memory, which guarantees indiscriminate services among the FEs. ##### BE[​](#be "Direct link to BE") Backend nodes are responsible for both data storage and executing query plans. *** #### Launch StarRocks[​](#launch-starrocks "Direct link to Launch StarRocks") ```bash docker pull starrocks/allin1-ubuntu docker run -p 9030:9030 -p 8030:8030 -p 8040:8040 -itd \ --name quickstart starrocks/allin1-ubuntu ``` *** #### SQL clients[​](#sql-clients "Direct link to SQL clients") These three clients are tested with this tutorial, you only need one: * mysql CLI: You can run this from the Docker environment or your machine. * [DBeaver](https://dbeaver.io/download/) is available as a community version and a Pro version. * [MySQL Workbench](https://dev.mysql.com/downloads/workbench/) ###### Configuring the client[​](#configuring-the-client "Direct link to Configuring the client") * mysql CLI * DBeaver * MySQL Workbench The easiest way to use the mysql CLI is to run it from the StarRocks container `starrocks-fe`: ```bash docker exec -it quickstart \ mysql -P 9030 -h 127.0.0.1 -u root --prompt="StarRocks > " ``` If you would like to install the mysql CLI expand **mysql client install** below: mysql client install * macOS: If you use Homebrew and do not need MySQL Server run `brew install mysql-client@8.0` to install the CLI. * Linux: Check your repository system for the `mysql` client. For example, `yum install mariadb`. * Microsoft Windows: Install the [MySQL Community Server](https://dev.mysql.com/downloads/mysql/) and run the provided client, or run `mysql` from WSL. - Install [DBeaver](https://dbeaver.io/download/), and add a connection: ![Add a connection](/assets/images/DBeaver-1-58907bd9df44bba1e40282214e4a056c.png) - Configure the port, IP, and username. Test the connection, and click Finish if the test succeeds: ![Configure and test](/assets/images/DBeaver-2-8e4bdc09135b4738944d7016bfb1af94.png) * Install the [MySQL Workbench](https://dev.mysql.com/downloads/workbench/), and add a connection. * Configure the port, IP, and username and then test the connection: ![Connection settings](/assets/images/Workbench-1-9fd3a3126d5f83478e51891287e412b9.png) * You will see warnings from the Workbench as it is checking for a specific MySQL version. You can ignore the warnings and when prompted, you can configure Workbench to stop displaying the warnings: ![Ignore warning](/assets/images/Workbench-2-efba18763e56101091746837ae666b5e.png) *** #### Download the data[​](#download-the-data "Direct link to Download the data") Download these two datasets to your machine. You can download them to the host machine where you are running Docker, they do not need to be downloaded inside the container. ##### New York City crash data[​](#new-york-city-crash-data "Direct link to New York City crash data") ```bash curl -O https://raw.githubusercontent.com/StarRocks/demo/master/documentation-samples/quickstart/datasets/NYPD_Crash_Data.csv ``` ##### Weather data[​](#weather-data "Direct link to Weather data") ```bash curl -O https://raw.githubusercontent.com/StarRocks/demo/master/documentation-samples/quickstart/datasets/72505394728.csv ``` *** ##### Connect to StarRocks with a SQL client[​](#connect-to-starrocks-with-a-sql-client "Direct link to Connect to StarRocks with a SQL client") tip If you are using a client other than the mysql CLI, open that now. This command will run the `mysql` command in the Docker container: ```sql docker exec -it quickstart \ mysql -P 9030 -h 127.0.0.1 -u root --prompt="StarRocks > " ``` *** #### Create some tables[​](#create-some-tables "Direct link to Create some tables") ##### Create a database[​](#create-a-database "Direct link to Create a database") Type these two lines in at the `StarRocks > `prompt and press enter after each: ```sql CREATE DATABASE IF NOT EXISTS quickstart; USE quickstart; ``` ##### Create two tables[​](#create-two-tables "Direct link to Create two tables") ###### Crashdata[​](#crashdata "Direct link to Crashdata") The crash dataset contains many more fields than these, the schema has been trimmed down to include only the fields that might be useful to answer questions about the impact weather has on driving conditions. ```sql CREATE TABLE IF NOT EXISTS crashdata ( CRASH_DATE DATETIME, BOROUGH STRING, ZIP_CODE STRING, LATITUDE INT, LONGITUDE INT, LOCATION STRING, ON_STREET_NAME STRING, CROSS_STREET_NAME STRING, OFF_STREET_NAME STRING, CONTRIBUTING_FACTOR_VEHICLE_1 STRING, CONTRIBUTING_FACTOR_VEHICLE_2 STRING, COLLISION_ID INT, VEHICLE_TYPE_CODE_1 STRING, VEHICLE_TYPE_CODE_2 STRING ); ``` ###### Weatherdata[​](#weatherdata "Direct link to Weatherdata") Similar to the crash data, the weather dataset has many more columns (a total of 125 columns) and only the ones that are expected to answer the questions are included in the database. ```sql CREATE TABLE IF NOT EXISTS weatherdata ( DATE DATETIME, NAME STRING, HourlyDewPointTemperature STRING, HourlyDryBulbTemperature STRING, HourlyPrecipitation STRING, HourlyPresentWeatherType STRING, HourlyPressureChange STRING, HourlyPressureTendency STRING, HourlyRelativeHumidity STRING, HourlySkyConditions STRING, HourlyVisibility STRING, HourlyWetBulbTemperature STRING, HourlyWindDirection STRING, HourlyWindGustSpeed STRING, HourlyWindSpeed STRING ); ``` *** #### Load two datasets[​](#load-two-datasets "Direct link to Load two datasets") There are many ways to load data into StarRocks. For this tutorial the simplest way is to use curl and StarRocks Stream Load. tip Open a new shell as these curl commands are run at the operating system prompt, not in the `mysql` client. The commands refer to the datasets that you downloaded, so run them from the directory where you downloaded the files. You will be prompted for a password. You probably have not assigned a password to the MySQL `root` user, so just hit enter. The `curl` commands look complex, but they are explained in detail at the end of the tutorial. For now, we recommend running the commands and running some SQL to analyze the data, and then reading about the data loading details at the end. ##### New York City collision data - Crashes[​](#new-york-city-collision-data---crashes "Direct link to New York City collision data - Crashes") ```bash curl --location-trusted -u root \ -T ./NYPD_Crash_Data.csv \ -H "label:crashdata-0" \ -H "column_separator:," \ -H "skip_header:1" \ -H "enclose:\"" \ -H "max_filter_ratio:1" \ -H "columns:tmp_CRASH_DATE, tmp_CRASH_TIME, CRASH_DATE=str_to_date(concat_ws(' ', tmp_CRASH_DATE, tmp_CRASH_TIME), '%m/%d/%Y %H:%i'),BOROUGH,ZIP_CODE,LATITUDE,LONGITUDE,LOCATION,ON_STREET_NAME,CROSS_STREET_NAME,OFF_STREET_NAME,NUMBER_OF_PERSONS_INJURED,NUMBER_OF_PERSONS_KILLED,NUMBER_OF_PEDESTRIANS_INJURED,NUMBER_OF_PEDESTRIANS_KILLED,NUMBER_OF_CYCLIST_INJURED,NUMBER_OF_CYCLIST_KILLED,NUMBER_OF_MOTORIST_INJURED,NUMBER_OF_MOTORIST_KILLED,CONTRIBUTING_FACTOR_VEHICLE_1,CONTRIBUTING_FACTOR_VEHICLE_2,CONTRIBUTING_FACTOR_VEHICLE_3,CONTRIBUTING_FACTOR_VEHICLE_4,CONTRIBUTING_FACTOR_VEHICLE_5,COLLISION_ID,VEHICLE_TYPE_CODE_1,VEHICLE_TYPE_CODE_2,VEHICLE_TYPE_CODE_3,VEHICLE_TYPE_CODE_4,VEHICLE_TYPE_CODE_5" \ -XPUT http://localhost:8030/api/quickstart/crashdata/_stream_load ``` Here is the output of the preceding command. The first highlighted section shows what you should expect to see (OK and all but one row inserted). One row was filtered out because it does not contain the correct number of columns. ```bash Enter host password for user 'root': { "TxnId": 2, "Label": "crashdata-0", "Status": "Success", "Message": "OK", "NumberTotalRows": 423726, "NumberLoadedRows": 423725, "NumberFilteredRows": 1, "NumberUnselectedRows": 0, "LoadBytes": 96227746, "LoadTimeMs": 1013, "BeginTxnTimeMs": 21, "StreamLoadPlanTimeMs": 63, "ReadDataTimeMs": 563, "WriteDataTimeMs": 870, "CommitAndPublishTimeMs": 57, "ErrorURL": "http://127.0.0.1:8040/api/_load_error_log?file=error_log_da41dd88276a7bfc_739087c94262ae9f" }% ``` If there was an error the output provides a URL to see the error messages. Open this in a browser to find out what happened. Expand the detail to see a sample error message: Reading error messages in the browser ```bash Error: Target column count: 29 doesn't match source value column count: 32. Column separator: ',', Row delimiter: '\n'. Row: 09/06/2015,14:15,,,40.6722269,-74.0110059,"(40.6722269, -74.0110059)",,,"R/O 1 BEARD ST. ( IKEA'S 09/14/2015,5:30,BRONX,10473,40.814551,-73.8490955,"(40.814551, -73.8490955)",TORRY AVENUE ,NORTON AVENUE ,,0,0,0,0,0,0,0,0,Driver Inattention/Distraction,Unspecified,,,,3297457,PASSENGER VEHICLE,PASSENGER VEHICLE,,, ``` ##### Weather data[​](#weather-data-1 "Direct link to Weather data") Load the weather dataset in the same manner as you loaded the crash data. ```bash curl --location-trusted -u root \ -T ./72505394728.csv \ -H "label:weather-0" \ -H "column_separator:," \ -H "skip_header:1" \ -H "enclose:\"" \ -H "max_filter_ratio:1" \ -H "columns: STATION, DATE, LATITUDE, LONGITUDE, ELEVATION, NAME, REPORT_TYPE, SOURCE, HourlyAltimeterSetting, HourlyDewPointTemperature, HourlyDryBulbTemperature, HourlyPrecipitation, HourlyPresentWeatherType, HourlyPressureChange, HourlyPressureTendency, HourlyRelativeHumidity, HourlySkyConditions, HourlySeaLevelPressure, HourlyStationPressure, HourlyVisibility, HourlyWetBulbTemperature, HourlyWindDirection, HourlyWindGustSpeed, HourlyWindSpeed, Sunrise, Sunset, DailyAverageDewPointTemperature, DailyAverageDryBulbTemperature, DailyAverageRelativeHumidity, DailyAverageSeaLevelPressure, DailyAverageStationPressure, DailyAverageWetBulbTemperature, DailyAverageWindSpeed, DailyCoolingDegreeDays, DailyDepartureFromNormalAverageTemperature, DailyHeatingDegreeDays, DailyMaximumDryBulbTemperature, DailyMinimumDryBulbTemperature, DailyPeakWindDirection, DailyPeakWindSpeed, DailyPrecipitation, DailySnowDepth, DailySnowfall, DailySustainedWindDirection, DailySustainedWindSpeed, DailyWeather, MonthlyAverageRH, MonthlyDaysWithGT001Precip, MonthlyDaysWithGT010Precip, MonthlyDaysWithGT32Temp, MonthlyDaysWithGT90Temp, MonthlyDaysWithLT0Temp, MonthlyDaysWithLT32Temp, MonthlyDepartureFromNormalAverageTemperature, MonthlyDepartureFromNormalCoolingDegreeDays, MonthlyDepartureFromNormalHeatingDegreeDays, MonthlyDepartureFromNormalMaximumTemperature, MonthlyDepartureFromNormalMinimumTemperature, MonthlyDepartureFromNormalPrecipitation, MonthlyDewpointTemperature, MonthlyGreatestPrecip, MonthlyGreatestPrecipDate, MonthlyGreatestSnowDepth, MonthlyGreatestSnowDepthDate, MonthlyGreatestSnowfall, MonthlyGreatestSnowfallDate, MonthlyMaxSeaLevelPressureValue, MonthlyMaxSeaLevelPressureValueDate, MonthlyMaxSeaLevelPressureValueTime, MonthlyMaximumTemperature, MonthlyMeanTemperature, MonthlyMinSeaLevelPressureValue, MonthlyMinSeaLevelPressureValueDate, MonthlyMinSeaLevelPressureValueTime, MonthlyMinimumTemperature, MonthlySeaLevelPressure, MonthlyStationPressure, MonthlyTotalLiquidPrecipitation, MonthlyTotalSnowfall, MonthlyWetBulb, AWND, CDSD, CLDD, DSNW, HDSD, HTDD, NormalsCoolingDegreeDay, NormalsHeatingDegreeDay, ShortDurationEndDate005, ShortDurationEndDate010, ShortDurationEndDate015, ShortDurationEndDate020, ShortDurationEndDate030, ShortDurationEndDate045, ShortDurationEndDate060, ShortDurationEndDate080, ShortDurationEndDate100, ShortDurationEndDate120, ShortDurationEndDate150, ShortDurationEndDate180, ShortDurationPrecipitationValue005, ShortDurationPrecipitationValue010, ShortDurationPrecipitationValue015, ShortDurationPrecipitationValue020, ShortDurationPrecipitationValue030, ShortDurationPrecipitationValue045, ShortDurationPrecipitationValue060, ShortDurationPrecipitationValue080, ShortDurationPrecipitationValue100, ShortDurationPrecipitationValue120, ShortDurationPrecipitationValue150, ShortDurationPrecipitationValue180, REM, BackupDirection, BackupDistance, BackupDistanceUnit, BackupElements, BackupElevation, BackupEquipment, BackupLatitude, BackupLongitude, BackupName, WindEquipmentChangeDate" \ -XPUT http://localhost:8030/api/quickstart/weatherdata/_stream_load ``` *** #### Answer some questions[​](#answer-some-questions "Direct link to Answer some questions") These queries can be run in your SQL client. All of the queries use the `quickstart` database. ```sql USE quickstart; ``` ###### How many crashes are there per hour in NYC?[​](#how-many-crashes-are-there-per-hour-in-nyc "Direct link to How many crashes are there per hour in NYC?") ```sql SELECT COUNT(*), date_trunc("hour", crashdata.CRASH_DATE) AS Time FROM crashdata GROUP BY Time ORDER BY Time ASC LIMIT 200; ``` Here is part of the output. Note that I am looking closer at January 6th and 7th as this is Monday and Tuesday of a non-holiday week. Looking at New Years Day is probably not indicative of a normal morning during rush-hour traffic. ```plaintext | 14 | 2014-01-06 06:00:00 | | 16 | 2014-01-06 07:00:00 | | 43 | 2014-01-06 08:00:00 | | 44 | 2014-01-06 09:00:00 | | 21 | 2014-01-06 10:00:00 | | 28 | 2014-01-06 11:00:00 | | 34 | 2014-01-06 12:00:00 | | 31 | 2014-01-06 13:00:00 | | 35 | 2014-01-06 14:00:00 | | 36 | 2014-01-06 15:00:00 | | 33 | 2014-01-06 16:00:00 | | 40 | 2014-01-06 17:00:00 | | 35 | 2014-01-06 18:00:00 | | 23 | 2014-01-06 19:00:00 | | 16 | 2014-01-06 20:00:00 | | 12 | 2014-01-06 21:00:00 | | 17 | 2014-01-06 22:00:00 | | 14 | 2014-01-06 23:00:00 | | 10 | 2014-01-07 00:00:00 | | 4 | 2014-01-07 01:00:00 | | 1 | 2014-01-07 02:00:00 | | 3 | 2014-01-07 03:00:00 | | 2 | 2014-01-07 04:00:00 | | 6 | 2014-01-07 06:00:00 | | 16 | 2014-01-07 07:00:00 | | 41 | 2014-01-07 08:00:00 | | 37 | 2014-01-07 09:00:00 | | 33 | 2014-01-07 10:00:00 | ``` It looks like about 40 accidents on a Monday or Tuesday morning during rush hour traffic, and around the same at 17:00 hours. ###### What is the average temperature in NYC?[​](#what-is-the-average-temperature-in-nyc "Direct link to What is the average temperature in NYC?") ```sql SELECT avg(HourlyDryBulbTemperature), date_trunc("hour", weatherdata.DATE) AS Time FROM weatherdata GROUP BY Time ORDER BY Time ASC LIMIT 100; ``` Output: Note that this is data from 2014, NYC has not been this cold lately. ```plaintext +-------------------------------+---------------------+ | avg(HourlyDryBulbTemperature) | Time | +-------------------------------+---------------------+ | 25 | 2014-01-01 00:00:00 | | 25 | 2014-01-01 01:00:00 | | 24 | 2014-01-01 02:00:00 | | 24 | 2014-01-01 03:00:00 | | 24 | 2014-01-01 04:00:00 | | 24 | 2014-01-01 05:00:00 | | 25 | 2014-01-01 06:00:00 | | 26 | 2014-01-01 07:00:00 | ``` ###### Is it safe to drive in NYC when visibility is poor?[​](#is-it-safe-to-drive-in-nyc-when-visibility-is-poor "Direct link to Is it safe to drive in NYC when visibility is poor?") Let's look at the number of crashes when visibility is poor (between 0 and 1.0 miles). To answer this question use a JOIN across the two tables on the DATETIME column. ```sql SELECT COUNT(DISTINCT c.COLLISION_ID) AS Crashes, truncate(avg(w.HourlyDryBulbTemperature), 1) AS Temp_F, truncate(avg(w.HourlyVisibility), 2) AS Visibility, max(w.HourlyPrecipitation) AS Precipitation, date_format((date_trunc("hour", c.CRASH_DATE)), '%d %b %Y %H:%i') AS Hour FROM crashdata c LEFT JOIN weatherdata w ON date_trunc("hour", c.CRASH_DATE)=date_trunc("hour", w.DATE) WHERE w.HourlyVisibility BETWEEN 0.0 AND 1.0 GROUP BY Hour ORDER BY Crashes DESC LIMIT 100; ``` The highest number of crashes in a single hour during low visibility is 129. There are multiple things to consider: * February 3rd 2014 was a Monday * 8AM is rush hour * It was raining (0.12 inches or precipitation that hour) * The temperature is 32 degrees F (the freezing point for water) * Visibility is bad at 0.25 miles, normal for NYC is 10 miles ```plaintext +---------+--------+------------+---------------+-------------------+ | Crashes | Temp_F | Visibility | Precipitation | Hour | +---------+--------+------------+---------------+-------------------+ | 129 | 32 | 0.25 | 0.12 | 03 Feb 2014 08:00 | | 114 | 32 | 0.25 | 0.12 | 03 Feb 2014 09:00 | | 104 | 23 | 0.33 | 0.03 | 09 Jan 2015 08:00 | | 96 | 26.3 | 0.33 | 0.07 | 01 Mar 2015 14:00 | | 95 | 26 | 0.37 | 0.12 | 01 Mar 2015 15:00 | | 93 | 35 | 0.75 | 0.09 | 18 Jan 2015 09:00 | | 92 | 31 | 0.25 | 0.12 | 03 Feb 2014 10:00 | | 87 | 26.8 | 0.5 | 0.09 | 01 Mar 2015 16:00 | | 85 | 55 | 0.75 | 0.20 | 23 Dec 2015 17:00 | | 85 | 20 | 0.62 | 0.01 | 06 Jan 2015 11:00 | | 83 | 19.6 | 0.41 | 0.04 | 05 Mar 2015 13:00 | | 80 | 20 | 0.37 | 0.02 | 06 Jan 2015 10:00 | | 76 | 26.5 | 0.25 | 0.06 | 05 Mar 2015 09:00 | | 71 | 26 | 0.25 | 0.09 | 05 Mar 2015 10:00 | | 71 | 24.2 | 0.25 | 0.04 | 05 Mar 2015 11:00 | ``` ###### What about driving in icy conditions?[​](#what-about-driving-in-icy-conditions "Direct link to What about driving in icy conditions?") Water vapor can desublimate to ice at 40 degrees F; this query looks at temps between 0 and 40 degrees F. ```sql SELECT COUNT(DISTINCT c.COLLISION_ID) AS Crashes, truncate(avg(w.HourlyDryBulbTemperature), 1) AS Temp_F, truncate(avg(w.HourlyVisibility), 2) AS Visibility, max(w.HourlyPrecipitation) AS Precipitation, date_format((date_trunc("hour", c.CRASH_DATE)), '%d %b %Y %H:%i') AS Hour FROM crashdata c LEFT JOIN weatherdata w ON date_trunc("hour", c.CRASH_DATE)=date_trunc("hour", w.DATE) WHERE w.HourlyDryBulbTemperature BETWEEN 0.0 AND 40.5 GROUP BY Hour ORDER BY Crashes DESC LIMIT 100; ``` The results for freezing temperatures suprised me a little, I did not expect too much traffic on a Sunday morning in the city on a cold January day.A quick look at [weather.com](https://weather.com/storms/winter/news/northeast-storm-rain-snow-wind) showed that there was a big storm with many crashes that day, just like what can be seen in the data. ```plaintext +---------+--------+------------+---------------+-------------------+ | Crashes | Temp_F | Visibility | Precipitation | Hour | +---------+--------+------------+---------------+-------------------+ | 192 | 34 | 1.5 | 0.09 | 18 Jan 2015 08:00 | | 170 | 21 | NULL | | 21 Jan 2014 10:00 | | 145 | 19 | NULL | | 21 Jan 2014 11:00 | | 138 | 33.5 | 5 | 0.02 | 18 Jan 2015 07:00 | | 137 | 21 | NULL | | 21 Jan 2014 09:00 | | 129 | 32 | 0.25 | 0.12 | 03 Feb 2014 08:00 | | 114 | 32 | 0.25 | 0.12 | 03 Feb 2014 09:00 | | 104 | 23 | 0.7 | 0.04 | 09 Jan 2015 08:00 | | 98 | 16 | 8 | 0.00 | 06 Mar 2015 08:00 | | 96 | 26.3 | 0.33 | 0.07 | 01 Mar 2015 14:00 | ``` Drive carefully! *** #### Summary[​](#summary "Direct link to Summary") In this tutorial you: * Deployed StarRocks in Docker * Loaded crash data provided by New York City and weather data provided by NOAA * Analyzed the data using SQL JOINs to find out that driving in low visibility or icy streets is a bad idea There is more to learn; we intentionally glossed over the data transformation done during the Stream Load. The details on that are in the notes on the curl commands below. *** #### Notes on the curl commands[​](#notes-on-the-curl-commands "Direct link to Notes on the curl commands") StarRocks Stream Load and curl take many arguments. Only the ones used in this tutorial are described here, the rest will be linked to in the more information section. ###### `--location-trusted`[​](#--location-trusted "Direct link to --location-trusted") This configures curl to pass credentials to any redirected URLs. ###### `-u root`[​](#-u-root "Direct link to -u-root") The username used to log in to StarRocks ###### `-T filename`[​](#-t-filename "Direct link to -t-filename") T is for transfer, the filename to transfer. ###### `label:name-num`[​](#labelname-num "Direct link to labelname-num") The label to associate with this Stream Load job. The label must be unique, so if you run the job multiple times you can add a number and keep incrementing that. ###### `column_separator:,`[​](#column_separator "Direct link to column_separator") If you load a file that uses a single `,` then set it as shown above, if you use a different delimiter then set that delimiter here. Common choices are `\t`, `,`, and `|`. ###### `skip_header:1`[​](#skip_header1 "Direct link to skip_header1") Some CSV files have a single header row with all of the column names listed, and some add a second line with datatypes. Set skip\_header to `1` or `2` if you have one or two header lines, and set it to `0` if you have none. ###### `enclose:\"`[​](#enclose "Direct link to enclose") It is common to enclose strings that contain embedded commas with double-quotes. The sample datasets used in this tutorial have geo locations that contain commas and so the enclose setting is set to `\"`. Remember to escape the `"` with a `\`. ###### `max_filter_ratio:1`[​](#max_filter_ratio1 "Direct link to max_filter_ratio1") This allows some errors in the data. Ideally this would be set to `0` and the job would fail with any errors. It is set to `1` to allow all rows to fail during debugging. ###### `columns:`[​](#columns "Direct link to columns") The mapping of CSV file columns to StarRocks table columns. You will notice that there are many more columns in the CSV files than columns in the table. Any columns that are not included in the table are skipped. You will also notice that there is some transformation of data included in the `columns:` line for the crash dataset. It is very common to find dates and times in CSV files that do not conform to standards. This is the logic for converting the CSV data for the time and date of the crash to a DATETIME type: ###### The columns line[​](#the-columns-line "Direct link to The columns line") This is the beginning of one data record. The date is in `MM/DD/YYYY` format, and the time is `HH:MI`. Since DATETIME is generally `YYYY-MM-DD HH:MI:SS` we need to transform this data. ```plaintext 08/05/2014,9:10,BRONX,10469,40.8733019,-73.8536375,"(40.8733019, -73.8536375)", ``` This is the beginning of the `columns:` parameter: ```bash -H "columns:tmp_CRASH_DATE, tmp_CRASH_TIME, CRASH_DATE=str_to_date(concat_ws(' ', tmp_CRASH_DATE, tmp_CRASH_TIME), '%m/%d/%Y %H:%i') ``` This instructs StarRocks to: * Assign the content of the first column of the CSV file to `tmp_CRASH_DATE` * Assign the content of the second column of the CSV file to `tmp_CRASH_TIME` * `concat_ws()` concatenates `tmp_CRASH_DATE` and `tmp_CRASH_TIME` together with a space between them * `str_to_date()` creates a DATETIME from the concatenated string * store the resulting DATETIME in the column `CRASH_DATE` *** #### More information[​](#more-information "Direct link to More information") [StarRocks table design](https://docs.starrocks.io/docs/table_design/StarRocks_table_design.md) [Stream Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md) The [Motor Vehicle Collisions - Crashes](https://data.cityofnewyork.us/Public-Safety/Motor-Vehicle-Collisions-Crashes/h9gi-nx95) dataset is provided by New York City subject to these [terms of use](https://www.nyc.gov/home/terms-of-use.page) and [privacy policy](https://www.nyc.gov/home/privacy-policy.page). The [Local Climatological Data](https://www.ncdc.noaa.gov/cdo-web/datatools/lcd)(LCD) is provided by NOAA with this [disclaimer](https://www.noaa.gov/disclaimer) and this [privacy policy](https://www.noaa.gov/protecting-your-privacy). --- ## Release 2.5 ### StarRocks version 2.5 #### 2.5.22[​](#2522 "Direct link to 2.5.22") Release date: June 20, 2024 ##### Improvements[​](#improvements "Direct link to Improvements") * Optimized a partition check logic used for building query execution plan, significantly reducing the time consumption of complex queries that involve multiple tables. [#46781](https://github.com/StarRocks/starrocks/pull/46781) ##### Bug Fixes[​](#bug-fixes "Direct link to Bug Fixes") Fixed the following issues: * Function Call does not handle child errors correctly. [#42590](https://github.com/StarRocks/starrocks/pull/42590) * The internal data statistics were not cleaned up regularly, causing inaccurate estimated information and thereby inefficient query plans. This will cause a drop in query performance and a surge in memory usage. [#45839](https://github.com/StarRocks/starrocks/pull/45839) * Using a stale column histogram may lead to the Division by Zero exception. [#45614](https://github.com/StarRocks/starrocks/pull/45614) #### 2.5.21[​](#2521 "Direct link to 2.5.21") Release date: May 15, 2024 ##### Improvements[​](#improvements-1 "Direct link to Improvements") * Optimized the usage of database locks for materialized view refresh to prevent deadlock. [#42801](https://github.com/StarRocks/starrocks/pull/42801) * Both `s3a://` and `s3://` can be used to access data in AWS S3. [#42460](https://github.com/StarRocks/starrocks/pull/42460) ##### Bug Fixes[​](#bug-fixes-1 "Direct link to Bug Fixes") Fixed the following issues: * Schema change may cause issues in prefix index sorting, leading to incorrect results for queries based on prefix indexes. [#44941](https://github.com/StarRocks/starrocks/pull/44941) * After a Routine Load task is paused due to Kafka cluster abnormalities, the background still attempts to connect to this abnormal Kafka cluster, which prevents other Routine Load tasks in this StarRocks cluster from consuming normal Kafka messages. [#45029](https://github.com/StarRocks/starrocks/pull/45029) * When querying views in `information_schema`, the database lock is held for an unexpectedly long time, which prolongs the overall query time. [#45392](https://github.com/StarRocks/starrocks/pull/45392) * Enabling Query Cache may cause BEs to crash if the SQL query contains a HAVING clause. This issue can be resolved by disabling Query Cache using `set enable_query_cache=false`. [#43823](https://github.com/StarRocks/starrocks/pull/43823) * When Query Cache is enabled, some queries may return an error message `All slotIds should be remapped`. [#42861](https://github.com/StarRocks/starrocks/pull/42861) #### 2.5.20[​](#2520 "Direct link to 2.5.20") Release date: March 22, 2024 ##### Improvements[​](#improvements-2 "Direct link to Improvements") * `replace_if_not_null` supports BITMAP columns in an Aggregate table. Users can specify `replace_if_not_null` as the aggregate function for BITMAP columns in an Aggregate table. [#42104](https://github.com/StarRocks/starrocks/pull/42104) * G1 Garbage Collector is used for JDK 9 and later by default. [#41374](https://github.com/StarRocks/starrocks/pull/41374) ##### Parameter Changes[​](#parameter-changes "Direct link to Parameter Changes") * The default value of the BE parameter `update_compaction_size_threshold` is changed from 256 MB to 64 MB to accelerate compaction. [#42776](https://github.com/StarRocks/starrocks/pull/42776) ##### Bug Fixes[​](#bug-fixes-2 "Direct link to Bug Fixes") Fixed the following issues: * Synchronizing data using StarRocks external tables encounters the error "commit and publish txn failed". The synchronization succeeds after a retry but the same copy of data is loaded twice. [#25165](https://github.com/StarRocks/starrocks/pull/25165) * RPC transmit resources are temporarily unavailable due to GC issues. [#41636](https://github.com/StarRocks/starrocks/pull/41636) * array\_agg() in v2.5 processes NULLs in a different way than it does in v2.3. As a result, the query result is incorrect after an upgrade from v2.3 to v2.5. [#42639](https://github.com/StarRocks/starrocks/pull/42639) * The Sink Operator in a query unexpectedly exits, which causes BEs to crash. [#38662](https://github.com/StarRocks/starrocks/pull/38662) * Executing the DELETE command on an Aggregate table results in a race for accessing tablet metadata, which causes BEs to crash. [#42174](https://github.com/StarRocks/starrocks/pull/42174) * The MemTracker encounters the Use-After-Free issue during UDF calling, which causes BEs to crash. [#41710](https://github.com/StarRocks/starrocks/pull/41710) * The unnest() function does not support aliases. [#42138](https://github.com/StarRocks/starrocks/pull/42138) #### 2.5.19[​](#2519 "Direct link to 2.5.19") Release date: February 8, 2024 ##### New features[​](#new-features "Direct link to New features") * Added Bitmap value processing functions: serialize, deserialize, and serializeToString. [#40162](https://github.com/StarRocks/starrocks/pull/40162/files) ##### Improvements[​](#improvements-3 "Direct link to Improvements") * Supports automatic activation of inactive materialized views when refreshing these materialized views. [#38521](https://github.com/StarRocks/starrocks/pull/38521) * Optimized BE log printing to prevent too many irrelevant logs. [#22820](https://github.com/StarRocks/starrocks/pull/22820) [#36187](https://github.com/StarRocks/starrocks/pull/36187) * Supports using [Hive UDFs](https://docs.starrocks.io/docs/integrations/hive_bitmap_udf/) to process and load Bitmap data into StarRocks and export Bitmap data from StarRocks to Hive. [#40165](https://github.com/StarRocks/starrocks/pull/40165) [#40168](https://github.com/StarRocks/starrocks/pull/40168) * Added date formats `yyyy-MM-ddTHH:mm` and `yyyy-MM-dd HH:mm` to support TIMESTAMP partition fields in Apache Iceberg tables. [#39986](https://github.com/StarRocks/starrocks/pull/39986) ##### Bug Fixes[​](#bug-fixes-3 "Direct link to Bug Fixes") Fixed the following issues: * Running a Spark Load job that has no PROPERTIES specified causes null pointer exceptions (NPEs). [#38765](https://github.com/StarRocks/starrocks/pull/38765) * INSERT INTO SELECT occasionally encounters the error "timeout by txn manager". [#36688](https://github.com/StarRocks/starrocks/pull/36688) * The memory consumption of PageCache exceeds the threshold specified by the BE dynamic parameter `storage_page_cache_limit` in certain circumstances. [#37740](https://github.com/StarRocks/starrocks/pull/37740) * After a table is dropped and then re-created with the same table name, refreshing asynchronous materialized views created on that table fails. [#38008](https://github.com/StarRocks/starrocks/pull/38008) [#38982](https://github.com/StarRocks/starrocks/pull/38982) * Writing data to S3 buckets using SELECT INTO occasionally encounters the error "The tablet write operation update metadata take a long time". [#38443](https://github.com/StarRocks/starrocks/pull/38443) * Some operations during data loading may encounter "reached timeout". [#36746](https://github.com/StarRocks/starrocks/pull/36746) * The DECIMAL type returned by SHOW CREATE TABLE is inconsistent with that specified in CREATE TABLE. [#39297](https://github.com/StarRocks/starrocks/pull/39297) * If partition columns in external tables contain null values, queries against those tables will cause BEs to crash. [#38888](https://github.com/StarRocks/starrocks/pull/38888) * When deleting data from a Duplicate Key table, if the condition in the WHERE clause of the DELETE statement has a leading space, the deleted data can still be queried using SELECT. [#39797](https://github.com/StarRocks/starrocks/pull/39797) * Loading `array` data from ORC files into StarRocks (`array`) may cause BEs to crash. [#39233](https://github.com/StarRocks/starrocks/pull/39233) * Querying Hive catalogs may be stuck and even expire. [#39863](https://github.com/StarRocks/starrocks/pull/39863) * Partitions cannot be dynamically created if hour-level partitions are specified in the PARTITION BY clause. [#40256](https://github.com/StarRocks/starrocks/pull/40256) * The error message "failed to call frontend service" is returned during loading from Apache Flink. [#40710](https://github.com/StarRocks/starrocks/pull/40710) #### 2.5.18[​](#2518 "Direct link to 2.5.18") Release date: Jan 10, 2024 ##### New Features[​](#new-features-1 "Direct link to New Features") * Users can set or modify session variables when they [CREATE](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-definition/CREATE_MATERIALIZED_VIEW/#parameters) or [ALTER](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-definition/ALTER_MATERIALIZED_VIEW/) asynchronous materialized views. [#37401](https://github.com/StarRocks/starrocks/pull/37401) ##### Improvements[​](#improvements-4 "Direct link to Improvements") * When using JDK, the default GC algorithm is G1. [#37498](https://github.com/StarRocks/starrocks/pull/37498) * The result returned by the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/SHOW_ROUTINE_LOAD/) statement now includes the timestamps of consumption messages from each partition. [#36222](https://github.com/StarRocks/starrocks/pull/36222) ##### Behavior Change[​](#behavior-change "Direct link to Behavior Change") * Added the session variable `enable_materialized_view_for_insert`, which controls whether materialized views rewrite the queries in INSERT INTO SELECT statements. The default value is `false`. [#37505](https://github.com/StarRocks/starrocks/pull/37505) * Added the session variable `enable_strict_order_by`. When this variable is set to the default value `TRUE`, an error is reported for such a query pattern: Duplicate alias is used in different expressions of the query and this alias is also a sorting field in ORDER BY, for example, `select distinct t1.* from tbl1 t1 order by t1.k1;`. The logic is the same as that in v2.3 and earlier. When this variable is set to `FALSE`, a loose deduplication mechanism is used, which processes such queries as valid SQL queries. [#37910](https://github.com/StarRocks/starrocks/pull/37910) ##### Parameter Change[​](#parameter-change "Direct link to Parameter Change") * Added session variables `transaction_read_only` and `tx_read_only` to specify the transaction access mode, which are compatible with MySQL versions 5.7.20 and above. [#37249](https://github.com/StarRocks/starrocks/pull/37249) * Added the FE configuration item `routine_load_unstable_threshold_second`. [#36222](https://github.com/StarRocks/starrocks/pull/36222) * Added the FE configuration item `http_worker_threads_num`, which specifies the number of threads for HTTP server to deal with HTTP requests. The default value is `0`. If the value for this parameter is set to a negative value or 0, the actual thread number is twice the number of CPU cores. [#37530](https://github.com/StarRocks/starrocks/pull/37530) * Added the BE configuration item `pindex_major_compaction_limit_per_disk` to configure the maximum concurrency of compaction on a disk. This addresses the issue of uneven I/O across disks due to compaction. This issue can cause excessively high I/O for certain disks. The default value is `1`. [#37695](https://github.com/StarRocks/starrocks/pull/37695) ##### Bug Fixes[​](#bug-fixes-4 "Direct link to Bug Fixes") Fixed the following issues: * Using NaN (Not a Number) columns as ORDER BY columns may cause BEs to crash. [#30759](https://github.com/StarRocks/starrocks/pull/30759) * Failure to update primary key indexes may cause the error "get\_applied\_rowsets failed". [#27488](https://github.com/StarRocks/starrocks/pull/27488) * Hive metadata in [Hive catalogs](https://docs.starrocks.io/docs/2.5/data_source/catalog/hive_catalog/) is not automatically refreshed when new fields are added to Hive tables. [#37668](https://github.com/StarRocks/starrocks/pull/37668) * When `SELECT ... FROM ... INTO OUTFILE` is executed to export data into CSV files, the error "Unmatched number of columns" is reported if the FROM clause contains multiple constants. [#38045](https://github.com/StarRocks/starrocks/pull/38045) * In some cases, `bitmap_to_string` may return incorrect result due to data type overflow. [#37405](https://github.com/StarRocks/starrocks/pull/37405) #### 2.5.17[​](#2517 "Direct link to 2.5.17") Release date: December 19, 2023 ##### New Features[​](#new-features-2 "Direct link to New Features") * Added a new metric `max_tablet_rowset_num` for setting the maximum allowed number of rowsets. This metric helps detect possible compaction issues and thus reduces the occurrences of the error "too many versions". [#36539](https://github.com/StarRocks/starrocks/pull/36539) * Added the [subdivide\_bitmap](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/subdivide_bitmap/) function. [#35817](https://github.com/StarRocks/starrocks/pull/35817) ##### Improvements[​](#improvements-5 "Direct link to Improvements") * The result returned by the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/SHOW_ROUTINE_LOAD/) statement provides a new field `OtherMsg`, which shows information about the last failed task. [#35806](https://github.com/StarRocks/starrocks/pull/35806) * The default retention period of trash files is changed to 1 day from the original 3 days. [#37113](https://github.com/StarRocks/starrocks/pull/37113) * Optimized the performance of persistent index update when compaction is performed on all rowsets of a Primary Key table, which reduces disk read I/O. [#36819](https://github.com/StarRocks/starrocks/pull/36819) * Optimized the logic used to compute compaction scores for Primary Key tables, thereby aligning the compaction scores for Primary Key tables within a more consistent range with the other three table types. [#36534](https://github.com/StarRocks/starrocks/pull/36534) * Queries on MySQL external tables and the external tables within JDBC catalogs support including keywords in the WHERE clause. [#35917](https://github.com/StarRocks/starrocks/pull/35917) * Added the bitmap\_from\_binary function to Spark Load to support loading Binary data. [#36050](https://github.com/StarRocks/starrocks/pull/36050) * The bRPC expiration time is shortened from 1 hour to the duration specified by the session variable [`query_timeout`](https://docs.starrocks.io/zh/docs/3.2/reference/System_variable/#query_timeout). This prevents query failures caused by RPC request expiration. [#36778](https://github.com/StarRocks/starrocks/pull/36778) ##### Parameter Change[​](#parameter-change-1 "Direct link to Parameter Change") * Added a BE configuration item `enable_stream_load_verbose_log` is added. The default value is `false`. With this parameter set to `true`, StarRocks can record the HTTP requests and responses for Stream Load jobs, making troubleshooting easier. [#36113](https://github.com/StarRocks/starrocks/pull/36113) * Added a BE static parameter `update_compaction_per_tablet_min_interval_seconds` becomes mutable. [#36819](https://github.com/StarRocks/starrocks/pull/36819) ##### Bug Fixes[​](#bug-fixes-5 "Direct link to Bug Fixes") Fixed the following issues: * Queries fail during hash joins, causing BEs to crash. [#32219](https://github.com/StarRocks/starrocks/pull/32219) * The FE performance plunges after the FE configuration item `enable_collect_query_detail_info` is set to `true`. [#35945](https://github.com/StarRocks/starrocks/pull/35945) * Errors may be thrown if large amounts of data are loaded into a Primary Key table with persistent index enabled. [#34352](https://github.com/StarRocks/starrocks/pull/34352) * The starrocks\_be process may exit unexpectedly when `./agentctl.sh stop be` is used to stop a BE. [#35108](https://github.com/StarRocks/starrocks/pull/35108) * The [array\_distinct](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_distinct/) function occasionally causes the BEs to crash. [#36377](https://github.com/StarRocks/starrocks/pull/36377) * Deadlocks may occur when users refresh materialized views. [#35736](https://github.com/StarRocks/starrocks/pull/35736) * In some scenarios, dynamic partitioning may encounter an error, which causes FE start failures. [#36846](https://github.com/StarRocks/starrocks/pull/36846) #### 2.5.16[​](#2516 "Direct link to 2.5.16") Release date: December 1, 2023 ##### Bug Fixes[​](#bug-fixes-6 "Direct link to Bug Fixes") Fixed the following issues: * Global Runtime Filter may cause BEs to crash in certain scenarios. [#35776](https://github.com/StarRocks/starrocks/pull/35776) #### 2.5.15[​](#2515 "Direct link to 2.5.15") Release date: November 29, 2023 ##### Improvements[​](#improvements-6 "Direct link to Improvements") * Added slow request logs to track slow requests. [#33908](https://github.com/StarRocks/starrocks/pull/33908) * Optimized the performance of using Spark Load to read Parquet and ORC files when there are a large number of files. [#34787](https://github.com/StarRocks/starrocks/pull/34787) * Optimized the performance of some Bitmap-related operations, including: * Optimized nested loop joins. [#340804](https://github.com/StarRocks/starrocks/pull/34804) [#35003](https://github.com/StarRocks/starrocks/pull/35003) * Optimized the `bitmap_xor` function. [#34069](https://github.com/StarRocks/starrocks/pull/34069) * Supports Copy on Write to optimize Bitmap performance and reduce memory consumption. [#34047](https://github.com/StarRocks/starrocks/pull/34047) ##### Parameter Change[​](#parameter-change-2 "Direct link to Parameter Change") * The FE dynamic parameter `enable_new_publish_mechanism` is changed to a static parameter. You must restart the FE after you modify the parameter settings. [#35338](https://github.com/StarRocks/starrocks/pull/35338) ##### Bug Fixes[​](#bug-fixes-7 "Direct link to Bug Fixes") * If a filtering condition is specified in a Broker Load job, BEs may crash during the data loading in certain circumstances. [#29832](https://github.com/StarRocks/starrocks/pull/29832) * Failures in replaying replica operations may cause FEs to crash. [#32295](https://github.com/StarRocks/starrocks/pull/32295) * Setting the FE parameter `recover_with_empty_tablet` to `true` may cause FEs to crash. [#33071](https://github.com/StarRocks/starrocks/pull/33071) * The error "get\_applied\_rowsets failed, tablet updates is in error state: tablet:18849 actual row size changed after compaction" is returned for queries. [#33246](https://github.com/StarRocks/starrocks/pull/33246) * A query that contains a window function may cause BEs to crash. [#33671](https://github.com/StarRocks/starrocks/pull/33671) * Running `show proc '/statistic'` may cause a deadlock. [#34237](https://github.com/StarRocks/starrocks/pull/34237/files) * Errors may be thrown if large amounts of data are loaded into a Primary Key table with persistent index enabled. [#34566](https://github.com/StarRocks/starrocks/pull/34566) * After StarRocks is upgraded from v2.4 or earlier to a later version, compaction scores may rise unexpectedly. [#34618](https://github.com/StarRocks/starrocks/pull/34618) * If `INFORMATION_SCHEMA` is queried by using the database driver MariaDB ODBC, the `CATALOG_NAME` column returned in the `schemata` view holds only `null` values. [#34627](https://github.com/StarRocks/starrocks/pull/34627) * If schema changes are being executed while a Stream Load job is in the **PREPARED** state, a portion of the source data to be loaded by the job is lost. [#34381](https://github.com/StarRocks/starrocks/pull/34381) * Including two or more slashes (`/`) at the end of the HDFS storage path causes the backup and restore of the data from HDFS to fail. [#34601](https://github.com/StarRocks/starrocks/pull/34601) * Running a loading task or a query may cause the FEs to hang. [#34569](https://github.com/StarRocks/starrocks/pull/34569) #### 2.5.14[​](#2514 "Direct link to 2.5.14") Release date: November 14, 2023 ##### Improvements[​](#improvements-7 "Direct link to Improvements") * The `COLUMNS` table in the system database `INFORMATION_SCHEMA` can display ARRAY, MAP, and STRUCT columns. [#33431](https://github.com/StarRocks/starrocks/pull/33431) ##### Parameter change[​](#parameter-change-3 "Direct link to Parameter change") ###### System variables[​](#system-variables "Direct link to System variables") * Added a session variable `cbo_decimal_cast_string_strict`, which controls how the CBO converts data from the DECIMAL type to the STRING type. If this variable is set to `true`, the logic built in v2.5.x and later versions prevails and the system implements strict conversion (namely, the system truncates the generated string and fills 0s based on the scale length). If this variable is set to `false`, the logic built in versions earlier than v2.5.x prevails and the system processes all valid digits to generate a string. The default value is `true`. [#34208](https://github.com/StarRocks/starrocks/pull/34208) * Added a session variable `cbo_eq_base_type`, which specifies the data type used for data comparison between DECIMAL-type data and STRING-type data. The default value is `VARCHAR`, and DECIMAL is also a valid value. [#34208](https://github.com/StarRocks/starrocks/pull/34208) ##### Bug Fixes[​](#bug-fixes-8 "Direct link to Bug Fixes") Fixed the following issues: * The error `java.lang.IllegalStateException: null` is reported if the ON condition is nested with a subquery. [#30876](https://github.com/StarRocks/starrocks/pull/30876) * The result of COUNT(*) is inconsistent among replicas if COUNT(*) is run immediately after `INSERT INTO SELECT ... LIMIT` is successfully executed. [#24435](https://github.com/StarRocks/starrocks/pull/24435) * BE may crash for specific data types if the target data type specified in the cast() function is the same as the original data type. [#31465](https://github.com/StarRocks/starrocks/pull/31465) * An error is reported if specific path formats are used during data loading via Broker Load: `msg:Fail to parse columnsFromPath, expected: [rec_dt]`. [#32721](https://github.com/StarRocks/starrocks/issues/32721) * During an upgrade to 3.x, if some column types are also upgraded (for example, Decimal is upgraded to Decimal v3), BEs crash when Compaction is performed on tables with specific characteristics. [#31626](https://github.com/StarRocks/starrocks/pull/31626) * When data is loaded by using Flink Connector, the load job is suspended unexpectedly if there are highly concurrent load jobs and both the number of HTTP and Scan threads have reached their upper limits. [#32251](https://github.com/StarRocks/starrocks/pull/32251) * BEs crash when libcurl is invoked. [#31667](https://github.com/StarRocks/starrocks/pull/31667) * Adding BITMAP columns to a Primary Key table fails with the following error: `Analyze columnDef error: No aggregate function specified for 'userid'`. [#31763](https://github.com/StarRocks/starrocks/pull/31763) * Long-time, frequent data loading into a Primary Key table with persistent index enabled may cause BEs to crash. [#33220](https://github.com/StarRocks/starrocks/pull/33220) * The query result is incorrect when Query Cache is enabled. [#32778](https://github.com/StarRocks/starrocks/pull/32778) * Specifying a nullable Sort Key when creating a Primary Key table causes compaction to fail. [#29225](https://github.com/StarRocks/starrocks/pull/29225) * The error "StarRocks planner use long time 10000 ms in logical phase" occasionally occurs for complex Join queries. [#34177](https://github.com/StarRocks/starrocks/pull/34177) #### 2.5.13[​](#2513 "Direct link to 2.5.13") Release date: September 28, 2023 ##### Improvements[​](#improvements-8 "Direct link to Improvements") * Window functions COVAR\_SAMP, COVAR\_POP, CORR, VARIANCE, VAR\_SAMP, STD, and STDDEV\_SAMP now support the ORDER BY clause and Window clause. [#30786](https://github.com/StarRocks/starrocks/pull/30786) * An error instead of NULL is returned if a decimal overflow occurs during queries on the DECIMAL type data. [#30419](https://github.com/StarRocks/starrocks/pull/30419) * Executing SQL commands with invalid comments now returns results consistent with MySQL. [#30210](https://github.com/StarRocks/starrocks/pull/30210) * Rowsets corresponding to tablets that have been deleted are cleaned up, reducing the memory usage during BE startup. [#30625](https://github.com/StarRocks/starrocks/pull/30625) ##### Bug Fixes[​](#bug-fixes-9 "Direct link to Bug Fixes") Fixed the following issues: * An error "Set cancelled by MemoryScratchSinkOperator" occurs when users read data from StarRocks using the Spark Connector or Flink Connector. [#30702](https://github.com/StarRocks/starrocks/pull/30702) [#30751](https://github.com/StarRocks/starrocks/pull/30751) * An error "java.lang.IllegalStateException: null" occurs during queries with an ORDER BY clause that includes aggregate functions. [#30108](https://github.com/StarRocks/starrocks/pull/30108) * FEs fail to restart when there are inactive materialized views. [#30015](https://github.com/StarRocks/starrocks/pull/30015) * Performing INSERT OVERWRITE operations on duplicate partitions corrupts the metadata, leading to FE restart failures. [#27545](https://github.com/StarRocks/starrocks/pull/27545) * An error "java.lang.NullPointerException: null" occurs when users modify columns that do not exist in a Primary Key table. [#30366](https://github.com/StarRocks/starrocks/pull/30366) * An error "get TableMeta failed from TNetworkAddress" occurs when users load data into a partitioned StarRocks external table. [#30124](https://github.com/StarRocks/starrocks/pull/30124) * In certain scenarios, an error occurs when users load data via CloudCanal. [#30799](https://github.com/StarRocks/starrocks/pull/30799) * An error "current running txns on db xxx is 200, larger than limit 200" occurs when users load data via the Flink Connector or perform DELETE and INSERT operations. [#18393](https://github.com/StarRocks/starrocks/pull/18393) * Asynchronous materialized views which use HAVING clauses that include aggregate functions cannot rewrite queries properly. [#29976](https://github.com/StarRocks/starrocks/pull/29976) #### 2.5.12[​](#2512 "Direct link to 2.5.12") Release date: September 4, 2023 ##### Improvements[​](#improvements-9 "Direct link to Improvements") * Comments in an SQL are retained in the Audit Log. [#29747](https://github.com/StarRocks/starrocks/pull/29747) * Added CPU and memory statistics of INSERT INTO SELECT to the Audit Log. [#29901](https://github.com/StarRocks/starrocks/pull/29901) ##### Bug Fixes[​](#bug-fixes-10 "Direct link to Bug Fixes") Fixed the following issues: * When Broker Load is used to load data, the NOT NULL attribute of some fields may cause BEs to crash or cause the "msg:mismatched row count" error. [#29832](https://github.com/StarRocks/starrocks/pull/29832) * Queries against ORC-formatted files fail because the bugfix ORC-1304 ([apache/orc#1299](https://github.com/apache/orc/pull/1299)) from Apache ORC is not merged. [#29804](https://github.com/StarRocks/starrocks/pull/29804) * Restoring Primary Key tables causes metadata inconsistency after BEs are restarted. [#30135](https://github.com/StarRocks/starrocks/pull/30135) #### 2.5.11[​](#2511 "Direct link to 2.5.11") Release date: August 28, 2023 ##### Improvements[​](#improvements-10 "Direct link to Improvements") * Supports implicit conversions for all compound predicates and for all expressions in the WHERE clause. You can enable or disable implicit conversions by using the [session variable](https://docs.starrocks.io/docs/sql-reference/System_variable/#enable_strict_type) `enable_strict_type`. The default value is `false`. [#21870](https://github.com/StarRocks/starrocks/pull/21870) * Optimized the prompt returned if users do not specify `hive.metastore.uri` when they create an Iceberg Catalog. The error prompt is more accurate. [#16543](https://github.com/StarRocks/starrocks/issues/16543) * Added more prompts in the error message `xxx too many versions xxx`. [#28397](https://github.com/StarRocks/starrocks/pull/28397) * Dynamic partitioning further supports the partitioning unit to be `year`. [#28386](https://github.com/StarRocks/starrocks/pull/28386) ##### Bug Fixes[​](#bug-fixes-11 "Direct link to Bug Fixes") Fixed the following issues: * When data is loaded into tables with multiple replicas, a large number of invalid log records are written if some partitions of the tables are empty. [#28824](https://github.com/StarRocks/starrocks/issues/28824) * The DELETE operation fails if the field in the WHERE condition is a BITMAP or HLL field. [#28592](https://github.com/StarRocks/starrocks/pull/28592) * Manually refreshing an asynchronous materialized view via a synchronous call (SYNC MODE) results in multiple INSERT OVERWRITE records in the `information_schema.task_runs` table. [#28060](https://github.com/StarRocks/starrocks/pull/28060) * If CLONE operations are triggered on tablets in an ERROR state, disk usage increases. [#28488](https://github.com/StarRocks/starrocks/pull/28488) * When Join Reorder is enabled, the query result is incorrect if the column to query is a constant. [#29239](https://github.com/StarRocks/starrocks/pull/29239) * During tablet migration between SSDs and HDDs, if the FE sends excessive migration tasks to BEs, BEs will encounter OOM issues. [#29055](https://github.com/StarRocks/starrocks/pull/29055) * The security vulnerability in `/apache_hdfs_broker/lib/log4j-1.2.17.jar`. [#28866](https://github.com/StarRocks/starrocks/pull/28866) * During data queries through Hive Catalog, if a partitioning column and an OR operator are used in the WHERE clause, the query result is incorrect. [#28876](https://github.com/StarRocks/starrocks/pull/28876) * The error "java.util.ConcurrentModificationException: null" occasionally occurs during data queries. [#29296](https://github.com/StarRocks/starrocks/pull/29296) * FEs cannot be restarted if the base table of an asynchronous materialized view is dropped. [#29318](https://github.com/StarRocks/starrocks/pull/29318) * For an asynchronous materialized view that is created across databases, the Leader FE occasionally encounters a deadlock when data is being written into base tables of this materialized view. [#29432](https://github.com/StarRocks/starrocks/pull/29432) #### 2.5.10[​](#2510 "Direct link to 2.5.10") Release date: August 7, 2023 ##### New features[​](#new-features-3 "Direct link to New features") * Supports aggregate functions [COVAR\_SAMP](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/covar_samp/), [COVAR\_POP](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/covar_pop/), and [CORR](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/corr/). * Supports the following [window functions](https://docs.starrocks.io/docs/sql-reference/sql-functions/Window_function/): COVAR\_SAMP, COVAR\_POP, CORR, VARIANCE, VAR\_SAMP, STD, and STDDEV\_SAMP. ##### Improvements[​](#improvements-11 "Direct link to Improvements") * Optimized the scheduling logic of TabletChecker to prevent the checker from repeatedly scheduling tablets that are not repaired. [#27648](https://github.com/StarRocks/starrocks/pull/27648) * When Schema Change and Routine Load occur simultaneously, Routine Load jobs may fail if Schema Change completes first. The error message reported in this situation is optimized. [#28425](https://github.com/StarRocks/starrocks/pull/28425) * Users are prohibited from defining NOT NULL columns when they create external tables (If NOT NULL columns are defined, errors will occur after an upgrade and the table must be created again). External catalogs are recommended starting from v2.3.0 to replace external tables. [#25485](https://github.com/StarRocks/starrocks/pull/25441) * Added an error message when Broker Load retries encounter an error. This facilitates troubleshooting and debugging during data loading. [#21982](https://github.com/StarRocks/starrocks/pull/21982) * Supports large-scale data writes when a load job involves both UPSERT and DELETE operations. [#17264](https://github.com/StarRocks/starrocks/pull/17264) * Optimized query rewrite using materialized views. [#27934](https://github.com/StarRocks/starrocks/pull/27934) [#25542](https://github.com/StarRocks/starrocks/pull/25542) [#22300](https://github.com/StarRocks/starrocks/pull/22300) [#27557](https://github.com/StarRocks/starrocks/pull/27557) [#22300](https://github.com/StarRocks/starrocks/pull/22300) [#26957](https://github.com/StarRocks/starrocks/pull/26957) [#27728](https://github.com/StarRocks/starrocks/pull/27728) [#27900](https://github.com/StarRocks/starrocks/pull/27900) ##### Bug Fixes[​](#bug-fixes-12 "Direct link to Bug Fixes") Fixed the following issues: * When CAST is used to convert a string into an array, the result may be incorrect if the input includes constants. [#19793](https://github.com/StarRocks/starrocks/pull/19793) * SHOW TABLET returns incorrect results if it contains ORDER BY and LIMIT. [#23375](https://github.com/StarRocks/starrocks/pull/23375) * Outer join and Anti join rewrite errors for materialized views. [#28028](https://github.com/StarRocks/starrocks/pull/28028) * Incorrect table-level scan statistics in FE cause inaccurate metrics for table queries and loading. [#27779](https://github.com/StarRocks/starrocks/pull/27779) * `An exception occurred when using the current long link to access metastore. msg: Failed to get next notification based on last event id: 707602` is reported in FE logs if event listener is configured on the HMS to incrementally update Hive metadata. [#21056](https://github.com/StarRocks/starrocks/pull/21056) * The query result is not stable if the sort key is modified for a partitioned table. [#27850](https://github.com/StarRocks/starrocks/pull/27850) * Data loaded using Spark Load may be distributed to the wrong buckets if the bucketing column is a DATE, DATETIME, or DECIMAL column. [#27005](https://github.com/StarRocks/starrocks/pull/27005) * The regex\_replace function may cause BEs to crash in some scenarios. [#27117](https://github.com/StarRocks/starrocks/pull/27117) * BE crashes if the input of the sub\_bitmap function is not a BITMAP value. [#27982](https://github.com/StarRocks/starrocks/pull/27982) * "Unknown error" is returned for a query when Join Reorder is enabled. [#27472](https://github.com/StarRocks/starrocks/pull/27472) * Inaccurate estimation of average row size causes Primary Key partial updates to occupy excessively large memory. [#27485](https://github.com/StarRocks/starrocks/pull/27485) * Some INSERT jobs return `[42000][1064] Dict Decode failed, Dict can't take cover all key :0` if low-cardinality optimization is enabled. [#26463](https://github.com/StarRocks/starrocks/pull/26463) * If users specify `"hadoop.security.authentication" = "simple"` in their Broker Load jobs created to load data from HDFS, the job fails. [#27774](https://github.com/StarRocks/starrocks/pull/27774) * Modifying the refresh mode of materialized views causes inconsistent metadata between the leader FE and follower FE. [#28082](https://github.com/StarRocks/starrocks/pull/28082) [#28097](https://github.com/StarRocks/starrocks/pull/28097) * Passwords are not hidden when SHOW CREATE CATALOG and SHOW RESOURCES are used to query specific information. [#28059](https://github.com/StarRocks/starrocks/pull/28059) * FE memory leak caused by blocked LabelCleaner threads. [#28311](https://github.com/StarRocks/starrocks/pull/28311) #### 2.5.9[​](#259 "Direct link to 2.5.9") Release date: July 19, 2023 ##### New features[​](#new-features-4 "Direct link to New features") * Queries that contain a different type of join than the materialized view can be rewritten. [#25099](https://github.com/StarRocks/starrocks/pull/25099) ##### Improvements[​](#improvements-12 "Direct link to Improvements") * StarRocks external tables whose destination cluster is the current StarRocks cluster cannot be created. [#25441](https://github.com/StarRocks/starrocks/pull/25441) * If the queried fields are not included in the output columns of a materialized view but are included in the predicate of the materialized view, the query can still be rewritten. [#23028](https://github.com/StarRocks/starrocks/issues/23028) * Added a new field `table_id` to the table `tables_config` in the database `Information_schema`. You can join `tables_config` with `be_tablets` on the column `table_id` to query the names of the database and table to which a tablet belongs. [#24061](https://github.com/StarRocks/starrocks/pull/24061) ##### Bug Fixes[​](#bug-fixes-13 "Direct link to Bug Fixes") Fixed the following issues: * Count Distinct result is incorrect for Duplicate Key tables. [#24222](https://github.com/StarRocks/starrocks/pull/24222) * BEs may crash if the Join key is a large BINARY column. [#25084](https://github.com/StarRocks/starrocks/pull/25084) * The INSERT operation hangs if the length of CHAR data in a STRUCT to be inserted exceeds the maximum CHAR length defined in the STRUCT column. [#25942](https://github.com/StarRocks/starrocks/pull/25942) * The result of coalesce() is incorrect. [#26250](https://github.com/StarRocks/starrocks/pull/26250) * The version number for a tablet is inconsistent between the BE and FE after data is restored. [#26518](https://github.com/StarRocks/starrocks/pull/26518/files) * Partitions cannot be automatically created for recovered tables. [#26813](https://github.com/StarRocks/starrocks/pull/26813) #### 2.5.8[​](#258 "Direct link to 2.5.8") Release date: June 30, 2023 ##### Improvements[​](#improvements-13 "Direct link to Improvements") * Optimized the error message reported when partitions are added to a non-partitioned table. [#25266](https://github.com/StarRocks/starrocks/pull/25266) * Optimized the [auto tablet distribution policy](https://docs.starrocks.io/docs/2.5/table_design/Data_distribution/#determine-the-number-of-tablets) for tables. [#24543](https://github.com/StarRocks/starrocks/pull/24543) * Optimized the default comments in the CREATE TABLE statement. [#24803](https://github.com/StarRocks/starrocks/pull/24803) * Optimized the manual refreshing of asynchronous materialized views. Supports using the REFRESH MATERIALIZED VIEW WITH SYNC MODE syntax to synchronously invoke materialized view refresh tasks. [#25910](https://github.com/StarRocks/starrocks/pull/25910) ##### Bug Fixes[​](#bug-fixes-14 "Direct link to Bug Fixes") Fixed the following issues: * The COUNT result of an asynchronous materialized view may be inaccurate if the materialized view is built on Union results. [#24460](https://github.com/StarRocks/starrocks/issues/24460) * "Unknown error" is reported when users attempt to forcibly reset the root password. [#25492](https://github.com/StarRocks/starrocks/pull/25492) * Inaccurate error message is displayed when INSERT OVERWRITE is executed on a cluster with less than three alive BEs. [#25314](https://github.com/StarRocks/starrocks/pull/25314) #### 2.5.7[​](#257 "Direct link to 2.5.7") Release date: June 14, 2023 ##### New features[​](#new-features-5 "Direct link to New features") * Inactive materialized views can be manually activated using `ALTER MATERIALIZED VIEW ACTIVE`. You can use this SQL command to activate materialized views whose base tables were dropped and then recreated. For more information, see [ALTER MATERIALIZED VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-definition/ALTER_MATERIALIZED_VIEW/). [#24001](https://github.com/StarRocks/starrocks/pull/24001) * StarRocks can automatically set an appropriate number of tablets when you create a table or add a partition, eliminating the need for manual operations. For more information, see [Determine the number of tablets](https://docs.starrocks.io/docs/2.5/table_design/Data_distribution/#determine-the-number-of-tablets). [#10614](https://github.com/StarRocks/starrocks/pull/10614) ##### Improvements[​](#improvements-14 "Direct link to Improvements") * Optimized the I/O concurrency of Scan nodes used in external table queries, which reduces memory usage and improves the stability of data loading from external tables. [#23617](https://github.com/StarRocks/starrocks/pull/23617) [#23624](https://github.com/StarRocks/starrocks/pull/23624) [#23626](https://github.com/StarRocks/starrocks/pull/23626) * Optimized the error message for Broker Load jobs. The error message contains retry information and the name of erroneous files. [#18038](https://github.com/StarRocks/starrocks/pull/18038) [#21982](https://github.com/StarRocks/starrocks/pull/21982) * Optimized the error message returned when CREATE TABLE times out and added parameter tuning tips. [#24510](https://github.com/StarRocks/starrocks/pull/24510) * Optimized the error message returned when ALTER TABLE fails because the table status is not Normal. [#24381](https://github.com/StarRocks/starrocks/pull/24381) * Ignores full-width spaces in the CREATE TABLE statement. [#23885](https://github.com/StarRocks/starrocks/pull/23885) * Optimized the Broker access timeout to increase the success rate of Broker Load jobs. [#22699](https://github.com/StarRocks/starrocks/pull/22699) * For Primary Key tables, the `VersionCount` field returned by SHOW TABLET contains Rowsets that are in the Pending state. [#23847](https://github.com/StarRocks/starrocks/pull/23847) * Optimized the Persistent Index policy. [#22140](https://github.com/StarRocks/starrocks/pull/22140) ##### Bug Fixes[​](#bug-fixes-15 "Direct link to Bug Fixes") Fixed the following issues: * When users load Parquet data into StarRocks, DATETIME values overflow during type conversion, causing data errors. [#22356](https://github.com/StarRocks/starrocks/pull/22356) * Bucket information is lost after Dynamic Partitioning is disabled. [#22595](https://github.com/StarRocks/starrocks/pull/22595) * Using unsupported properties in the CREATE TABLE statement causes null pointer exceptions (NPEs). [#23859](https://github.com/StarRocks/starrocks/pull/23859) * Table permission filtering in `information_schema` becomes ineffective. As a result, users can view tables they do not have permission to. [#23804](https://github.com/StarRocks/starrocks/pull/23804) * Information returned by SHOW TABLE STATUS is incomplete. [#24279](https://github.com/StarRocks/starrocks/issues/24279) * A schema change sometimes may be hung if data loading occurs simultaneously with the schema change. [#23456](https://github.com/StarRocks/starrocks/pull/23456) * RocksDB WAL flush blocks the brpc worker from processing bthreads, which interrupts high-frequency data loading into Primary Key tables. [#22489](https://github.com/StarRocks/starrocks/pull/22489) * TIME-type columns that are not supported in StarRocks can be successfully created. [#23474](https://github.com/StarRocks/starrocks/pull/23474) * Materialized view Union rewrite fails. [#22922](https://github.com/StarRocks/starrocks/pull/22922) #### 2.5.6[​](#256 "Direct link to 2.5.6") Release date: May 19, 2023 ##### Improvements[​](#improvements-15 "Direct link to Improvements") * Optimized the error message reported when INSERT INTO ... SELECT expires due to a small `thrift_server_max_worker_thread` value. [#21964](https://github.com/StarRocks/starrocks/pull/21964) * Tables created using CTAS have three replicas by default, which is consistent with the default replica number for common tables. [#22854](https://github.com/StarRocks/starrocks/pull/22854) ##### Bug Fixes[​](#bug-fixes-16 "Direct link to Bug Fixes") * Truncating partitions fails because the TRUNCATE operation is case-sensitive to partition names. [#21809](https://github.com/StarRocks/starrocks/pull/21809) * Decommissioning BE fails due to the failure in creating temporary partitions for materialized views. [#22745](https://github.com/StarRocks/starrocks/pull/22745) * Dynamic FE parameters that require an ARRAY value cannot be set to an empty array. [#22225](https://github.com/StarRocks/starrocks/pull/22225) * Materialized views with the `partition_refresh_number` property specified may fail to completely refresh. [#21619](https://github.com/StarRocks/starrocks/pull/21619) * SHOW CREATE TABLE masks cloud credential information, which causes incorrect credential information in memory. [#21311](https://github.com/StarRocks/starrocks/pull/21311) * Predicates cannot take effect on some ORC files that are queried via external tables. [#21901](https://github.com/StarRocks/starrocks/pull/21901) * The min-max filter cannot properly handle lower- and upper-case letters in column names. [#22626](https://github.com/StarRocks/starrocks/pull/22626) * Late materialization causes errors in querying complex data types (STRUCT or MAP). [#22862](https://github.com/StarRocks/starrocks/pull/22862) * The issue that occurs when restoring a Primary Key table. [#23384](https://github.com/StarRocks/starrocks/pull/23384) #### 2.5.5[​](#255 "Direct link to 2.5.5") Release date: April 28, 2023 ##### New features[​](#new-features-6 "Direct link to New features") Added a metric to monitor the tablet status of Primary Key tables: * Added the FE metric `err_state_metric`. * Added the `ErrorStateTabletNum` column to the output of `SHOW PROC '/statistic/'` to display the number of **err\_state** tablets. * Added the `ErrorStateTablets` column to the output of `SHOW PROC '/statistic//'` to display the IDs of **err\_state** tablets. For more information, see [SHOW PROC](https://docs.starrocks.io/docs/sql-reference/sql-statements/Administration/SHOW_PROC/). ##### Improvements[​](#improvements-16 "Direct link to Improvements") * Optimized the disk balancing speed when multiple BEs are added. [# 19418](https://github.com/StarRocks/starrocks/pull/19418) * Optimized the inference of `storage_medium`. When BEs use both SSD and HDD as storage devices, if the property `storage_cooldown_time` is specified, StarRocks sets `storage_medium` to `SSD`. Otherwise, StarRocks sets `storage_medium` to `HDD`. [#18649](https://github.com/StarRocks/starrocks/pull/18649) * Optimized the performance of Unique Key tables by forbidding the collection of statistics from value columns. [#19563](https://github.com/StarRocks/starrocks/pull/19563) ##### Bug Fixes[​](#bug-fixes-17 "Direct link to Bug Fixes") * For Colocation tables, the replica status can be manually specified as `bad` by using statements like `ADMIN SET REPLICA STATUS PROPERTIES ("tablet_id" = "10003", "backend_id" = "10001", "status" = "bad");`. If the number of BEs is less than or equal to the number of replicas, the corrupted replica cannot be repaired. [# 17876](https://github.com/StarRocks/starrocks/issues/17876) * After a BE is started, its process exists but the BE port cannot be enabled. [# 19347](https://github.com/StarRocks/starrocks/pull/19347) * Wrong results are returned for aggregate queries whose subquery is nested with a window function. [# 19725](https://github.com/StarRocks/starrocks/issues/19725) * `auto_refresh_partitions_limit` does not take effect when the materialized view (MV) is refreshed for the first time. As a result, all the partitions are refreshed. [# 19759](https://github.com/StarRocks/starrocks/issues/19759) * An error occurs when querying a CSV Hive external table whose array data is nested with complex data such as MAP and STRUCT. [# 20233](https://github.com/StarRocks/starrocks/pull/20233) * Queries that use Spark connector time out. [# 20264](https://github.com/StarRocks/starrocks/pull/20264) * If one replica of a two-replica table is corrupted, the table cannot recover. [# 20681](https://github.com/StarRocks/starrocks/pull/20681) * Query failure caused by MV query rewrite failure. [# 19549](https://github.com/StarRocks/starrocks/issues/19549) * The metric interface expires due to database lock. [# 20790](https://github.com/StarRocks/starrocks/pull/20790) * Wrong results are returned for Broadcast Join. [# 20952](https://github.com/StarRocks/starrocks/issues/20952) * NPE is returned when an unsupported data type is used in CREATE TABLE. [# 20999](https://github.com/StarRocks/starrocks/issues/20999) * The issue caused by using window\_funnel() with the Query Cache feature. [# 21474](https://github.com/StarRocks/starrocks/issues/21474) * Optimization plan selection takes an unexpectedly long time after the CTE is rewritten. [# 16515](https://github.com/StarRocks/starrocks/pull/16515) #### 2.5.4[​](#254 "Direct link to 2.5.4") Release date: April 4, 2023 ##### Improvements[​](#improvements-17 "Direct link to Improvements") * Optimized the performance of rewriting queries on materialized views during query planning. The amount of time taken for query planning is reduced by about 70%. [#19579](https://github.com/StarRocks/starrocks/pull/19579) * Optimized the type inference logic. If a query like `SELECT sum(CASE WHEN XXX);` contains a constant `0`, such as `SELECT sum(CASE WHEN k1 = 1 THEN v1 ELSE 0 END) FROM test;`, pre-aggregation is automatically enabled to accelerate the query. [#19474](https://github.com/StarRocks/starrocks/pull/19474) * Supports using `SHOW CREATE VIEW` to view the creation statement of a materialized view. [#19999](https://github.com/StarRocks/starrocks/pull/19999) * Supports transmitting packets that are 2 GB or larger in size for a single bRPC request between BE nodes. [#20283](https://github.com/StarRocks/starrocks/pull/20283) [#20230](https://github.com/StarRocks/starrocks/pull/20230) * Supports using [SHOW CREATE CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/SHOW_CREATE_CATALOG/) to query the creation statement of an external catalog. ##### Bug Fixes[​](#bug-fixes-18 "Direct link to Bug Fixes") The following bugs are fixed: * After queries on materialized views are rewritten, the global dictionary for low-cardinality optimization does not take effect. [#19615](https://github.com/StarRocks/starrocks/pull/19615) * If a query on materialized views fails to be rewritten, the query fails. [#19774](https://github.com/StarRocks/starrocks/pull/19774) * If a materialized view is created based on a Primary Key or Unique Key table, queries on that materialized view cannot be rewritten. [#19600](https://github.com/StarRocks/starrocks/pull/19600) * The column names of materialized views are case-sensitive. However, when you create a table, the table is successfully created without an error message even if column names are incorrect in the `PROPERTIES` of the table creation statement, and moreover the rewriting of queries on materialized views created on that table fails. [#19780](https://github.com/StarRocks/starrocks/pull/19780) * After a query on materialized views is rewritten, the query plan ma contain partition column-based, invalid predicates, which affect query performance. [#19784](https://github.com/StarRocks/starrocks/pull/19784) * When data is loaded into a newly created partition, queries on materialized views may fail to be rewritten. [#20323](https://github.com/StarRocks/starrocks/pull/20323) * Configuring `"storage_medium" = "SSD"` at the creation of materialized views causes the refresh of the materialized views to fail. [#19539](https://github.com/StarRocks/starrocks/pull/19539) [#19626](https://github.com/StarRocks/starrocks/pull/19626) * Concurrent compaction may happen on Primary Key tables. [#19692](https://github.com/StarRocks/starrocks/pull/19692) * Compaction does not occur promptly after a large number of DELETE operations. [#19623](https://github.com/StarRocks/starrocks/pull/19623) * If the expression of a statement contains multiple low-cardinality columns, the expression may fail to be properly rewritten. As a result, the global dictionary for low-cardinality optimization does not take effect. [#20161](https://github.com/StarRocks/starrocks/pull/20161) #### 2.5.3[​](#253 "Direct link to 2.5.3") Release date: March 10, 2023 ##### Improvements[​](#improvements-18 "Direct link to Improvements") * Optimized query rewrite for materialized views (MVs). * Supports rewriting queries with Outer Join and Cross Join. [#18629](https://github.com/StarRocks/starrocks/pull/18629) * Optimized the data scan logic for MVs, further accelerating the rewritten queries. [#18629](https://github.com/StarRocks/starrocks/pull/18629) * Enhanced rewrite capabilities for single-table aggregate queries. [#18629](https://github.com/StarRocks/starrocks/pull/18629) * Enhanced rewrite capabilities in View Delta scenarios, which is when the queried tables are a subset of the MV's base tables. [#18800](https://github.com/StarRocks/starrocks/pull/18800) * Optimized the performance and memory usage when the window function RANK() is used as a filter or a sort key. [#17553](https://github.com/StarRocks/starrocks/issues/17553) ##### Bug Fixes[​](#bug-fixes-19 "Direct link to Bug Fixes") The following bugs are fixed: * Errors caused by null literals `[]` in ARRAY data. [#18563](https://github.com/StarRocks/starrocks/pull/18563) * Misuse of the low-cardinality optimization dictionary in some complex query scenarios. The dictionary mapping check is now added before applying the dictionary. [#17318](https://github.com/StarRocks/starrocks/pull/17318) * In a single BE environment, Local Shuffle causes GROUP BY to produce duplicate results. [#17845](https://github.com/StarRocks/starrocks/pull/17845) * Misuses of partition-related PROPERTIES for a non-partitioned MV may cause the MV refresh to fail. The partition PROPERTIES check is now performed when users create an MV. [#18741](https://github.com/StarRocks/starrocks/pull/18741) * Errors in parsing Parquet Repetition columns. [#17626](https://github.com/StarRocks/starrocks/pull/17626) [#17788](https://github.com/StarRocks/starrocks/pull/17788) [#18051](https://github.com/StarRocks/starrocks/pull/18051) * The obtained column's nullable information is incorrect. Solution: When CTAS is used to create a Primary Key table, only the primary key columns are non-nullable; non-primary key columns are nullable. [#16431](https://github.com/StarRocks/starrocks/pull/16431) * Some issues caused by deleting data from Primary Key tables. [#18768](https://github.com/StarRocks/starrocks/pull/18768) #### 2.5.2[​](#252 "Direct link to 2.5.2") Release date: February 21, 2023 ##### New Features[​](#new-features-7 "Direct link to New Features") * Supports using the Instance Profile and Assumed Role-based credential methods to access AWS S3 and AWS Glue. [#15958](https://github.com/StarRocks/starrocks/pull/15958) * Supports the following bit functions: bit\_shift\_left, bit\_shift\_right, and bit\_shift\_right\_logical. [#14151](https://github.com/StarRocks/starrocks/pull/14151) ##### Improvements[​](#improvements-19 "Direct link to Improvements") * Optimized the memory release logic, which significantly reduces peak memory usage when a query contains a large number of aggregate queries. [#16913](https://github.com/StarRocks/starrocks/pull/16913) * Reduced the memory usage of sorting. The memory consumption is halved when a query involves window functions or sorting. [#16937](https://github.com/StarRocks/starrocks/pull/16937) [#17362](https://github.com/StarRocks/starrocks/pull/17362) [#17408](https://github.com/StarRocks/starrocks/pull/17408) ##### Bug Fixes[​](#bug-fixes-20 "Direct link to Bug Fixes") The following bugs are fixed: * Apache Hive external tables that contain MAP and ARRAY data cannot be refreshed. [#17548](https://github.com/StarRocks/starrocks/pull/17548) * Superset cannot identify column types of materialized views. [#17686](https://github.com/StarRocks/starrocks/pull/17686) * BI connectivity fails because SET GLOBAL/SESSION TRANSACTION cannot be parsed. [#17295](https://github.com/StarRocks/starrocks/pull/17295) * The bucket number of dynamic partitioned tables in a Colocate Group cannot be modified and an error message is returned. [#17418](https://github.com/StarRocks/starrocks/pull/17418/) * Potential issues caused by a failure in the Prepare stage. [#17323](https://github.com/StarRocks/starrocks/pull/17323) ##### Behavior Change[​](#behavior-change-1 "Direct link to Behavior Change") * Changed the default value of `enable_experimental_mv` from `false` to `true`, which means asynchronous materialized view is enabled by default. * Added CHARACTER to the reserved keyword list. [#17488](https://github.com/StarRocks/starrocks/pull/17488) #### 2.5.1[​](#251 "Direct link to 2.5.1") Release date: February 5, 2023 ##### Improvements[​](#improvements-20 "Direct link to Improvements") * Asynchronous materialized views created based on external catalogs support query rewrite. [#11116](https://github.com/StarRocks/starrocks/issues/11116) [#15791](https://github.com/StarRocks/starrocks/issues/15791) * Allows users to specify a collection period for automatic CBO statistics collection, which prevents cluster performance jitter caused by automatic full collection. [#14996](https://github.com/StarRocks/starrocks/pull/14996) * Added Thrift server queue. Requests that cannot be processed immediately during INSERT INTO SELECT can be pending in the Thrift server queue, preventing requests from being rejected. [#14571](https://github.com/StarRocks/starrocks/pull/14571) * Deprecated the FE parameter `default_storage_medium`. If `storage_medium` is not explicitly specified when users create a table, the system automatically infers the storage medium of the table based on BE disk type. For more information, see description of `storage_medium` in [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-definition/CREATE_VIEW/). [#14394](https://github.com/StarRocks/starrocks/pull/14394) ##### Bug Fixes[​](#bug-fixes-21 "Direct link to Bug Fixes") The following bugs are fixed: * Null pointer exception (NPE) caused by SET PASSWORD. [#15247](https://github.com/StarRocks/starrocks/pull/15247) * JSON data with empty keys cannot be parsed. [#16852](https://github.com/StarRocks/starrocks/pull/16852) * Data of invalid types can be successfully converted into ARRAY data. [#16866](https://github.com/StarRocks/starrocks/pull/16866) * Nested Loop Join cannot be interrupted when an exception occurs. [#16875](https://github.com/StarRocks/starrocks/pull/16875) ##### Behavior Change[​](#behavior-change-2 "Direct link to Behavior Change") * Deprecated the FE parameter `default_storage_medium`. The storage medium of a table is automatically inferred by the system. [#14394](https://github.com/StarRocks/starrocks/pull/14394) #### 2.5.0[​](#250 "Direct link to 2.5.0") Release date: January 22, 2023 ##### New Features[​](#new-features-8 "Direct link to New Features") * Supports querying Merge On Read tables using [Hudi catalogs](https://docs.starrocks.io/docs/data_source/catalog/hudi_catalog/) and [Hudi external tables](https://docs.starrocks.io/docs/data_source/External_table#deprecated-hudi-external-table). [#6780](https://github.com/StarRocks/starrocks/pull/6780) * Supports querying STRUCT and MAP data using [Hive catalogs](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog/), Hudi catalogs, and [Iceberg catalogs](https://docs.starrocks.io/docs/data_source/catalog/iceberg_catalog/). [#10677](https://github.com/StarRocks/starrocks/issues/10677) * Provides [Data Cache](https://docs.starrocks.io/docs/data_source/data_cache/) to improve access performance of hot data stored in external storage systems, such as HDFS. [#11597](https://github.com/StarRocks/starrocks/pull/11579) * Supports creating [Delta Lake catalogs](https://docs.starrocks.io/docs/data_source/catalog/deltalake_catalog/), which allow direct queries on data from Delta Lake. [#11972](https://github.com/StarRocks/starrocks/issues/11972) * Hive, Hudi, and Iceberg catalogs are compatible with AWS Glue. [#12249](https://github.com/StarRocks/starrocks/issues/12249) * Supports creating [file external tables](https://docs.starrocks.io/docs/data_source/file_external_table/), which allow direct queries on Parquet and ORC files from HDFS and object stores. [#13064](https://github.com/StarRocks/starrocks/pull/13064) * Supports creating materialized views based on Hive, Hudi, Iceberg catalogs, and materialized views. For more information, see [Materialized view](https://docs.starrocks.io/docs/using_starrocks/Materialized_view/). [#11116](https://github.com/StarRocks/starrocks/issues/11116) [#11873](https://github.com/StarRocks/starrocks/pull/11873) * Supports conditional updates for tables that use the Primary Key table. For more information, see [Change data through loading](https://docs.starrocks.io/docs/loading/Load_to_Primary_Key_tables/). [#12159](https://github.com/StarRocks/starrocks/pull/12159) * Supports [Query Cache](https://docs.starrocks.io/docs/using_starrocks/query_cache/), which stores intermediate computation results of queries, improving the QPS and reduces the average latency of highly-concurrent, simple queries. [#9194](https://github.com/StarRocks/starrocks/pull/9194) * Supports specifying the priority of Broker Load jobs. For more information, see [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/BROKER_LOAD/) [#11029](https://github.com/StarRocks/starrocks/pull/11029) * Supports specifying the number of replicas for data loading for StarRocks native tables. For more information, see [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-definition/CREATE_TABLE/). [#11253](https://github.com/StarRocks/starrocks/pull/11253) * Supports [query queues](https://docs.starrocks.io/docs/administration/query_queues/). [#12594](https://github.com/StarRocks/starrocks/pull/12594) * Supports isolating compute resources occupied by data loading, thereby limiting the resource consumption of data loading tasks. For more information, see [Resource group](https://docs.starrocks.io/docs/administration/resource_group/). [#12606](https://github.com/StarRocks/starrocks/pull/12606) * Supports specifying the following data compression algorithms for StarRocks native tables: LZ4, Zstd, Snappy, and Zlib. For more information, see [Data compression](https://docs.starrocks.io/docs/table_design/data_compression/). [#10097](https://github.com/StarRocks/starrocks/pull/10097) [#12020](https://github.com/StarRocks/starrocks/pull/12020) * Supports [user-defined variables](https://docs.starrocks.io/docs/reference/user_defined_variables/). [#10011](https://github.com/StarRocks/starrocks/pull/10011) * Supports [lambda expression](https://docs.starrocks.io/docs/sql-reference/sql-functions/Lambda_expression/) and the following higher-order functions: [array\_map](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_map/), [array\_sum](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_sum/), and [array\_sortby](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_sortby/). [#9461](https://github.com/StarRocks/starrocks/pull/9461) [#9806](https://github.com/StarRocks/starrocks/pull/9806) [#10323](https://github.com/StarRocks/starrocks/pull/10323) [#14034](https://github.com/StarRocks/starrocks/pull/14034) * Provides the QUALIFY clause that filters the results of [window functions](https://docs.starrocks.io/docs/sql-reference/sql-functions/Window_function/). [#13239](https://github.com/StarRocks/starrocks/pull/13239) * Supports using the result returned by the uuid() and uuid\_numeric() functions as the default value of a column when you create a table. For more information, see [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-definition/CREATE_TABLE/). [#11155](https://github.com/StarRocks/starrocks/pull/11155) * Supports the following functions: [map\_size](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_size/), [map\_keys](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_keys/), [map\_values](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_values/), [max\_by](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/max_by/), [sub\_bitmap](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/sub_bitmap/), [bitmap\_to\_base64](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_to_base64/), [host\_name](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/host_name/), and [date\_slice](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/date_slice/). [#11299](https://github.com/StarRocks/starrocks/pull/11299) [#11323](https://github.com/StarRocks/starrocks/pull/11323) [#12243](https://github.com/StarRocks/starrocks/pull/12243) [#11776](https://github.com/StarRocks/starrocks/pull/11776) [#12634](https://github.com/StarRocks/starrocks/pull/12634) [#14225](https://github.com/StarRocks/starrocks/pull/14225) ##### Improvements[​](#improvements-21 "Direct link to Improvements") * Optimized the metadata access performance when you query external data using [Hive catalogs](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog/), [Hudi catalogs](https://docs.starrocks.io/docs/data_source/catalog/hudi_catalog/), and [Iceberg catalogs](https://docs.starrocks.io/docs/data_source/catalog/iceberg_catalog/). [#11349](https://github.com/StarRocks/starrocks/issues/11349) * Supports querying ARRAY data using [Elasticsearch external tables](https://docs.starrocks.io/docs/data_source/External_table#deprecated-elasticsearch-external-table). [#9693](https://github.com/StarRocks/starrocks/pull/9693) * Optimized the following aspects of materialized views: * Asynchronous materialized views support automatic and transparent query rewrite based on the SPJG-type materialized views. For more information, see [Materialized view](https://docs.starrocks.io/docs/using_starrocks/Materialized_view#rewrite-and-accelerate-queries-with-the-asynchronous-materialized-view). [#13193](https://github.com/StarRocks/starrocks/issues/13193) * Asynchronous materialized views support multiple async refresh mechanisms. For more information, see [Materialized view](https://docs.starrocks.io/docs/using_starrocks/Materialized_view#manually-refresh-an-asynchronous-materialized-view). [#12712](https://github.com/StarRocks/starrocks/pull/12712) [#13171](https://github.com/StarRocks/starrocks/pull/13171) [#13229](https://github.com/StarRocks/starrocks/pull/13229) [#12926](https://github.com/StarRocks/starrocks/pull/12926) * The efficiency of refreshing materialized views is improved. [#13167](https://github.com/StarRocks/starrocks/issues/13167) * Optimized the following aspects of data loading: * Optimized loading performance in multi-replica scenarios by supporting the "single leader replication" mode. Data loading gains a one-fold performance lift. For more information about "single leader replication", see `replicated_storage` in [CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-definition/CREATE_TABLE/). [#10138](https://github.com/StarRocks/starrocks/pull/10138) * Broker Load and Spark Load no longer need to depend on brokers for data loading when only one HDFS cluster or one Kerberos user is configured. However, if you have multiple HDFS clusters or multiple Kerberos users, you still need to deploy a broker. For more information, see [Load data from HDFS or cloud storage](https://docs.starrocks.io/docs/loading/BrokerLoad/) and [Bulk load using Apache Spark™](https://docs.starrocks.io/docs/loading/SparkLoad/). [#9049](https://github.com/starrocks/starrocks/pull/9049) [#9228](https://github.com/StarRocks/starrocks/pull/9228) * Optimized the performance of Broker Load when a large number of small ORC files are loaded. [#11380](https://github.com/StarRocks/starrocks/pull/11380) * Reduced the memory usage when you load data into Primary Key tables. * Optimized the `information_schema` database and the `tables` and `columns` tables within. Adds a new table `table_config`. For more information, see [Information Schema](https://docs.starrocks.io/docs/reference/overview-pages/information_schema/). [#10033](https://github.com/StarRocks/starrocks/pull/10033) * Optimized data backup and restore: * Supports backing up and restoring data from multiple tables in a database at a time. For more information, see [Backup and restore data](https://docs.starrocks.io/docs/administration/Backup_and_restore/). [#11619](https://github.com/StarRocks/starrocks/issues/11619) * Supports backing up and restoring data from Primary Key tables. For more information, see Backup and restore. [#11885](https://github.com/StarRocks/starrocks/pull/11885) * Optimized the following functions: * Added an optional parameter for the [time\_slice](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/time_slice/) function, which is used to determine whether the beginning or end of the time interval is returned. [#11216](https://github.com/StarRocks/starrocks/pull/11216) * Added a new mode `INCREASE` for the [window\_funnel](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/window_funnel/) function to avoid computing duplicate timestamps. [#10134](https://github.com/StarRocks/starrocks/pull/10134) * Supports specifying multiple arguments in the [unnest](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/unnest/) function. [#12484](https://github.com/StarRocks/starrocks/pull/12484) * lead() and lag() functions support querying HLL and BITMAP data. For more information, see [Window function](https://docs.starrocks.io/docs/sql-reference/sql-functions/Window_function/). [#12108](https://github.com/StarRocks/starrocks/pull/12108) * The following ARRAY functions support querying JSON data: [array\_agg](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_agg/), [array\_sort](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_sort/), [array\_concat](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_concat/), [array\_slice](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_slice/), and [reverse](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/reverse/). [#13155](https://github.com/StarRocks/starrocks/pull/13155) * Optimized the use of some functions. The `current_date`, `current_timestamp`, `current_time`, `localtimestamp`, and `localtime` functions can be executed without using `()`, for example, you can directly run `select current_date;`. [# 14319](https://github.com/StarRocks/starrocks/pull/14319) * Removed some redundant information from FE logs. [# 15374](https://github.com/StarRocks/starrocks/pull/15374) ##### Bug Fixes[​](#bug-fixes-22 "Direct link to Bug Fixes") The following bugs are fixed: * The append\_trailing\_char\_if\_absent() function may return an incorrect result when the first argument is empty. [#13762](https://github.com/StarRocks/starrocks/pull/13762) * After a table is restored using the RECOVER statement, the table does not exist. [#13921](https://github.com/StarRocks/starrocks/pull/13921) * The result returned by the SHOW CREATE MATERIALIZED VIEW statement does not contain the database and catalog specified in the query statement when the materialized view was created. [#12833](https://github.com/StarRocks/starrocks/pull/12833) * Schema change jobs in the `waiting_stable` state cannot be canceled. [#12530](https://github.com/StarRocks/starrocks/pull/12530) * Running the `SHOW PROC '/statistic';` command on a Leader FE and non-Leader FE returns different results. [#12491](https://github.com/StarRocks/starrocks/issues/12491) * The position of the ORDER BY clause is incorrect in the result returned by SHOW CREATE TABLE. [# 13809](https://github.com/StarRocks/starrocks/pull/13809) * When users use Hive Catalog to query Hive data, if the execution plan generated by FE does not contain partition IDs, BEs fail to query Hive partition data. [# 15486](https://github.com/StarRocks/starrocks/pull/15486). ##### Behavior Change[​](#behavior-change-3 "Direct link to Behavior Change") * Changed the default value of the `AWS_EC2_METADATA_DISABLED` parameter to `False`, which means that the metadata of Amazon EC2 is obtained to access AWS resources. * Renamed session variable `is_report_success` to `enable_profile`, which can be queried using the SHOW VARIABLES statement. * Added four reserved keywords: `CURRENT_DATE`, `CURRENT_TIME`, `LOCALTIME`, and `LOCALTIMESTAMP`. [# 14319](https://github.com/StarRocks/starrocks/pull/14319) * The maximum length of table and database names can be up to 1023 characters. [# 14929](https://github.com/StarRocks/starrocks/pull/14929) [# 15020](https://github.com/StarRocks/starrocks/pull/15020) * BE configuration items `enable_event_based_compaction_framework` and `enable_size_tiered_compaction_strategy` are set to `true` by default, which significantly reduces compaction overheads when there are a large number of tablets or a single tablet has large data volume. ##### Upgrade Notes[​](#upgrade-notes "Direct link to Upgrade Notes") * You can upgrade your cluster to 2.5.0 from 2.0.x, 2.1.x, 2.2.x, 2.3.x, or 2.4.x. However, if you need to perform a rollback, we recommend that you roll back only to 2.4.x. --- ## Release 3.0 ### StarRocks version 3.0 #### 3.0.9[​](#309 "Direct link to 3.0.9") Release date: January 2, 2024 ##### New features[​](#new-features "Direct link to New features") * Added the [percentile\_disc](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/percentile_disc/) function. [#36352](https://github.com/StarRocks/starrocks/pull/36352) * Added a new metric `max_tablet_rowset_num` for setting the maximum allowed number of rowsets. This metric helps detect possible compaction issues and thus reduces the occurrences of the error "too many versions". [#36539](https://github.com/StarRocks/starrocks/pull/36539) ##### Improvements[​](#improvements "Direct link to Improvements") * A new value option `GROUP_CONCAT_LEGACY` is added to the session variable [sql\_mode](https://docs.starrocks.io/docs/sql-reference/System_variable/#sql_mode) to provide compatibility with the implementation logic of the [group\_concat](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/group_concat/) function in versions earlier than v2.5. [#36150](https://github.com/StarRocks/starrocks/pull/36150) * When using JDK, the default GC algorithm is G1. [#37386](https://github.com/StarRocks/starrocks/pull/37386) * The `be_tablets` view in the `information_schema` database provides a new field `INDEX_DISK`, which records the disk usage (measured in bytes) of persistent indexes [#35615](https://github.com/StarRocks/starrocks/pull/35615) * Queries on MySQL external tables and the external tables within JDBC catalogs support including keywords in the WHERE clause. [#35917](https://github.com/StarRocks/starrocks/pull/35917) * Supports updates onto the specified partitions of an automatically partitioned table. If the specified partitions do not exist, an error is returned. [#34777](https://github.com/StarRocks/starrocks/pull/34777) * The Primary Key table size returned by the [SHOW DATA](https://docs.starrocks.io/docs/sql-reference/sql-statements/Database/SHOW_DATA/) statement includes the sizes of **.cols** files (these are files related to partial column updates and generated columns) and persistent index files. [#34898](https://github.com/StarRocks/starrocks/pull/34898) * Optimized the performance of persistent index update when compaction is performed on all rowsets of a Primary Key table, which reduces disk read I/O. [#36819](https://github.com/StarRocks/starrocks/pull/36819) * When the string on the right side of the LIKE operator within the WHERE clause does not include `%` or `_`, the LIKE operator is converted into the `=` operator. [#37515](https://github.com/StarRocks/starrocks/pull/37515) * Optimized the logic used to compute compaction scores for Primary Key tables, thereby aligning the compaction scores for Primary Key tables within a more consistent range with the other three table types. [#36534](https://github.com/StarRocks/starrocks/pull/36534) * The result returned by the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD_TASK/) statement now includes the timestamps of consumption messages from each partition. [#36222](https://github.com/StarRocks/starrocks/pull/36222) * Optimized the performance of some Bitmap-related operations, including: * Optimized nested loop joins. [#340804](https://github.com/StarRocks/starrocks/pull/34804) [#35003](https://github.com/StarRocks/starrocks/pull/35003) * Optimized the `bitmap_xor` function. [#34069](https://github.com/StarRocks/starrocks/pull/34069) * Supports Copy on Write to optimize Bitmap performance and reduce memory consumption. [#34047](https://github.com/StarRocks/starrocks/pull/34047) ##### Behavior Change[​](#behavior-change "Direct link to Behavior Change") * Added the session variable `enable_materialized_view_for_insert`, which controls whether materialized views rewrite the queries in INSERT INTO SELECT statements. The default value is `false`. [#37505](https://github.com/StarRocks/starrocks/pull/37505) * Changed the FE configuration item `enable_new_publish_mechanism` to a static parameter from a dynamic one. You must restart the FE after you modify the parameter settings. [#35338](https://github.com/StarRocks/starrocks/pull/35338) * Changed the default retention period of trash files to 1 day from the original 3 days. [#37113](https://github.com/StarRocks/starrocks/pull/37113) ##### Parameter Change[​](#parameter-change "Direct link to Parameter Change") ###### Session variables[​](#session-variables "Direct link to Session variables") * Added session variable `cbo_decimal_cast_string_strict`, which controls how the CBO converts data from the DECIMAL type to the STRING type. If this variable is set to `true`, the logic built in v2.5.x and later versions prevails and the system implements strict conversion (namely, the system truncates the generated string and fills 0s based on the scale length). If this variable is set to `false`, the logic built in versions earlier than v2.5.x prevails and the system processes all valid digits to generate a string. The default value is `true`. [#34208](https://github.com/StarRocks/starrocks/pull/34208) * Added session variables `transaction_read_only` and `tx_read_only` to specify the transaction access mode, which are compatible with MySQL versions 5.7.20 and above. [#37249](https://github.com/StarRocks/starrocks/pull/37249) ###### FE Parameters[​](#fe-parameters "Direct link to FE Parameters") * Added the FE configuration item `routine_load_unstable_threshold_second`. [#36222](https://github.com/StarRocks/starrocks/pull/36222) * Added the FE configuration item `http_worker_threads_num`, which specifies the number of threads for HTTP server to deal with HTTP requests. The default value is `0`. If the value for this parameter is set to a negative value or 0, the actual thread number is twice the number of CPU cores. [#37530](https://github.com/StarRocks/starrocks/pull/37530) * Added the FE configuration item `default_mv_refresh_immediate`, which specifies whether to immediately refresh the materialized view after the materialized view is created. The default value is `true`. [#37093](https://github.com/StarRocks/starrocks/pull/37093) ###### BE Parameters[​](#be-parameters "Direct link to BE Parameters") * Added the BE configuration item `enable_stream_load_verbose_log`. The default value is `false`. With this parameter set to `true`, StarRocks can record the HTTP requests and responses for Stream Load jobs, making troubleshooting easier. [#36113](https://github.com/StarRocks/starrocks/pull/36113) * Added the BE configuration item `pindex_major_compaction_limit_per_disk` to configure the maximum concurrency of compaction on a disk. This addresses the issue of uneven I/O across disks due to compaction. This issue can cause excessively high I/O for certain disks. The default value is `1`. [#37694](https://github.com/StarRocks/starrocks/pull/37694) * Added BE configuration items to specify the timeout duration for connecting to object storage: * `object_storage_connect_timeout_ms`: Timeout duration to establish socket connections with object storage. The default value is `-1`, which means to use the default timeout duration of the SDK configurations. * `object_storage_request_timeout_ms`: Timeout duration to establish HTTP connections with object storage. The default value is `-1`, which means to use the default timeout duration of the SDK configurations. ##### Bug Fixes[​](#bug-fixes "Direct link to Bug Fixes") Fixed the following issues: * In some cases, BEs may crash when a Catalog is used to read ORC external tables. [#27971](https://github.com/StarRocks/starrocks/pull/27971) * The BEs crash if users create persistent indexes in the event of data corruption. [#30841](https://github.com/StarRocks/starrocks/pull/30841) * BEs occasionally crash after a Bitmap index is added. [#26463](https://github.com/StarRocks/starrocks/pull/26463) * Failures in replaying replica operations may cause FEs to crash. [#32295](https://github.com/StarRocks/starrocks/pull/32295) * Setting the FE parameter `recover_with_empty_tablet` to `true` may cause FEs to crash. [#33071](https://github.com/StarRocks/starrocks/pull/33071) * Queries fail during hash joins, causing BEs to crash. [#32219](https://github.com/StarRocks/starrocks/pull/32219) * In a StarRocks shared-nothing cluster, queries against Iceberg or Hive tables may cause BEs to crash. [#34682](https://github.com/StarRocks/starrocks/pull/34682) * The error "get\_applied\_rowsets failed, tablet updates is in error state: tablet:18849 actual row size changed after compaction" is returned for queries. [#33246](https://github.com/StarRocks/starrocks/pull/33246) * Running `show proc '/statistic'` may cause a deadlock. [#34237](https://github.com/StarRocks/starrocks/pull/34237/files) * The FE performance plunges after the FE configuration item `enable_collect_query_detail_info` is set to `true`. [#35945](https://github.com/StarRocks/starrocks/pull/35945) * Errors may be thrown if large amounts of data are loaded into a Primary Key table with persistent index enabled. [#34352](https://github.com/StarRocks/starrocks/pull/34352) * After StarRocks is upgraded from v2.4 or earlier to a later version, compaction scores may rise unexpectedly. [#34618](https://github.com/StarRocks/starrocks/pull/34618) * If `INFORMATION_SCHEMA` is queried by using the database driver MariaDB ODBC, the `CATALOG_NAME` column returned in the `schemata` view holds only `null` values. [#34627](https://github.com/StarRocks/starrocks/pull/34627) * FEs crash due to the abnormal data loaded and cannot restart. [#34590](https://github.com/StarRocks/starrocks/pull/34590) * If schema changes are being executed while a Stream Load job is in the **PREPARED** state, a portion of the source data to be loaded by the job is lost. [#34381](https://github.com/StarRocks/starrocks/pull/34381) * Including two or more slashes (`/`) at the end of the HDFS storage path causes the backup and restore of the data from HDFS to fail. [#34601](https://github.com/StarRocks/starrocks/pull/34601) * The `partition_live_number` property added by using the ALTER TABLE statement does not take effect. [#34842](https://github.com/StarRocks/starrocks/pull/34842) * The [array\_distinct](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_distinct/) function occasionally causes the BEs to crash. [#36377](https://github.com/StarRocks/starrocks/pull/36377) * Deadlocks may occur when users refresh materialized views. [#35736](https://github.com/StarRocks/starrocks/pull/35736) * Global Runtime Filter may cause BEs to crash in certain scenarios. [#35776](https://github.com/StarRocks/starrocks/pull/35776) * In some cases, `bitmap_to_string` may return incorrect result due to data type overflow. [#37405](https://github.com/StarRocks/starrocks/pull/37405) #### 3.0.8[​](#308 "Direct link to 3.0.8") Release date: November 17, 2023 ##### Improvements[​](#improvements-1 "Direct link to Improvements") * The `COLUMNS` view in the system database `INFORMATION_SCHEMA` can display ARRAY, MAP, and STRUCT columns. [#33431](https://github.com/StarRocks/starrocks/pull/33431) ##### Bug Fixes[​](#bug-fixes-1 "Direct link to Bug Fixes") Fixed the following issues: * When `show proc '/current_queries';` is being executed and meanwhile a query begins to be executed, BEs may crash. [#34316](https://github.com/StarRocks/starrocks/pull/34316) * When data is continuously loaded into a Primary Key table with a sort key specified at a high frequency, compaction failures may occur. [#26486](https://github.com/StarRocks/starrocks/pull/26486) * If a filtering condition is specified in a Broker Load job, BEs may crash during the data loading in certain circumstances. [#29832](https://github.com/StarRocks/starrocks/pull/29832) * An unknown error is reported when SHOW GRANTS is executed. [#30100](https://github.com/StarRocks/starrocks/pull/30100) * BE may crash for specific data types if the target data type specified in the `cast()` function is the same as the original data type. [#31465](https://github.com/StarRocks/starrocks/pull/31465) * `DATA_TYPE` and `COLUMN_TYPE` for BINARY or VARBINARY data types are displayed as `unknown` in the `information_schema.columns` view. [#32678](https://github.com/StarRocks/starrocks/pull/32678) * Long-time, frequent data loading into a Primary Key table with persistent index enabled may cause BEs to crash. [#33220](https://github.com/StarRocks/starrocks/pull/33220) * The query result is incorrect when Query Cache is enabled. [#32778](https://github.com/StarRocks/starrocks/pull/32778) * After a cluster is restarted, the data in a restored table may be inconsistent with the data in that table before being backed up. [#33567](https://github.com/StarRocks/starrocks/pull/33567) * If RESTORE is executed and meanwhile Compaction takes place, it may cause BEs to crash. [#32902](https://github.com/StarRocks/starrocks/pull/32902) #### 3.0.7[​](#307 "Direct link to 3.0.7") Release date: October 18, 2023 ##### Improvements[​](#improvements-2 "Direct link to Improvements") * Window functions COVAR\_SAMP, COVAR\_POP, CORR, VARIANCE, VAR\_SAMP, STD, and STDDEV\_SAMP now support the ORDER BY clause and Window clause. [#30786](https://github.com/StarRocks/starrocks/pull/30786) * The Publish phase of a load job that writes data into a Primary Key table is changed from asynchronous mode to synchronous mode. As such, the data loaded can be queried immediately after the load job finishes. [#27055](https://github.com/StarRocks/starrocks/pull/27055) * An error instead of NULL is returned if a decimal overflow occurs during queries on the DECIMAL type data. [#30419](https://github.com/StarRocks/starrocks/pull/30419) * Executing SQL commands with invalid comments now returns results consistent with MySQL. [#30210](https://github.com/StarRocks/starrocks/pull/30210) * For a StarRocks table that uses RANGE partitioning with only one partitioning column or expression partitioning, SQL predicates containing partition column expressions can also be used for partition pruning. [#30421](https://github.com/StarRocks/starrocks/pull/30421) ##### Bug Fixes[​](#bug-fixes-2 "Direct link to Bug Fixes") Fixed the following issues: * Concurrently creating and deleting databases and tables can, in certain cases, result in the table not being found and further leads to the failure of data loading into that table. [#28985](https://github.com/StarRocks/starrocks/pull/28985) * Using UDFs may lead to memory leaks in certain cases. [#29467](https://github.com/StarRocks/starrocks/pull/29467) [#29465](https://github.com/StarRocks/starrocks/pull/29465) * If the ORDER BY clause contains aggregate functions, an error "java.lang.IllegalStateException: null" is returned. [#30108](https://github.com/StarRocks/starrocks/pull/30108) * If users run queries against data stored in Tencent COS by using their Hive catalog which consists of multiple levels, the query results will be incorrect. [#30363](https://github.com/StarRocks/starrocks/pull/30363) * If some subcfields of the STRUCT in ARRAY\ type data are missing, the data length is incorrect when default values are filled in the missing subcfields during queries, which causes BEs to crash. * The version of Berkeley DB Java Edition is upgraded to avoid security vulnerabilities.[#30029](https://github.com/StarRocks/starrocks/pull/30029) * If users load data into a Primary Key table on which truncate operations and queries are concurrently performed, an error "java.lang.NullPointerException" is thrown in certain cases. [#30573](https://github.com/StarRocks/starrocks/pull/30573) * If the Schema Change execution time is too long, it may fail because the tablet of the specified version is garbage-collected. [#31376](https://github.com/StarRocks/starrocks/pull/31376) * If users use CloudCanal to load data into table columns that are set to `NOT NULL` but have no default value specified, an error "Unsupported dataFormat value is : \N" is thrown. [#30799](https://github.com/StarRocks/starrocks/pull/30799) * In StarRocks shared-data clusters, the information of table keys is not recorded in `information_schema.COLUMNS`. As a result, DELETE operations cannot be performed when data is loaded by using Flink Connector. [#31458](https://github.com/StarRocks/starrocks/pull/31458) * During the upgrade, if the types of certain columns are also upgraded (for example, from Decimal type to Decimal v3 type), compaction on certain tables with specific characteristics may cause BEs to crash. [#31626](https://github.com/StarRocks/starrocks/pull/31626) * When data is loaded by using Flink Connector, the load job is suspended unexpectedly if there are highly concurrent load jobs and both the number of HTTP threads and the number of Scan threads have reached their upper limits. [#32251](https://github.com/StarRocks/starrocks/pull/32251) * BEs crash when libcurl is invoked. [#31667](https://github.com/StarRocks/starrocks/pull/31667) * An error occurs when a column of BITMAP type is added to a Primary Key table. [#31763](https://github.com/StarRocks/starrocks/pull/31763) #### 3.0.6[​](#306 "Direct link to 3.0.6") Release date: September 12, 2023 ##### Behavior Change[​](#behavior-change-1 "Direct link to Behavior Change") * When using the [group\_concat](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/group_concat/) function, you must use the SEPARATOR keyword to declare the separator. ##### New Features[​](#new-features-1 "Direct link to New Features") * The aggregate function [group\_concat](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/group_concat/) supports the DISTINCT keyword and the ORDER BY clause. [#28778](https://github.com/StarRocks/starrocks/pull/28778) * Data in partitions can be automatically cooled down over time. (This feature is not supported for [list partitioning](https://docs.starrocks.io/docs/table_design/list_partitioning/).) [#29335](https://github.com/StarRocks/starrocks/pull/29335) [#29393](https://github.com/StarRocks/starrocks/pull/29393) ##### Improvements[​](#improvements-3 "Direct link to Improvements") * Supports implicit conversions for all compound predicates and for all expressions in the WHERE clause. You can enable or disable implicit conversions by using the [session variable](https://docs.starrocks.io/docs/sql-reference/System_variable/) `enable_strict_type`. The default value of this session variable is `false`. [#21870](https://github.com/StarRocks/starrocks/pull/21870) * Unifies the logic between FEs and BEs in converting strings to integers. [#29969](https://github.com/StarRocks/starrocks/pull/29969) ##### Bug Fixes[​](#bug-fixes-3 "Direct link to Bug Fixes") * If `enable_orc_late_materialization` is set to `true`, an unexpected result is returned when a Hive catalog is used to query STRUCT-type data in ORC files. [#27971](https://github.com/StarRocks/starrocks/pull/27971) * During data queries through Hive Catalog, if a partitioning column and an OR operator are specified in the WHERE clause, the query result is incorrect. [#28876](https://github.com/StarRocks/starrocks/pull/28876) * The values returned by the RESTful API action `show_data` for cloud-native tables are incorrect. [#29473](https://github.com/StarRocks/starrocks/pull/29473) * If the [shared-data cluster](https://docs.starrocks.io/docs/deployment/shared_data/azure/) stores data in Azure Blob Storage and a table is created, the FE fails to start after the cluster is rolled back to version 3.0. [#29433](https://github.com/StarRocks/starrocks/pull/29433) * A user has no permission when querying a table in the Iceberg catalog even if the user is granted permission on that table. [#29173](https://github.com/StarRocks/starrocks/pull/29173) * The `Default` field values returned by the [SHOW FULL COLUMNS](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_FULL_COLUMNS/) statement for columns of the [BITMAP](https://docs.starrocks.io/docs/sql-reference/data-types/other-data-types/BITMAP/) or [HLL](https://docs.starrocks.io/docs/sql-reference/data-types/other-data-types/HLL/) data type are incorrect. [#29510](https://github.com/StarRocks/starrocks/pull/29510) * Modifying the FE dynamic parameter `max_broker_load_job_concurrency` using the `ADMIN SET FRONTEND CONFIG` command does not take effect. * The FE may fail to start when a materialized view is being refreshed while its refresh strategy is being modified. [#29964](https://github.com/StarRocks/starrocks/pull/29964) [#29720](https://github.com/StarRocks/starrocks/pull/29720) * The error `unknown error` is returned when `select count(distinct(int+double)) from table_name` is executed. [#29691](https://github.com/StarRocks/starrocks/pull/29691) * After a Primary Key table is restored, metadata errors occur and cause metadata inconsistencies occur if a BE is restarted. [#30135](https://github.com/StarRocks/starrocks/pull/30135) #### 3.0.5[​](#305 "Direct link to 3.0.5") Release date: August 16, 2023 ##### New Features[​](#new-features-2 "Direct link to New Features") * Supports aggregate functions [COVAR\_SAMP](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/covar_samp/), [COVAR\_POP](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/covar_pop/), and [CORR](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/corr/). * Supports the following [window functions](https://docs.starrocks.io/docs/sql-reference/sql-functions/Window_function/): COVAR\_SAMP, COVAR\_POP, CORR, VARIANCE, VAR\_SAMP, STD, and STDDEV\_SAMP. ##### Improvements[​](#improvements-4 "Direct link to Improvements") * Added more prompts in the error message `xxx too many versions xxx`. [#28397](https://github.com/StarRocks/starrocks/pull/28397) * Dynamic partitioning further supports the partitioning unit to be year. [#28386](https://github.com/StarRocks/starrocks/pull/28386) * The partitioning field is case-insensitive when expression partitioning is used at table creation and [INSERT OVERWRITE is used to overwrite data in a specific partition](https://docs.starrocks.io/docs/table_design/expression_partitioning#load-data-into-partitions). [#28309](https://github.com/StarRocks/starrocks/pull/28309) ##### Bug Fixes[​](#bug-fixes-4 "Direct link to Bug Fixes") Fixed the following issues: * Incorrect table-level scan statistics in FE cause inaccurate metrics for table queries and loading. [#27779](https://github.com/StarRocks/starrocks/pull/27779) * The query result is not stable if the sort key is modified for a partitioned table. [#27850](https://github.com/StarRocks/starrocks/pull/27850) * The version number for a tablet is inconsistent between the BE and FE after data is restored. [#26518](https://github.com/StarRocks/starrocks/pull/26518/files) * If the bucket number is not specified when users create a Colocation table, the number will be inferred as 0, which causes failures in adding new partitions. [#27086](https://github.com/StarRocks/starrocks/pull/27086) * When the SELECT result set of INSERT INTO SELECT is empty, the load job status returned by SHOW LOAD is `CANCELED`. [#26913](https://github.com/StarRocks/starrocks/pull/26913) * BEs may crash when the input values of the sub\_bitmap function are not of the BITMAP type. [#27982](https://github.com/StarRocks/starrocks/pull/27982) * BEs may crash when the AUTO\_INCREMENT column is being updated. [#27199](https://github.com/StarRocks/starrocks/pull/27199) * Outer join and Anti join rewrite errors for materialized views. [#28028](https://github.com/StarRocks/starrocks/pull/28028) * Inaccurate estimation of average row size causes Primary Key partial updates to occupy excessively large memory. [#27485](https://github.com/StarRocks/starrocks/pull/27485) * Activating an inactive materialized view may cause a FE to crash. [#27959](https://github.com/StarRocks/starrocks/pull/27959) * Queries can not be rewritten to materialized views created based on external tables in a Hudi catalog. [#28023](https://github.com/StarRocks/starrocks/pull/28023) * The data of a Hive table can still be queried even after the table is dropped and the metadata cache is manually updated. [#28223](https://github.com/StarRocks/starrocks/pull/28223) * Manually refreshing an asynchronous materialized view via a synchronous call results in multiple INSERT OVERWRITE records in the `information_schema.task_runs` table. [#28060](https://github.com/StarRocks/starrocks/pull/28060) * FE memory leak caused by blocked LabelCleaner threads. [#28311](https://github.com/StarRocks/starrocks/pull/28311) #### 3.0.4[​](#304 "Direct link to 3.0.4") Release date: July 18, 2023 ##### New Feature[​](#new-feature "Direct link to New Feature") Queries can be rewritten even when the queries contain a different type of join than the materialized view. [#25099](https://github.com/StarRocks/starrocks/pull/25099) ##### Improvements[​](#improvements-5 "Direct link to Improvements") * Optimized the manual refreshing of asynchronous materialized views. Supports using the REFRESH MATERIALIZED VIEW WITH SYNC MODE syntax to synchronously invoke materialized view refresh tasks. [#25910](https://github.com/StarRocks/starrocks/pull/25910) * If the queried fields are not included in the output columns of a materialized view but are included in the predicate of the materialized view, the query can still be rewritten to benefit from the materialized view. [#23028](https://github.com/StarRocks/starrocks/issues/23028) * [When the SQL dialect (`sql_dialect`) is set to `trino`](https://docs.starrocks.io/docs/sql-reference/System_variable/), table aliases are not case-sensitive. [#26094](https://github.com/StarRocks/starrocks/pull/26094) [#25282](https://github.com/StarRocks/starrocks/pull/25282) * Added a new field `table_id` to the table `Information_schema.tables_config`. You can join the table `tables_config` with the table `be_tablets` on the column `table_id` in the database `Information_schema` to query the names of the database and table to which a tablet belongs. [#24061](https://github.com/StarRocks/starrocks/pull/24061) ##### Bug Fixes[​](#bug-fixes-5 "Direct link to Bug Fixes") Fixed the following issues: * If a query that contains the sum aggregate function is rewritten to directly obtain query results from a single-table materialized view, the values in sum() field may be incorrect due to type inference issues. [#25512](https://github.com/StarRocks/starrocks/pull/25512) * An error occurs when SHOW PROC is used to view information about tablets in a StarRocks shared-data cluster. * The INSERT operation hangs when the length of CHAR data in a STRUCT to be inserted exceeds the maximum length. [#25942](https://github.com/StarRocks/starrocks/pull/25942) * Some data rows queried fail to be returned for INSERT INTO SELECT with FULL JOIN. [#26603](https://github.com/StarRocks/starrocks/pull/26603) * An error `ERROR xxx: Unknown table property xxx` occurs when the ALTER TABLE statement is used to modify the table's property `default.storage_medium`. [#25870](https://github.com/StarRocks/starrocks/issues/25870) * An error occurs when Broker Load is used to load empty files. [#26212](https://github.com/StarRocks/starrocks/pull/26212) * Decommissioning a BE sometimes hangs. [#26509](https://github.com/StarRocks/starrocks/pull/26509) #### 3.0.3[​](#303 "Direct link to 3.0.3") Release date: June 28, 2023 ##### Improvements[​](#improvements-6 "Direct link to Improvements") * Metadata synchronization of StarRocks external tables has been changed to occur during data loading. [#24739](https://github.com/StarRocks/starrocks/pull/24739) * Users can specify partitions when they run INSERT OVERWRITE on tables whose partitions are automatically created. For more information, see [Automatic partitioning](https://docs.starrocks.io/docs/table_design/expression_partitioning/). [#25005](https://github.com/StarRocks/starrocks/pull/25005) * Optimized the error message reported when partitions are added to a non-partitioned table. [#25266](https://github.com/StarRocks/starrocks/pull/25266) ##### Bug Fixes[​](#bug-fixes-6 "Direct link to Bug Fixes") Fixed the following issues: * The min/max filter gets the wrong Parquet field when the Parquet file contains complex data types. [#23976](https://github.com/StarRocks/starrocks/pull/23976) * Load tasks are still queuing even when the related database or table has been dropped. [#24801](https://github.com/StarRocks/starrocks/pull/24801) * There is a low probability that an FE restart may cause BEs to crash. [#25037](https://github.com/StarRocks/starrocks/pull/25037) * Load and query jobs occasionally freeze when the variable `enable_profile` is set to `true`. [#25060](https://github.com/StarRocks/starrocks/pull/25060) * Inaccurate error message is displayed when INSERT OVERWRITE is executed on a cluster with less than three alive BEs. [#25314](https://github.com/StarRocks/starrocks/pull/25314) #### 3.0.2[​](#302 "Direct link to 3.0.2") Release date: June 13, 2023 ##### Improvements[​](#improvements-7 "Direct link to Improvements") * Predicates in a UNION query can be pushed down after the query is rewritten by an asynchronous materialized view. [#23312](https://github.com/StarRocks/starrocks/pull/23312) * Optimized the auto tablet distribution policy for tables. [#24543](https://github.com/StarRocks/starrocks/pull/24543) * Removed the dependency of NetworkTime on system clocks, which fixes incorrect NetworkTime caused by inconsistent system clocks across servers. [#24858](https://github.com/StarRocks/starrocks/pull/24858) ##### Bug Fixes[​](#bug-fixes-7 "Direct link to Bug Fixes") Fixed the following issues: * A schema change sometimes may be hung if data loading occurs simultaneously with the schema change. [#23456](https://github.com/StarRocks/starrocks/pull/23456) * Queries encounter an error when the session variable `pipeline_profile_level` is set to `0`. [#23873](https://github.com/StarRocks/starrocks/pull/23873) * CREATE TABLE encounters an error when `cloud_native_storage_type` is set to `S3`. * LDAP authentication succeeds even when no password is used. [#24862](https://github.com/StarRocks/starrocks/pull/24862) * CANCEL LOAD fails if the table involved in the load job does not exist. [#24922](https://github.com/StarRocks/starrocks/pull/24922) ##### Upgrade Notes[​](#upgrade-notes "Direct link to Upgrade Notes") If your system has a database named `starrocks`, change it to another name using ALTER DATABASE RENAME before the upgrade. This is because `starrocks` is the name of a default system database that stores privilege information. #### 3.0.1[​](#301 "Direct link to 3.0.1") Release date: June 1, 2023 ##### New Features[​](#new-features-3 "Direct link to New Features") * \[Preview] Supports spilling intermediate computation results of large operators to disks to reduce the memory consumption of large operators. For more information, see [Spill to disk](https://docs.starrocks.io/docs/administration/spill_to_disk/). * [Routine Load](https://docs.starrocks.io/docs/loading/RoutineLoad#load-avro-format-data) supports loading Avro data. * Supports [Microsoft Azure Storage](https://docs.starrocks.io/docs/integrations/authenticate_to_azure_storage/) (including Azure Blob Storage and Azure Data Lake Storage). ##### Improvements[​](#improvements-8 "Direct link to Improvements") * Shared-data clusters support using StarRocks external tables to synchronize data with another StarRocks cluster. * Added `load_tracking_logs` to [Information Schema](https://docs.starrocks.io/docs/sql-reference/information_schema/load_tracking_logs/) to record recent loading errors. * Ignores special characters in CREATE TABLE statements. [#23885](https://github.com/StarRocks/starrocks/pull/23885) ##### Bug Fixes[​](#bug-fixes-8 "Direct link to Bug Fixes") Fixed the following issues: * Information returned by SHOW CREATE TABLE is incorrect for Primary Key tables. [#24237](https://github.com/StarRocks/starrocks/issues/24237) * BEs may crash during a Routine Load job. [#20677](https://github.com/StarRocks/starrocks/issues/20677) * Null pointer exception (NPE) occurs if you specify unsupported properties when creating a partitioned table. [#21374](https://github.com/StarRocks/starrocks/issues/21374) * Information returned by SHOW TABLE STATUS is incomplete. [#24279](https://github.com/StarRocks/starrocks/issues/24279) ##### Upgrade Notes[​](#upgrade-notes-1 "Direct link to Upgrade Notes") If your system has a database named `starrocks`, change it to another name using ALTER DATABASE RENAME before the upgrade. This is because `starrocks` is the name of a default system database that stores privilege information. #### 3.0.0[​](#300 "Direct link to 3.0.0") Release date: April 28, 2023 ##### New Features[​](#new-features-4 "Direct link to New Features") ###### System architecture[​](#system-architecture "Direct link to System architecture") * **Decouple storage and compute.** StarRocks now supports data persistence into S3-compatible object storage, enhancing resource isolation, reducing storage costs, and making compute resources more scalable. Local disks are used as hot data cache for boosting query performance. The query performance of the new shared-data architecture is comparable to the classic architecture (shared-nothing) when local disk cache is hit. For more information, see [Deploy and use shared-data StarRocks](https://docs.starrocks.io/docs/deployment/shared_data/s3/). ###### Storage engine and data ingestion[​](#storage-engine-and-data-ingestion "Direct link to Storage engine and data ingestion") * The [AUTO\_INCREMENT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/auto_increment/) attribute is supported to provide globally unique IDs, which simplifies data management. * Automatic partitioning and partitioning expressions are supported, which makes partition creation easier to use and more flexible. * Primary Key tables support more complete [UPDATE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/UPDATE/) and [DELETE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/DELETE/#primary-key-tables) syntax, including the use of CTEs and references to multiple tables. * Added Load Profile for Broker Load and INSERT INTO jobs. You can view the details of a load job by querying the load profile. The usage is the same as [Analyze query profile](https://docs.starrocks.io/docs/administration/query_profile_overview/). ###### Data Lake Analytics[​](#data-lake-analytics "Direct link to Data Lake Analytics") * \[Preview] Supports Presto/Trino compatible dialect. Presto/Trino's SQL can be automatically rewritten into StarRocks' SQL pattern. For more information, see [the system variable](https://docs.starrocks.io/docs/reference/System_variable/) `sql_dialect`. * \[Preview] Supports [JDBC catalogs](https://docs.starrocks.io/docs/data_source/catalog/jdbc_catalog/). * Supports using [SET CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/Catalog/SET_CATALOG/) to manually switch between catalogs in the current session. ###### Privileges and security[​](#privileges-and-security "Direct link to Privileges and security") * Provides a new privilege system with full RBAC functionalities, supporting role inheritance and default roles. For more information, see [Overview of privileges](https://docs.starrocks.io/docs/administration/privilege_overview/). * Provides more privilege management objects and more fine-grained privileges. For more information, see [Privileges supported by StarRocks](https://docs.starrocks.io/docs/administration/privilege_item/). ###### Query engine[​](#query-engine "Direct link to Query engine") * Allows more queries on joined tables to benefit from the [query cache](https://docs.starrocks.io/docs/using_starrocks/query_cache/). For example, the query cache now supports Broadcast Join and Bucket Shuffle Join. * Supports [Global UDFs](https://docs.starrocks.io/docs/sql-reference/sql-functions/JAVA_UDF/). * Dynamic adaptive parallelism: StarRocks can automatically adjust the `pipeline_dop` parameter for query concurrency. ###### SQL reference[​](#sql-reference "Direct link to SQL reference") * Added the following privilege-related SQL statements: [SET DEFAULT ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SET_DEFAULT_ROLE/), [SET ROLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SET_ROLE/), [SHOW ROLES](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_ROLES/), and [SHOW USERS](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/SHOW_USERS/). * Added the following semi-structured data analysis functions: [map\_apply](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_apply/), [map\_from\_arrays](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_from_arrays/). * [array\_agg](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_agg/) supports ORDER BY. * Window functions [lead](https://docs.starrocks.io/docs/sql-reference/sql-functions/Window_function#lead) and [lag](https://docs.starrocks.io/docs/sql-reference/sql-functions/Window_function#lag) support IGNORE NULLS. * Added string functions [replace](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/replace/), [hex\_decode\_binary](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/hex_decode_binary/), and [hex\_decode\_string()](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/hex_decode_string/). * Added encryption functions [base64\_decode\_binary](https://docs.starrocks.io/docs/sql-reference/sql-functions/crytographic-functions/base64_decode_binary/) and [base64\_decode\_string](https://docs.starrocks.io/docs/sql-reference/sql-functions/crytographic-functions/base64_decode_string/). * Added math functions [sinh](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/sinh/), [cosh](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/cosh/), and [tanh](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/tanh/). * Added utility function [current\_role](https://docs.starrocks.io/docs/sql-reference/sql-functions/utility-functions/current_role/). ##### Improvements[​](#improvements-9 "Direct link to Improvements") ###### Deployment[​](#deployment "Direct link to Deployment") * Updated Docker image and the related [Docker deployment document](https://docs.starrocks.io/docs/quick_start/deploy_with_docker/) for version 3.0. [#20623](https://github.com/StarRocks/starrocks/pull/20623) [#21021](https://github.com/StarRocks/starrocks/pull/21021) ###### Storage engine and data ingestion[​](#storage-engine-and-data-ingestion-1 "Direct link to Storage engine and data ingestion") * Supports more CSV parameters for data ingestion, including SKIP\_HEADER, TRIM\_SPACE, ENCLOSE, and ESCAPE. See [STREAM LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/STREAM_LOAD/), [BROKER LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/BROKER_LOAD/), and [ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/routine_load/CREATE_ROUTINE_LOAD/). * The primary key and sort key are decoupled in [Primary Key tables](https://docs.starrocks.io/docs/table_design/table_types/primary_key_table/). The sort key can be separately specified in `ORDER BY` when you create a table. * Optimized the memory usage of data ingestion into Primary Key tables in scenarios such as large-volume ingestion, partial updates, and persistent primary indexes. * Supports creating asynchronous INSERT tasks. For more information, see [INSERT](https://docs.starrocks.io/docs/loading/InsertInto#load-data-asynchronously-using-insert) and [SUBMIT TASK](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/ETL/SUBMIT_TASK/). [#20609](https://github.com/StarRocks/starrocks/issues/20609) ###### Materialized view[​](#materialized-view "Direct link to Materialized view") * Optimized the rewriting capabilities of [materialized views](https://docs.starrocks.io/docs/using_starrocks/Materialized_view/), including: * Supports rewrite of View Delta Join, Outer Join, and Cross Join. * Optimized SQL rewrite of Union with partition. * Improved materialized view building capabilities: supporting CTE, select \*, and Union. * Optimized the information returned by [SHOW MATERIALIZED VIEWS](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/SHOW_MATERIALIZED_VIEW/). * Supports adding MV partitions in batches, which improves the efficiency of partition addition during materialized view building. [#21167](https://github.com/StarRocks/starrocks/pull/21167) ###### Query engine[​](#query-engine-1 "Direct link to Query engine") * All operators are supported in the pipeline engine. Non-pipeline code will be removed in later versions. * Improved [Big Query Positioning](https://docs.starrocks.io/docs/administration/monitor_manage_big_queries/) and added big query log. [SHOW PROCESSLIST](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROCESSLIST/) supports viewing CPU and memory information. * Optimized Outer Join Reorder. * Optimized error messages in the SQL parsing stage, providing more accurate error positioning and clearer error messages. ###### Data Lake Analytics[​](#data-lake-analytics-1 "Direct link to Data Lake Analytics") * Optimized metadata statistics collection. * Supports using [SHOW CREATE TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/SHOW_CREATE_TABLE/) to view the creation statements of the tables that are managed by an external catalog and are stored in Apache Hive™, Apache Iceberg, Apache Hudi, or Delta Lake. ##### Bug Fixes[​](#bug-fixes-9 "Direct link to Bug Fixes") * Some URLs in the license header of StarRocks' source file cannot be accessed. [#2224](https://github.com/StarRocks/starrocks/issues/2224) * An unknown error is returned during SELECT queries. [#19731](https://github.com/StarRocks/starrocks/issues/19731) * Supports SHOW/SET CHARACTER. [#17480](https://github.com/StarRocks/starrocks/issues/17480) * When the loaded data exceeds the field length supported by StarRocks, the error message returned is not correct. [#14](https://github.com/StarRocks/DataX/issues/14) * Supports `show full fields from 'table'`. [#17233](https://github.com/StarRocks/starrocks/issues/17233) * Partition pruning causes MV rewrites to fail. [#14641](https://github.com/StarRocks/starrocks/issues/14641) * MV rewrite fails when the CREATE MATERIALIZED VIEW statement contains `count(distinct)` and `count(distinct)` is applied to the DISTRIBUTED BY column. [#16558](https://github.com/StarRocks/starrocks/issues/16558) * FEs fail to start when a VARCHAR column is used as the partitioning column of a materialized view. [#19366](https://github.com/StarRocks/starrocks/issues/19366) * Window functions [LEAD](https://docs.starrocks.io/docs/sql-reference/sql-functions/Window_function#lead) and [LAG](https://docs.starrocks.io/docs/sql-reference/sql-functions/Window_function#lag) incorrectly handle IGNORE NULLS. [#21001](https://github.com/StarRocks/starrocks/pull/21001) * Adding temporary partitions conflicts with automatic partition creation. [#21222](https://github.com/StarRocks/starrocks/issues/21222) ##### Behavior Change[​](#behavior-change-2 "Direct link to Behavior Change") * The new role-based access control (RBAC) system supports the previous privileges and roles. However, the syntax of related statements such as [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT/) and [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE/) is changed. * Renamed SHOW MATERIALIZED VIEW as [SHOW MATERIALIZED VIEWS](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/SHOW_MATERIALIZED_VIEW/). * Added the following [Reserved keywords](https://docs.starrocks.io/docs/sql-reference/sql-statements/keywords/): AUTO\_INCREMENT, CURRENT\_ROLE, DEFERRED, ENCLOSE, ESCAPE, IMMEDIATE, PRIVILEGES, SKIP\_HEADER, TRIM\_SPACE, VARBINARY. ##### Upgrade Notes[​](#upgrade-notes-2 "Direct link to Upgrade Notes") You can upgrade from v2.5 to v3.0 or downgrade from v3.0 to v2.5. > In theory, an upgrade from a version earlier than v2.5 is also supported. To ensure system availability, we recommend that you first upgrade your cluster to v2.5 and then to v3.0. Take note of the following points when you perform a downgrade from v3.0 to v2.5. ###### BDBJE[​](#bdbje "Direct link to BDBJE") StarRocks upgrades the BDB library in v3.0. However, BDBJE cannot be rolled back. You must use BDB library of v3.0 after a downgrade. Perform the following steps: 1. After you replace the FE package with a v2.5 package, copy `fe/lib/starrocks-bdb-je-18.3.13.jar` of v3.0 to the `fe/lib` directory of v2.5. 2. Delete `fe/lib/je-7.*.jar`. ###### Privilege system[​](#privilege-system "Direct link to Privilege system") The new RBAC privilege system is used by default after you upgrade to v3.0. You can only downgrade to v2.5. After a downgrade, run [ALTER SYSTEM CREATE IMAGE](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM/) to create a new image and wait for the new image to be synchronized to all follower FEs. If you do not run this command, some of the downgrade operations may fail. This command is supported from 2.5.3 and later. For details about the differences between the privilege system of v2.5 and v3.0, see "Upgrade notes" in [Privileges supported by StarRocks](https://docs.starrocks.io/docs/administration/privilege_item/). --- ## Release 3.1 ### StarRocks version 3.1 #### 3.1.17[​](#3117 "Direct link to 3.1.17") Release Date: January 3, 2025 ##### Bug Fixes[​](#bug-fixes "Direct link to Bug Fixes") Fixed the following issues: * Cross-cluster Data Migration Tool caused the Follower FE to crash during data synchronization and commit, due to not accounting for the deletion of partitions in the target cluster. [#54061](https://github.com/StarRocks/starrocks/pull/54061) * BE in the target cluster might crash when synchronizing tables with DELETE operations using Cross-cluster Data Migration Tool. [#54081](https://github.com/StarRocks/starrocks/pull/54081) * A bug in the BDBJE handshake mechanism where Leader FE would reject reconnection attempts from Follower FE when connection is being re-established, causing Follower FE nodes to exit. [#50412](https://github.com/StarRocks/starrocks/pull/50412) * Duplicate memory statistics in FE leads to excessive memory usage. [#53055](https://github.com/StarRocks/starrocks/pull/53055) * The statuses of asynchronous materialized view refresh tasks are inconsistent across multiple FE nodes, which lead to inaccurate states of the materialized view during queries. [#54236](https://github.com/StarRocks/starrocks/pull/54236) #### 3.1.16[​](#3116 "Direct link to 3.1.16") Release date: December 16, 2024 ##### Improvements[​](#improvements "Direct link to Improvements") * Optimized table-related statistics. [#50316](https://github.com/StarRocks/starrocks/pull/50316) ##### Bug Fixes[​](#bug-fixes-1 "Direct link to Bug Fixes") Fixed the following issues: * Insufficient granularity in error code handling for disk full scenarios caused the BE to mistakenly identify disk errors and delete data. [#51411](https://github.com/StarRocks/starrocks/pull/51411) * Stream Load failures when submitted using HTTP 1.0. [#53010](https://github.com/StarRocks/starrocks/pull/53010) [#53008](https://github.com/StarRocks/starrocks/pull/53008) * Routine Load tasks were canceled due to expired transactions (now tasks are canceled only if the database or table no longer exists and paused when transactions expired). [#50334](https://github.com/StarRocks/starrocks/pull/50334) * Unloading data using `EXPORT` with Broker to `file://` resulted in a file rename error, causing the export to fail. [#52544](https://github.com/StarRocks/starrocks/pull/52544) * If the join condition in an equal-join is an expression based on a low-cardinality column, the system may incorrectly push down a Runtime Filter predicate, leading to a BE crash. [#50690](https://github.com/StarRocks/starrocks/pull/50690) #### 3.1.15[​](#3115 "Direct link to 3.1.15") Release date: September 4, 2024 ##### Bug Fixes[​](#bug-fixes-2 "Direct link to Bug Fixes") Fixed the following issues: * During query rewrite with asynchronous materialized views, `count(*)` on certain tables returns NULL. [#49288](https://github.com/StarRocks/starrocks/pull/49288) * `partition_linve_nubmer` does not take effect. [#49213](https://github.com/StarRocks/starrocks/pull/49213) * FE throws a tablet exception: BE disk offline, and cannot migrate tablets. [#47833](https://github.com/StarRocks/starrocks/pull/47833) #### 3.1.14[​](#3114 "Direct link to 3.1.14") Release date: July 29, 2024 ##### Improvements[​](#improvements-1 "Direct link to Improvements") * Stream Load now supports using `\t` and `\n` as row and column delimiters. Users do not need to convert them to their hexadecimal ASCII codes. [#47302](https://github.com/StarRocks/starrocks/pull/47302) ##### Bug Fixes[​](#bug-fixes-3 "Direct link to Bug Fixes") Fixed the following issues: * Frequent INSERT and UPDATE operations on Primary Key tables may cause write and query delays in the database. [#47838](https://github.com/StarRocks/starrocks/pull/47838) * When a Primary Key table encounters data persistence failures, the persistent index may fail to capture the error, leading to data loss and reporting the error "Insert found duplicate key". [#48045](https://github.com/StarRocks/starrocks/pull/48045) * Materialized views may report insufficient permissions when refreshed. [#47561](https://github.com/StarRocks/starrocks/pull/47561) * Materialized view reports the error "For input string" when refreshed. [#46131](https://github.com/StarRocks/starrocks/pull/46131) * During materialized view refresh, the lock is held excessively long, causing the Leader FE to be restarted by the deadlock detection script. [#48256](https://github.com/StarRocks/starrocks/pull/48256) * Queries against views with the IN clause in its definition may return inaccurate results. [#47484](https://github.com/StarRocks/starrocks/pull/47484) * Global Runtime Filter causes incorrect results. [#48496](https://github.com/StarRocks/starrocks/pull/48496) * MySQL protocol `COM_CHANGE_USER` does not support `conn_attr`. [#47796](https://github.com/StarRocks/starrocks/pull/47796) ##### Behavior Changes[​](#behavior-changes "Direct link to Behavior Changes") * When users create a non-partitioned table without specifying the bucket number, the minimum bucket number the system sets for the table is `16` (instead of `2` based on the formula `2*BE or CN count`). If users want to set a smaller bucket number when creating a small table, they must set it explicitly. [#47005](https://github.com/StarRocks/starrocks/pull/47005) #### 3.1.13[​](#3113 "Direct link to 3.1.13") Release date: June 26, 2024 ##### Improvements[​](#improvements-2 "Direct link to Improvements") * The Broker process supports access to Tencent Cloud COS Posix buckets. Users can load data from COS Posix buckets using Broker Load or unload data to COS Posix buckets using the SELECT INTO OUTFILE statement. [#46597](https://github.com/StarRocks/starrocks/pull/46597) * Supports viewing comments of Hive tables in Hive Catalogs using SHOW CREATE TABLE. [#37686](https://github.com/StarRocks/starrocks/pull/37686) * Optimized the evaluation time of Conjunct in WHERE clauses, such as multiple LIKE clauses on the same column or CASE WHEN expressions. [#46914](https://github.com/StarRocks/starrocks/pull/46914) ##### Bug Fixes[​](#bug-fixes-4 "Direct link to Bug Fixes") Fixed the following issues: * DELETE statements fail in shared-data clusters if there are excessive number of partitions to be deleted. [#46229](https://github.com/StarRocks/starrocks/pull/46229) #### 3.1.12[​](#3112 "Direct link to 3.1.12") Release date: May 30, 2024 ##### New Features[​](#new-features "Direct link to New Features") * Flink connector supports reading complex data types ARRAY, MAP, and STRUCT from StarRocks. [#42932](https://github.com/StarRocks/starrocks/pull/42932) [#347](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/347) ##### Improvements[​](#improvements-3 "Direct link to Improvements") * Previously, when BE failed to communicate with FE via RPC, FE would return a generic error message: `call frontend service failed reason=xxx`, making it unclear what the specific issue was. The error messages are now optimized to include specific reasons, such as timeout or server busy. [#44153](https://github.com/StarRocks/starrocks/pull/44153) * Improved error messages to indicate specific issues during data loading, such as the number of error data rows exceeding limits, mismatched column numbers, invalid column names, and no data in any partition. ##### Security[​](#security "Direct link to Security") * Upgraded Kafka client dependency to v3.4.0 to fix the CVE-2023-25194 security issue. [#45382](https://github.com/StarRocks/starrocks/pull/45382) ##### Bug Fixes[​](#bug-fixes-5 "Direct link to Bug Fixes") Fixed the following issues: * If a materialized view definition includes multiple self-joins of the same table and incremental refreshes by partitions based on that table, incorrect results would occur due to wrong partition selection. [#45936](https://github.com/StarRocks/starrocks/pull/45936) * FEs crash when a Bitmap index is created in a materialized view in shared-data clusters. [#45665](https://github.com/StarRocks/starrocks/pull/45665) * BEs crash due to null pointer issues when FE follower is connected via ODBC and CREATE TABLE is executed. [#45043](https://github.com/StarRocks/starrocks/pull/45043) * Querying `information_schema.task_runs` fails frequently when many asynchronous tasks exist. [#45520](https://github.com/StarRocks/starrocks/pull/45520) * When a SQL statement contains multiple COUNT DISTINCT and includes LIMIT, LIMIT is wrongly processed, resulting in inconsistent data returned each time the statement is executed. [#44749](https://github.com/StarRocks/starrocks/pull/44749) * Queries with ORDER BY LIMIT clauses on Duplicate Key tables and Aggregate tables produce incorrect results. [#45037](https://github.com/StarRocks/starrocks/pull/45037) #### 3.1.11[​](#3111 "Direct link to 3.1.11") Release date: April 28, 2024 ##### Behavior Changes[​](#behavior-changes-1 "Direct link to Behavior Changes") * Users are not allowed to drop views in the system database `information_schema` using DROP TABLE. [#43556](https://github.com/StarRocks/starrocks/pull/43556) * Users are not allowed to specify duplicate keys in the ORDER BY clause when creating a Primary Key table. [#43374](https://github.com/StarRocks/starrocks/pull/43374) ##### Improvements[​](#improvements-4 "Direct link to Improvements") * Queries on Parquet-formatted Iceberg v2 tables support equality deletes. ##### Bug Fixes[​](#bug-fixes-6 "Direct link to Bug Fixes") Fixed the following issues: * When a user queries data from an external table in an external catalog, access to this table is denied even when the user has the SELECT privilege on this table. SHOW GRANTS also shows that the user has this privilege. [#44061](https://github.com/StarRocks/starrocks/pull/44061) * `str_to_map` may cause BEs to crash. [#43930](https://github.com/StarRocks/starrocks/pull/43930) * When a Routine Load job is going on, running `show proc '/routine_loads'` is stuck due to deadlock. [#44249](https://github.com/StarRocks/starrocks/pull/44249) * Persistent Index of Primary Key tables may cause BEs to crash due to issues in concurrency control. [#43720](https://github.com/StarRocks/starrocks/pull/43720) * The `pending_task_run_count` displayed on the page of `leaderFE_IP:8030` is incorrect. The displayed number is the sum of Pending and Running tasks, not Pending tasks. In addition, the information of the metric `refresh_pending` cannot be displayed using `followerFE_IP:8030`. [#43052](https://github.com/StarRocks/starrocks/pull/43052) * Querying `information_schema.task_runs` fails frequently. [#43052](https://github.com/StarRocks/starrocks/pull/43052) * Some SQL queries that contain CTEs may encounter the `Invalid plan: PhysicalTopNOperator` error. [#44185](https://github.com/StarRocks/starrocks/pull/44185) #### 3.1.10 (Yanked)[​](#3110-yanked "Direct link to 3.1.10 (Yanked)") tip This version has been taken offline due to privilege issues in querying external tables in external catalogs such as Hive and Iceberg. * **Problem**: When a user queries data from an external table in an external catalog, access to this table is denied even when the user has the SELECT privilege on this table. SHOW GRANTS also shows that the user has this privilege. * **Impact scope**: This problem only affects queries on external tables in external catalogs. Other queries are not affected. * **Temporary workaround**: The query succeeds after the SELECT privilege on this table is granted to the user again. But `SHOW GRANTS` will return duplicate privilege entries. After an upgrade to v3.1.11, users can run `REVOKE` to remove one of the privilege entries. Release date: March 29, 2024 ##### New Features[​](#new-features-1 "Direct link to New Features") * Primary Key tables support Size-tiered Compaction. [#42474](https://github.com/StarRocks/starrocks/pull/42474) * Added a pattern-matching function `regexp_extract_all`. [#42178](https://github.com/StarRocks/starrocks/pull/42178) ##### Behavior Changes[​](#behavior-changes-2 "Direct link to Behavior Changes") * When null values in JSON data are evaluated based on the `IS NULL` operator, they are considered NULL values following SQL language. For example, `true` is returned for `SELECT parse_json('{"a": null}') -> 'a' IS NULL` (before this behavior change, `false` is returned). [#42815](https://github.com/StarRocks/starrocks/pull/42815) ##### Improvements[​](#improvements-5 "Direct link to Improvements") * When Broker Load is used to load data from ORC files that contain TIMESTAMP-type data, StarRocks supports retaining microseconds in the timestamps when converting the timestamps to match its own DATETIME data type. [#42348](https://github.com/StarRocks/starrocks/pull/42348) ##### Bug Fixes[​](#bug-fixes-7 "Direct link to Bug Fixes") Fixed the following issues: * In shared-data mode, the garbage collection and thread eviction mechanisms for handling persistent indexes created on Primary Key tables cannot take effect on CN nodes. As a result, obsolete data cannot be deleted. [#42241](https://github.com/StarRocks/starrocks/pull/42241) * When users query ORC files by using Hive catalogs, the query results may be incorrect because StarRocks used to read ORC files from Hive based on mapping by position. To resolve this issue, users can set the session variable `orc_use_column_names` to `true`, which specifies to read ORC files from Hive based on mapping by column name. [#42905](https://github.com/StarRocks/starrocks/pull/42905) * When LDAP authentication for the AD system is adopted, logins without passwords are allowed. [#42476](https://github.com/StarRocks/starrocks/pull/42476) * When disk device names end with digits, the values of monitoring metrics remain 0s because the disk device names may be invalid after such digits are removed. [#42741](https://github.com/StarRocks/starrocks/pull/42741) #### 3.1.9[​](#319 "Direct link to 3.1.9") Release date: March 8, 2024 ##### New Features[​](#new-features-2 "Direct link to New Features") * Cloud-native Primary Key tables in shared-data clusters support Size-tiered Compaction to reduce write I/O amplification for the loading of a large number of small-sized files. [#41610](https://github.com/StarRocks/starrocks/pull/41610) * Added the view `information_schema.partitions_meta`, which records detailed metadata of partitions. [#41101](https://github.com/StarRocks/starrocks/pull/41101) * Added the view `sys.fe_memory_usage`, which records the memory usage for StarRocks. [#41083](https://github.com/StarRocks/starrocks/pull/41083) ##### Behavior Changes[​](#behavior-changes-3 "Direct link to Behavior Changes") * The logic of dynamic partitioning is changed. Now partition columns of the DATE type do not support hour-level data. Note that partition columns of the DATETIME type still support hour-level data. [#40328](https://github.com/StarRocks/starrocks/pull/40328) * The user who can refresh materialized views is changed from the `root` user to the user who creates the materialized views. This change does not affect existing materialized views. [#40698](https://github.com/StarRocks/starrocks/pull/40698) * By default, when comparing columns of constant and string types, StarRocks compares them as strings. Users can use the session variable `cbo_eq_base_type` to adjust the default rule used for the comparison. For example, users can set `cbo_eq_base_type` to `decimal`, and StarRocks then compares the columns as numeric values. [#41712](https://github.com/StarRocks/starrocks/pull/41712) ##### Improvements[​](#improvements-6 "Direct link to Improvements") * StarRocks supports using the parameter `s3_compatible_fs_list` to specify which S3-compatible object storage can be accessed via AWS SDK, and supports using the parameter `fallback_to_hadoop_fs_list` to specify non-S3-compatible object storage that require access via HDFS Schema (this method necessitates the use of vendor-provided JAR packages). [#41612](https://github.com/StarRocks/starrocks/pull/41612) * The compatibility with Trino's SQL statement syntax is optimized to support converting the following functions of Trino: `current_catalog`, `current_schema`, `to_char`, `from_hex`, `to_date`, `to_timestamp`, and `index`. [#41505](https://github.com/StarRocks/starrocks/pull/41505) [#41270](https://github.com/StarRocks/starrocks/pull/41270) [#40838](https://github.com/StarRocks/starrocks/pull/40838) * A new session variable `cbo_materialized_view_rewrite_related_mvs_limit` is added to control the maximum number of candidate materialized views allowed during query planning. The default value of this session variable is `64`. This session variable helps mitigate the excessive resource consumption caused by a large number of candidate materialized views for a query during the query planning. [#39829](https://github.com/StarRocks/starrocks/pull/39829) * The `agg_type` of BITMAP-type columns in an Aggregate table can be set to `replace_if_not_null` to support updates only to a few columns of the table. [#42102](https://github.com/StarRocks/starrocks/pull/42102) * The session variable `cbo_eq_base_type` is optimized to support specifying the implicit conversion rule applied to the comparison of data that contains both string and numeric data types. By default, such data is compared as strings. [#40619](https://github.com/StarRocks/starrocks/pull/41712) * More DATE-type data (for example, "%Y-%m-%e %H:%i") can be recognized to better support partition expressions for Iceberg tables. [#40474](https://github.com/StarRocks/starrocks/pull/40474) * The JDBC connector supports the TIME data type. [#31940](https://github.com/StarRocks/starrocks/pull/31940) * The `path` parameter in the SQL statement for creating a file external table supports wildcards (`*`). However, like the `DATA INFILE` parameter in the SQL statement for creating a Broker Load job, the `path` parameter supports using wildcards (`*`) to match at most one level of directory or file. [#40844](https://github.com/StarRocks/starrocks/pull/40844) * A new internal SQL log file is added to record log data related to statistics and materialized views. [#40682](https://github.com/StarRocks/starrocks/pull/40682) ##### Bug Fixes[​](#bug-fixes-8 "Direct link to Bug Fixes") Fixed the following issues: * "Analyze Error" is thrown if inconsistent letter cases are assigned to the names or aliases of tables or views queried in the creation of a Hive view. [#40921](https://github.com/StarRocks/starrocks/pull/40921) * I/O usage reaches the upper limit if persistent indexes are created on Primary Key tables. [#39959](https://github.com/StarRocks/starrocks/pull/39959) * In shared-data clusters, the primary key index directory is deleted every 5 hours. [#40745](https://github.com/StarRocks/starrocks/pull/40745) * After a table for which list partitioning is enabled is truncated or its partitions are truncated, queries based on the partitioning keys of the table return no data. [#40495](https://github.com/StarRocks/starrocks/pull/40495) * After users execute ALTER TABLE COMPACT by hand, the memory usage statistics for compaction operations are abnormal. [#41150](https://github.com/StarRocks/starrocks/pull/41150) * During data migration between clusters, if only some columns are updated in column mode, the destination cluster may crash. [#40692](https://github.com/StarRocks/starrocks/pull/40692) * The SQL blacklist may not take effect if the submitted SQL statement contains multiple spaces or newline characters. [#40457](https://github.com/StarRocks/starrocks/pull/40457) #### 3.1.8[​](#318 "Direct link to 3.1.8") Release date: February 5, 2024 ##### New Features[​](#new-features-3 "Direct link to New Features") * StarRocks Community provides the StarRocks Cross-cluster Data Migration Tool, which supports migrating data from a shared-nothing cluster to either another shared-nothing cluster or a shared-data cluster. * Supports creating synchronous materialized views with the WHERE clause specified. * Added metrics that show memory usage of the data cache to MemTracker. [#39600](https://github.com/StarRocks/starrocks/pull/39600) ##### Parameter Change[​](#parameter-change "Direct link to Parameter Change") * Added a BE configuration item, `lake_pk_compaction_max_input_rowsets`, which controls the maximum number of input rowsets allowed in a Primary Key table compaction task in a shared-data StarRocks cluster. This helps optimize resource consumption for compaction tasks. [#39611](https://github.com/StarRocks/starrocks/pull/39611) ##### Improvements[​](#improvements-7 "Direct link to Improvements") * Supports ORDER BY and INDEX clauses in CTAS statements. [#38886](https://github.com/StarRocks/starrocks/pull/38886) * Supports equality deletes on ORC-formatted Iceberg v2 tables. [#37419](https://github.com/StarRocks/starrocks/pull/37419) * Supports setting the `datacache.partition_duration` property for cloud-native tables created with the list partitioning strategy. This property controls the validity period of the data cache and can be dynamically configured. [#35681](https://github.com/StarRocks/starrocks/pull/35681) [#38509](https://github.com/StarRocks/starrocks/pull/38509) * Optimized the BE configuration item `update_compaction_per_tablet_min_interval_seconds`. This parameter is originally used only to control the frequency of compaction tasks on Primary Key tables. After the optimization, it can also be used to control the frequency of major compaction tasks on Primary Key table indexes. [#39640](https://github.com/StarRocks/starrocks/pull/39640) * Parquet Reader supports converting INT32-type data in Parquet-formatted data to DATETIME-type data and storing the resulting data to StarRocks. [#39808](https://github.com/StarRocks/starrocks/pull/39808) ##### Bug Fixes[​](#bug-fixes-9 "Direct link to Bug Fixes") Fixed the following issues: * Using NaN (Not a Number) columns as ORDER BY columns may cause BEs to crash. [#30759](https://github.com/StarRocks/starrocks/pull/30759) * Failure to update primary key indexes may cause the error "get\_applied\_rowsets failed". [#27488](https://github.com/StarRocks/starrocks/pull/27488) * The resources occupied by compaction\_state\_cache are not recycled after compaction task failures. [#38499](https://github.com/StarRocks/starrocks/pull/38499) * If partition columns in external tables contain null values, queries against those tables will cause BEs to crash. [#38888](https://github.com/StarRocks/starrocks/pull/38888) * After a table is dropped and then re-created with the same table name, refreshing asynchronous materialized views created on that table fails. [#38008](https://github.com/StarRocks/starrocks/pull/38008) * Refreshing asynchronous materialized views created on empty Iceberg tables fail. [#24068](https://starrocks.atlassian.net/browse/SR-24068) #### 3.1.7[​](#317 "Direct link to 3.1.7") Release date: January 12, 2024 ##### New Features[​](#new-features-4 "Direct link to New Features") * Added a new function, `unnest_bitmap`. [#38136](https://github.com/StarRocks/starrocks/pull/38136) * Supports conditional updates for [Broker Load](https://docs.starrocks.io/docs/3.1/sql-reference/sql-statements/data-manipulation/BROKER_LOAD/#opt_properties). [#37400](https://github.com/StarRocks/starrocks/pull/37400) ##### Behavior Change[​](#behavior-change "Direct link to Behavior Change") * Added the session variable `enable_materialized_view_for_insert`, which controls whether materialized views rewrite the queries in INSERT INTO SELECT statements. The default value is `false`. [#37505](https://github.com/StarRocks/starrocks/pull/37505) * The FE dynamic parameter `enable_new_publish_mechanism` is changed to a static parameter. You must restart the FE after you modify the parameter settings. [#35338](https://github.com/StarRocks/starrocks/pull/35338) * Added the session variable `enable_strict_order_by`. When this variable is set to the default value `TRUE`, an error is reported for such a query pattern: Duplicate alias is used in different expressions of the query and this alias is also a sorting field in ORDER BY, for example, `select distinct t1.* from tbl1 t1 order by t1.k1;`. The logic is the same as that in v2.3 and earlier. When this variable is set to `FALSE`, a loose deduplication mechanism is used, which processes such queries as valid SQL queries. [#37910](https://github.com/StarRocks/starrocks/pull/37910) ##### Parameter Change[​](#parameter-change-1 "Direct link to Parameter Change") * Added the FE configuration item `routine_load_unstable_threshold_second`. [#36222](https://github.com/StarRocks/starrocks/pull/36222) * Added the FE configuration item `http_worker_threads_num`, which specifies the number of threads for HTTP server to deal with HTTP requests. The default value is `0`. If the value for this parameter is set to a negative value or `0`, the actual thread number is twice the number of CPU cores. [#37530](https://github.com/StarRocks/starrocks/pull/37530) * Added the BE configuration item `pindex_major_compaction_limit_per_disk` to configure the maximum concurrency of compaction on a disk. This addresses the issue of uneven I/O across disks due to compaction. This issue can cause excessively high I/O for certain disks. The default value is `1`. [#36681](https://github.com/StarRocks/starrocks/pull/36681) * Added session variables `transaction_read_only` and `tx_read_only` to specify the transaction access mode, which are compatible with MySQL versions 5.7.20 and above. [#37249](https://github.com/StarRocks/starrocks/pull/37249) * Added the FE configuration item `default_mv_refresh_immediate`, which specifies whether to immediately refresh the materialized view after the materialized view is created. The default value is `true`. [#37093](https://github.com/StarRocks/starrocks/pull/37093) * Added a new BE configuration item `lake_enable_vertical_compaction_fill_data_cache`, which specifies whether to allow compaction tasks to cache data on local disks in a shared-data cluster. The default value is `false`. [#37296](https://github.com/StarRocks/starrocks/pull/37296) ##### Improvements[​](#improvements-8 "Direct link to Improvements") * INSERT INTO FILE() SELECT FROM supports reading BINARY-type data from tables and exporting the data to Parquet-formatted files in remote storage. [#36797](https://github.com/StarRocks/starrocks/pull/36797) * Asynchronous materialized views support dynamically setting the `datacache.partition_duration` property, which controls the validity period of the hot data in the data cache. [#35681](https://github.com/StarRocks/starrocks/pull/35681) * Wen using JDK, the default GC algorithm is G1. [#37386](https://github.com/StarRocks/starrocks/pull/37386) * The `date_trunc`, `adddate`, and `time_slice` functions support setting the `interval` parameter to values that are accurate to the millisecond and microsecond. [#36386](https://github.com/StarRocks/starrocks/pull/36386) * When the string on the right side of the LIKE operator within the WHERE clause does not include `%` or `_`, the LIKE operator is converted into the `=` operator. [#37515](https://github.com/StarRocks/starrocks/pull/37515) * A new field `LatestSourcePosition` is added to the return result of [SHOW ROUTINE LOAD](https://docs.starrocks.io/zh/docs/3.1/sql-reference/sql-statements/data-manipulation/SHOW_ROUTINE_LOAD/) to record the position of the latest message in each partition of the Kafka topic, helping check the latencies of data loading. [#38298](https://github.com/StarRocks/starrocks/pull/38298) * Added a new resource group property, `spill_mem_limit_threshold`, to control the memory usage threshold (percentage) at which a resource group triggers the spilling of intermediate results when the system variable `spill_mode` is set to `auto`. The valid range is (0, 1). The default value is `1`, indicating the threshold does not take effect. [#37707](https://github.com/StarRocks/starrocks/pull/37707) * The result returned by the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/3.1/sql-reference/sql-statements/data-manipulation/SHOW_ROUTINE_LOAD/) statement now includes the timestamps of consumption messages from each partition. [#36222](https://github.com/StarRocks/starrocks/pull/36222) * The scheduling policy for Routine Load is optimized, so that slow tasks do not block the execution of the other normal tasks. [#37638](https://github.com/StarRocks/starrocks/pull/37638) ##### Bug Fixes[​](#bug-fixes-10 "Direct link to Bug Fixes") Fixed the following issues: * The execution of [ANALYZE TABLE](https://docs.starrocks.io/docs/3.1/sql-reference/sql-statements/data-definition/ANALYZE_TABLE/) gets stuck occasionally. [#36836](https://github.com/StarRocks/starrocks/pull/36836) * The memory consumption by PageCache exceeds the threshold specified by the BE dynamic parameter `storage_page_cache_limit` in certain circumstances. [#37740](https://github.com/StarRocks/starrocks/pull/37740) * Hive metadata in [Hive catalogs](https://docs.starrocks.io/docs/3.1/data_source/catalog/hive_catalog/) is not automatically refreshed when new fields are added to Hive tables. [#37668](https://github.com/StarRocks/starrocks/pull/37668) * In some cases, `bitmap_to_string` may return incorrect results due to data type overflow. [#37405](https://github.com/StarRocks/starrocks/pull/37405) * Executing the DELETE statement on an empty table returns "ERROR 1064 (HY000): Index: 0, Size: 0". [#37461](https://github.com/StarRocks/starrocks/pull/37461) * When the FE dynamic parameter `enable_sync_publish` is set to `TRUE`, queries on data that is written after the BEs crash and then restart may fail. [#37398](https://github.com/StarRocks/starrocks/pull/37398) * The value of the `TABLE_CATALOG` field in `views` of the StarRocks Information Schema is `null`. [#37570](https://github.com/StarRocks/starrocks/pull/37570) * When `SELECT ... FROM ... INTO OUTFILE` is executed to export data into CSV files, the error "Unmatched number of columns" is reported if the FROM clause contains multiple constants. [#38045](https://github.com/StarRocks/starrocks/pull/38045) #### 3.1.6[​](#316 "Direct link to 3.1.6") Release date: December 18, 2023 ##### New Features[​](#new-features-5 "Direct link to New Features") * Added the [now(p)](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/now/) function to return the current date and time with the specified fractional seconds precision (accurate to the microsecond). If `p` is not specified, this function returns only date and time accurate to the second. [#36676](https://github.com/StarRocks/starrocks/pull/36676) * Added a new metric `max_tablet_rowset_num` for setting the maximum allowed number of rowsets. This metric helps detect possible compaction issues and thus reduces the occurrences of the error "too many versions". [#36539](https://github.com/StarRocks/starrocks/pull/36539) * Supports obtaining heap profiles by using a command line tool, making troubleshooting easier.[#35322](https://github.com/StarRocks/starrocks/pull/35322) * Supports creating asynchronous materialized views with common table expressions (CTEs). [#36142](https://github.com/StarRocks/starrocks/pull/36142) * Added the following bitmap functions: [subdivide\_bitmap](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/subdivide_bitmap/), [bitmap\_from\_binary](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_from_binary/), and [bitmap\_to\_binary](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_to_binary/). [#35817](https://github.com/StarRocks/starrocks/pull/35817) [#35621](https://github.com/StarRocks/starrocks/pull/35621) * Optimized the logic used to compute compaction scores for Primary Key tables, thereby aligning the compaction scores for Primary Key tables within a more consistent range with the other three table types. [#36534](https://github.com/StarRocks/starrocks/pull/36534) ##### Parameter Change[​](#parameter-change-2 "Direct link to Parameter Change") * The default retention period of trash files is changed to 1 day from the original 3 days. [#37113](https://github.com/StarRocks/starrocks/pull/37113) * A new BE configuration item `enable_stream_load_verbose_log` is added. The default value is `false`. With this parameter set to `true`, StarRocks can record the HTTP requests and responses for Stream Load jobs, making troubleshooting easier. [#36113](https://github.com/StarRocks/starrocks/pull/36113) * A new BE configuration item `enable_lazy_delta_column_compaction` is added. The default value is `true`, indicating that StarRocks does not perform frequent compaction operations on delta columns. [#36654](https://github.com/StarRocks/starrocks/pull/36654) * A new FE configuration item `enable_mv_automatic_active_check` is added to control whether the system automatically checks and re-activates the asynchronous materialized views that are set inactive because their base tables (views) had undergone Schema Change or had been dropped and re-created. The default value is `true`. [#36463](https://github.com/StarRocks/starrocks/pull/36463) ##### Improvements[​](#improvements-9 "Direct link to Improvements") * A new value option `GROUP_CONCAT_LEGACY` is added to the session variable [sql\_mode](https://docs.starrocks.io/docs/reference/System_variable/#sql_mode) to provide compatibility with the implementation logic of the [group\_concat](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/group_concat/) function in versions earlier than v2.5. [#36150](https://github.com/StarRocks/starrocks/pull/36150) * The Primary Key table size returned by the [SHOW DATA](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/SHOW_DATA/) statement includes the sizes of **.cols** files (these are files related to partial column updates and generated columns) and persistent index files. [#34898](https://github.com/StarRocks/starrocks/pull/34898) * Queries on MySQL external tables and the external tables within JDBC catalogs support including keywords in the WHERE clause. [#35917](https://github.com/StarRocks/starrocks/pull/35917) * Plugin loading failures will no longer cause an error or cause an FE start failure. Instead, the FE can properly start, and the error status of the plug-in can be queried using [SHOW PLUGINS](https://docs.starrocks.io/docs/sql-reference/sql-statements/Administration/SHOW_PLUGINS/). [#36566](https://github.com/StarRocks/starrocks/pull/36566) * Dynamic partitioning supports random distribution. [#35513](https://github.com/StarRocks/starrocks/pull/35513) * The result returned by the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/SHOW_ROUTINE_LOAD/) statement provides a new field `OtherMsg`, which shows information about the last failed task. [#35806](https://github.com/StarRocks/starrocks/pull/35806) * The authentication information `aws.s3.access_key` and `aws.s3.access_secret` for AWS S3 in Broker Load jobs are hidden in audit logs. [#36571](https://github.com/StarRocks/starrocks/pull/36571) * The `be_tablets` view in the `information_schema` database provides a new field `INDEX_DISK`, which records the disk usage (measured in bytes) of persistent indexes [#35615](https://github.com/StarRocks/starrocks/pull/35615) ##### Bug Fixes[​](#bug-fixes-11 "Direct link to Bug Fixes") Fixed the following issues: * The BEs crash if users create persistent indexes in the event of data corruption. [#30841](https://github.com/StarRocks/starrocks/pull/30841) * If users create an asynchronous materialized view that contains nested queries, the error "resolve partition column failed" is reported. [#26078](https://github.com/StarRocks/starrocks/issues/26078) * If users create an asynchronous materialized view on a base table whose data is corrupted, the error "Unexpected exception: null" is reported. [#30038](https://github.com/StarRocks/starrocks/pull/30038) * If users run a query that contains a window function, the SQL error "\[1064] \[42000]: Row count of const column reach limit: 4294967296" is reported. [#33561](https://github.com/StarRocks/starrocks/pull/33561) * The FE performance plunges after the FE configuration item `enable_collect_query_detail_info` is set to `true`. [#35945](https://github.com/StarRocks/starrocks/pull/35945) * In the StarRocks shared-data mode, the error "Reduce your request rate" may be reported when users attempt to delete files from object storage. [#35566](https://github.com/StarRocks/starrocks/pull/35566) * Deadlocks may occur when users refresh materialized views. [#35736](https://github.com/StarRocks/starrocks/pull/35736) * After the DISTINCT window operator pushdown feature is enabled, errors are reported if SELECT DISTINCT operations are performed on the complex expressions of the columns computed by window functions. [#36357](https://github.com/StarRocks/starrocks/pull/36357) * The BEs crash if the source data file is in ORC format and contains nested arrays. [#36127](https://github.com/StarRocks/starrocks/pull/36127) * Some S3-compatible object storage returns duplicate files, causing the BEs to crash. [#36103](https://github.com/StarRocks/starrocks/pull/36103) * The [array\_distinct](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_distinct/) function occasionally causes the BEs to crash. [#36377](https://github.com/StarRocks/starrocks/pull/36377) * Global Runtime Filter may cause BEs to crash in certain scenarios. [#35776](https://github.com/StarRocks/starrocks/pull/35776) #### 3.1.5[​](#315 "Direct link to 3.1.5") Release date: November 28, 2023 ##### New features[​](#new-features-6 "Direct link to New features") * The CN nodes of a StarRocks shared-data cluster now support data export. [#34018](https://github.com/StarRocks/starrocks/pull/34018) ##### Improvements[​](#improvements-10 "Direct link to Improvements") * The [`COLUMNS`](https://docs.starrocks.io/docs/reference/information_schema/columns/) view in the system database `INFORMATION_SCHEMA` can display ARRAY, MAP, and STRUCT columns. [#33431](https://github.com/StarRocks/starrocks/pull/33431) * Supports queries against Parquet, ORC, and CSV formatted files that are compressed by using LZO and stored in [Hive](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog/). [#30923](https://github.com/StarRocks/starrocks/pull/30923) [#30721](https://github.com/StarRocks/starrocks/pull/30721) * Supports updates onto the specified partitions of an automatically partitioned table. If the specified partitions do not exist, an error is returned. [#34777](https://github.com/StarRocks/starrocks/pull/34777) * Supports automatic refresh of materialized views when Swap, Drop, or Schema Change operations are performed on the tables and views (including the other tables and materialized views associated with these views) on which these materialized views are created. [#32829](https://github.com/StarRocks/starrocks/pull/32829) * Optimized the performance of some Bitmap-related operations, including: * Optimized nested loop joins. [#340804](https://github.com/StarRocks/starrocks/pull/34804) [#35003](https://github.com/StarRocks/starrocks/pull/35003) * Optimized the `bitmap_xor` function. [#34069](https://github.com/StarRocks/starrocks/pull/34069) * Supports Copy on Write to optimize Bitmap performance and reduce memory consumption. [#34047](https://github.com/StarRocks/starrocks/pull/34047) ##### Bug Fixes[​](#bug-fixes-12 "Direct link to Bug Fixes") Fixed the following issues: * If a filtering condition is specified in a Broker Load job, BEs may crash during the data loading in certain circumstances. [#29832](https://github.com/StarRocks/starrocks/pull/29832) * An unknown error is reported when SHOW GRANTS is executed. [#30100](https://github.com/StarRocks/starrocks/pull/30100) * When data is loaded into a table that uses expression-based automatic partitioning, the error "Error: The row create partition failed since Runtime error: failed to analyse partition value" may be thrown. [#33513](https://github.com/StarRocks/starrocks/pull/33513) * The error "get\_applied\_rowsets failed, tablet updates is in error state: tablet:18849 actual row size changed after compaction" is returned for queries. [#33246](https://github.com/StarRocks/starrocks/pull/33246) * In a StarRocks shared-nothing cluster, queries against Iceberg or Hive tables may cause BEs to crash. [#34682](https://github.com/StarRocks/starrocks/pull/34682) * In a StarRocks shared-nothing cluster, if multiple partitions are automatically created during data loading, the data loaded may occasionally be written to unmatched partitions. [#34731](https://github.com/StarRocks/starrocks/pull/34731) * Long-time, frequent data loading into a Primary Key table with persistent index enabled may cause BEs to crash. [#33220](https://github.com/StarRocks/starrocks/pull/33220) * The error "Exception: java.lang.IllegalStateException: null" is returned for queries. [#33535](https://github.com/StarRocks/starrocks/pull/33535) * When `show proc '/current_queries';` is being executed and meanwhile a query begins to be executed, BEs may crash. [#34316](https://github.com/StarRocks/starrocks/pull/34316) * Errors may be thrown if large amounts of data are loaded into a Primary Key table with persistent index enabled. [#34352](https://github.com/StarRocks/starrocks/pull/34352) * After StarRocks is upgraded from v2.4 or earlier to a later version, compaction scores may rise unexpectedly. [#34618](https://github.com/StarRocks/starrocks/pull/34618) * If `INFORMATION_SCHEMA` is queried by using the database driver MariaDB ODBC, the `CATALOG_NAME` column returned in the `schemata` view holds only `null` values. [#34627](https://github.com/StarRocks/starrocks/pull/34627) * FEs crash due to the abnormal data loaded and cannot restart. [#34590](https://github.com/StarRocks/starrocks/pull/34590) * If schema changes are being executed while a Stream Load job is in the **PREPARED** state, a portion of the source data to be loaded by the job is lost. [#34381](https://github.com/StarRocks/starrocks/pull/34381) * Including two or more slashes (`/`) at the end of the HDFS storage path causes the backup and restore of the data from HDFS to fail. [#34601](https://github.com/StarRocks/starrocks/pull/34601) * Setting the session variable `enable_load_profile` to `true` makes Stream Load jobs prone to fail. [#34544](https://github.com/StarRocks/starrocks/pull/34544) * Performing partial updates in column mode onto a Primary Key table causes some tablets of the table to show data inconsistencies between their replicas. [#34555](https://github.com/StarRocks/starrocks/pull/34555) * The `partition_live_number` property added by using the ALTER TABLE statement does not take effect. [#34842](https://github.com/StarRocks/starrocks/pull/34842) * FEs fail to start and report the error "failed to load journal type 118". [#34590](https://github.com/StarRocks/starrocks/pull/34590) * Setting the FE parameter `recover_with_empty_tablet` to `true` may cause FEs to crash. [#33071](https://github.com/StarRocks/starrocks/pull/33071) * Failures in replaying replica operations may cause FEs to crash. [#32295](https://github.com/StarRocks/starrocks/pull/32295) ##### Parameter Change[​](#parameter-change-3 "Direct link to Parameter Change") ###### FE/BE Parameters[​](#febe-parameters "Direct link to FE/BE Parameters") * Added an FE configuration item [`enable_statistics_collect_profile`](https://docs.starrocks.io/docs/administration/FE_configuration#enable_statistics_collect_profile), which controls whether to generate profiles for statistics queries. The default value is `false`. [#33815](https://github.com/StarRocks/starrocks/pull/33815) * The FE configuration item [`mysql_server_version`](https://docs.starrocks.io/docs/administration/FE_configuration#mysql_server_version) is now mutable. The new setting can take effect for the current session without requiring an FE restart. [#34033](https://github.com/StarRocks/starrocks/pull/34033) * Added a BE/CN configuration item [`update_compaction_ratio_threshold`](https://docs.starrocks.io/docs/administration/BE_configuration#update_compaction_ratio_threshold), which controls the maximum proportion of data that a compaction can merge for a Primary Key table in a StarRocks shared-data cluster. The default value is `0.5`. We recommend shrinking this value if a single tablet becomes excessively large. For a StarRocks shared-nothing cluster, the proportion of data that a compaction can merge for a Primary Key table is still automatically adjusted. [#35129](https://github.com/StarRocks/starrocks/pull/35129) ###### System Variables[​](#system-variables "Direct link to System Variables") * Added a session variable `cbo_decimal_cast_string_strict`, which controls how the CBO converts data from the DECIMAL type to the STRING type. If this variable is set to `true`, the logic built in v2.5.x and later versions prevails and the system implements strict conversion (namely, the system truncates the generated string and fills 0s based on the scale length). If this variable is set to `false`, the logic built in versions earlier than v2.5.x prevails and the system processes all valid digits to generate a string. The default value is `true`. [#34208](https://github.com/StarRocks/starrocks/pull/34208) * Added a session variable `cbo_eq_base_type`, which specifies the data type used for data comparison between DECIMAL-type data and STRING-type data. The default value is `VARCHAR`, and `DECIMAL` is also a valid value. [#34208](https://github.com/StarRocks/starrocks/pull/34208) * Added a session variable `big_query_profile_second_threshold`. When the session variable [`enable_profile`](https://docs.starrocks.io/docs/reference/System_variable#enable_profile) is set to `false` and the amount of time taken by a query exceeds the threshold specified by the `big_query_profile_second_threshold` variable, a profile is generated for that query. [#33825](https://github.com/StarRocks/starrocks/pull/33825) #### 3.1.4[​](#314 "Direct link to 3.1.4") Release date: November 2, 2023 ##### New Features[​](#new-features-7 "Direct link to New Features") * Supports sort keys for Primary Key tables created in shared-data StarRocks clusters. * Supports using the str2date function to specify partition expressions for asynchronous materialized views. This helps facilitate incremental updates and query rewrites of asynchronous materialized views created on tables that reside in external catalogs and use the STRING-type data as their partitioning expressions. [#29923](https://github.com/StarRocks/starrocks/pull/29923) [#31964](https://github.com/StarRocks/starrocks/pull/31964) * Added a new session variable `enable_query_tablet_affinity`, which controls whether to direct multiple queries against the same tablet to a fixed replica. This session variable is set to `false` by default. [#33049](https://github.com/StarRocks/starrocks/pull/33049) * Added the utility function `is_role_in_session`, which is used to check whether the specified roles are activated in the current session. It supports checking nested roles granted to a user. [#32984](https://github.com/StarRocks/starrocks/pull/32984) * Supports setting resource group-level query queue, which is controlled by the global variable `enable_group_level_query_queue` (default value: `false`). When the global-level or resource group-level resource consumption reaches a predefined threshold, new queries are placed in queue, and will be run when both the global-level resource consumption and the resource group-level resource consumption fall below their thresholds. * Users can set `concurrency_limit` for each resource group to limit the maximum number of concurrent queries allowed per BE. * Users can set `max_cpu_cores` for each resource group to limit the maximum CPU consumption allowed per BE. * Added two parameters, `plan_cpu_cost_range` and `plan_mem_cost_range`, for resource group classifiers. * `plan_cpu_cost_range`: the CPU consumption range estimated by the system. The default value `NULL` indicates no limit is imposed. * `plan_mem_cost_range`: the memory consumption range estimated by the system. The default value `NULL` indicates no limit is imposed. ##### Improvements[​](#improvements-11 "Direct link to Improvements") * Window functions COVAR\_SAMP, COVAR\_POP, CORR, VARIANCE, VAR\_SAMP, STD, and STDDEV\_SAMP now support the ORDER BY clause and Window clause. [#30786](https://github.com/StarRocks/starrocks/pull/30786) * An error instead of NULL is returned if a decimal overflow occurs during queries on the DECIMAL type data. [#30419](https://github.com/StarRocks/starrocks/pull/30419) * The number of concurrent queries allowed in a query queue is now managed by the leader FE. Each follower FE notifies of the leader FE when a query starts and finishes. If the number of concurrent queries reaches the global-level or resource group-level `concurrency_limit`, new queries are rejected or placed in queue. ##### Bug Fixes[​](#bug-fixes-13 "Direct link to Bug Fixes") Fixed the following issues: * Spark or Flink may report data read errors due to inaccurate memory usage statistics. [#30702](https://github.com/StarRocks/starrocks/pull/30702) [#30751](https://github.com/StarRocks/starrocks/pull/30751) * Memory usage statistics for Metadata Cache are inaccurate. [#31978](https://github.com/StarRocks/starrocks/pull/31978) * BEs crash when libcurl is invoked. [#31667](https://github.com/StarRocks/starrocks/pull/31667) * When StarRocks materialized views created on Hive views are refreshed, an error "java.lang.ClassCastException: com.starrocks.catalog.HiveView cannot be cast to com.starrocks.catalog.HiveMetaStoreTable" is returned. [#31004](https://github.com/StarRocks/starrocks/pull/31004) * If the ORDER BY clause contains aggregate functions, an error "java.lang.IllegalStateException: null" is returned. [#30108](https://github.com/StarRocks/starrocks/pull/30108) * In shared-data StarRocks clusters, the information of table keys is not recorded in `information_schema.COLUMNS`. As a result, DELETE operations cannot be performed when data is loaded by using Flink Connector. [#31458](https://github.com/StarRocks/starrocks/pull/31458) * When data is loaded by using Flink Connector, the load job is suspended unexpectedly if there are highly concurrent load jobs and both the number of HTTP threads and the number of Scan threads have reached their upper limits. [#32251](https://github.com/StarRocks/starrocks/pull/32251) * When a field of only a few bytes is added, executing SELECT COUNT(\*) before the data change finishes returns an error that reads "error: invalid field name". [#33243](https://github.com/StarRocks/starrocks/pull/33243) * Query results are incorrect after the query cache is enabled. [#32781](https://github.com/StarRocks/starrocks/pull/32781) * Queries fail during hash joins, causing BEs to crash. [#32219](https://github.com/StarRocks/starrocks/pull/32219) * `DATA_TYPE` and `COLUMN_TYPE` for BINARY or VARBINARY data types are displayed as `unknown` in the `information_schema.columns` view. [#32678](https://github.com/StarRocks/starrocks/pull/32678) ##### Behavior Change[​](#behavior-change-1 "Direct link to Behavior Change") * From v3.1.4 onwards, persistent indexing is enabled by default for Primary Key tables created in new StarRocks clusters (this does not apply to existing StarRocks clusters whose versions are upgraded to v3.1.4 from an earlier version). [#33374](https://github.com/StarRocks/starrocks/pull/33374) * A new FE parameter `enable_sync_publish` which is set to `true` by default is added. When this parameter is set to `true`, the Publish phase of a data load into a Primary Key table returns the execution result only after the Apply task finishes. As such, the data loaded can be queried immediately after the load job returns a success message. However, setting this parameter to `true` may cause data loads into Primary Key tables to take a longer time. (Before this parameter is added, the Apply task is asynchronous with the Publish phase.) [#27055](https://github.com/StarRocks/starrocks/pull/27055) #### 3.1.3 (Yanked)[​](#313-yanked "Direct link to 3.1.3 (Yanked)") tip This version has been taken offline. Release date: September 25, 2023 ##### New Features[​](#new-features-8 "Direct link to New Features") * Primary Key tables created in shared-data StarRocks clusters support index persistence onto local disks in the same way as they do in shared-nothing StarRocks clusters. * The aggregate function [group\_concat](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/group_concat/) supports the DISTINCT keyword and the ORDER BY clause. [#28778](https://github.com/StarRocks/starrocks/pull/28778) * [Stream Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/STREAM_LOAD/), [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/BROKER_LOAD/), [Kafka Connector](https://docs.starrocks.io/docs/loading/Kafka-connector-starrocks/), [Flink Connector](https://docs.starrocks.io/docs/loading/Flink-connector-starrocks/), and [Spark Connector](https://docs.starrocks.io/docs/loading/Spark-connector-starrocks/) support partial updates in column mode on a Primary Key table. [#28288](https://github.com/StarRocks/starrocks/pull/28288) * Data in partitions can be automatically cooled down over time. (This feature is not supported for [list partitioning](https://docs.starrocks.io/docs/table_design/list_partitioning/).) [#29335](https://github.com/StarRocks/starrocks/pull/29335) [#29393](https://github.com/StarRocks/starrocks/pull/29393) ##### Improvements[​](#improvements-12 "Direct link to Improvements") Executing SQL commands with invalid comments now returns results consistent with MySQL. [#30210](https://github.com/StarRocks/starrocks/pull/30210) ##### Bug Fixes[​](#bug-fixes-14 "Direct link to Bug Fixes") Fixed the following issues: * If the [BITMAP](https://docs.starrocks.io/docs/sql-reference/data-types/other-data-types/BITMAP/) or [HLL](https://docs.starrocks.io/docs/sql-reference/data-types/other-data-types/HLL/) data type is specified in the WHERE clause of a [DELETE](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/DELETE/) statement to be executed, the statement cannot be properly executed. [#28592](https://github.com/StarRocks/starrocks/pull/28592) * After a follower FE is restarted, CpuCores statistics are not up-to-date, resulting in query performance degradation. [#28472](https://github.com/StarRocks/starrocks/pull/28472) [#30434](https://github.com/StarRocks/starrocks/pull/30434) * The execution cost of the [to\_bitmap()](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/to_bitmap/) function is incorrectly calculated. As a result, an inappropriate execution plan is selected for the function after materialized views are rewritten. [#29961](https://github.com/StarRocks/starrocks/pull/29961) * In certain use cases of the shared-data architecture, after a follower FE is restarted, queries submitted to the follower FE return an error that reads "Backend node not found. Check if any backend node is down". [#28615](https://github.com/StarRocks/starrocks/pull/28615) * If data is continuously loaded into a table that is being altered by using the [ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-definition/ALTER_TABLE/) statement, an error "Tablet is in error state" may be thrown. [#29364](https://github.com/StarRocks/starrocks/pull/29364) * Modifying the FE dynamic parameter `max_broker_load_job_concurrency` using the `ADMIN SET FRONTEND CONFIG` command does not take effect. [#29964](https://github.com/StarRocks/starrocks/pull/29964) [#29720](https://github.com/StarRocks/starrocks/pull/29720) * BEs crash if the time unit in the [date\_diff()](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/date_diff/) function is a constant but the dates are not constants. [#29937](https://github.com/StarRocks/starrocks/issues/29937) * In the shared-data architecture, automatic partitioning does not take effect after asynchronous load is enabled. [#29986](https://github.com/StarRocks/starrocks/issues/29986) * If users create a Primary Key table by using the [CREATE TABLE LIKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-definition/CREATE_TABLE_LIKE/) statement, an error `Unexpected exception: Unknown properties: {persistent_index_type=LOCAL}` is thrown. [#30255](https://github.com/StarRocks/starrocks/pull/30255) * Restoring Primary Key tables causes metadata inconsistency after BEs are restarted. [#30135](https://github.com/StarRocks/starrocks/pull/30135) * If users load data into a Primary Key table on which truncate operations and queries are concurrently performed, an error "java.lang.NullPointerException" is thrown in certain cases. [#30573](https://github.com/StarRocks/starrocks/pull/30573) * If predicate expressions are specified in materialized view creation statements, the refresh results of those materialized views are incorrect. [#29904](https://github.com/StarRocks/starrocks/pull/29904) * After users upgrade their StarRocks cluster to v3.1.2, the storage volume properties of the tables created before the upgrade are reset to `null`. [#30647](https://github.com/StarRocks/starrocks/pull/30647) * If checkpointing and restoration are concurrently performed on tablet metadata, some tablet replicas will be lost and cannot be retrieved. [#30603](https://github.com/StarRocks/starrocks/pull/30603) * If users use CloudCanal to load data into table columns that are set to `NOT NULL` but have no default value specified, an error "Unsupported dataFormat value is : \N" is thrown. [#30799](https://github.com/StarRocks/starrocks/pull/30799) ##### Behavior Change[​](#behavior-change-2 "Direct link to Behavior Change") * When using the [group\_concat](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/group_concat/) function, users must use the SEPARATOR keyword to declare the separator. * The default value of the session variable [`group_concat_max_len`](https://docs.starrocks.io/docs/reference/System_variable#group_concat_max_len) which controls the default maximum length of the string returned by the [group\_concat](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/group_concat/) function is changed from unlimited to `1024`. #### 3.1.2[​](#312 "Direct link to 3.1.2") Release date: August 25, 2023 ##### Bug Fixes[​](#bug-fixes-15 "Direct link to Bug Fixes") Fixed the following issues: * If a user specifies which database is to be connected by default and the user only has permissions on tables in the database but does not have permissions on the database, an error stating that the user does not have permissions on the database is thrown. [#29767](https://github.com/StarRocks/starrocks/pull/29767) * The values returned by the RESTful API action `show_data` for cloud-native tables are incorrect. [#29473](https://github.com/StarRocks/starrocks/pull/29473) * BEs crash if queries are canceled while the [array\_agg()](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_agg/) function is being run. [#29400](https://github.com/StarRocks/starrocks/issues/29400) * The `Default` field values returned by the [SHOW FULL COLUMNS](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/SHOW_FULL_COLUMNS/) statement for columns of the [BITMAP](https://docs.starrocks.io/docs/sql-reference/data-types/other-data-types/BITMAP/) or [HLL](https://docs.starrocks.io/docs/sql-reference/data-types/other-data-types/HLL/) data type are incorrect. [#29510](https://github.com/StarRocks/starrocks/pull/29510) * If the [array\_map()](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_map/) function in queries involves multiple tables, the queries fail due to pushdown strategy issues. [#29504](https://github.com/StarRocks/starrocks/pull/29504) * Queries against ORC-formatted files fail because the bugfix ORC-1304 ([apache/orc#1299](https://github.com/apache/orc/pull/1299)) from Apache ORC is not merged. [#29804](https://github.com/StarRocks/starrocks/pull/29804) ##### Behavior Change[​](#behavior-change-3 "Direct link to Behavior Change") For a newly deployed StarRocks v3.1 cluster, you must have the USAGE privilege on the destination external catalog if you want to run [SET CATALOG](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-definition/SET_CATALOG/) to switch to that catalog. You can use [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT/) to grant the required privileges. For a v3.1 cluster upgraded from an earlier version, you can run SET CATALOG with inherited privilege. #### 3.1.1[​](#311 "Direct link to 3.1.1") Release date: August 18, 2023 ##### New Features[​](#new-features-9 "Direct link to New Features") * Supports Azure Blob Storage for [shared-data clusters](https://docs.starrocks.io/docs/deployment/shared_data/s3/). * Supports List partitioning for [shared-data clusters](https://docs.starrocks.io/docs/deployment/shared_data/s3/). * Supports aggregate functions [COVAR\_SAMP](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/covar_samp/), [COVAR\_POP](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/covar_pop/), and [CORR](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/corr/). * Supports the following [window functions](https://docs.starrocks.io/docs/sql-reference/sql-functions/Window_function/): COVAR\_SAMP, COVAR\_POP, CORR, VARIANCE, VAR\_SAMP, STD, and STDDEV\_SAMP. ##### Improvements[​](#improvements-13 "Direct link to Improvements") Supports implicit conversions for all compound predicates and for all expressions in the WHERE clause. You can enable or disable implicit conversions by using the [session variable](https://docs.starrocks.io/docs/reference/System_variable/) `enable_strict_type`. The default value of this session variable is `false`. ##### Bug Fixes[​](#bug-fixes-16 "Direct link to Bug Fixes") Fixed the following issues: * When data is loaded into tables that have multiple replicas, a large number of invalid log records are written if some partitions of the tables are empty. [#28824](https://github.com/StarRocks/starrocks/issues/28824) * Inaccurate estimation of average row size causes partial updates in column mode on Primary Key tables to occupy excessively large memory. [#27485](https://github.com/StarRocks/starrocks/pull/27485) * If clone operations are triggered on tablets in an ERROR state, disk usage increases. [#28488](https://github.com/StarRocks/starrocks/pull/28488) * Compaction causes cold data to be written to the local cache. [#28831](https://github.com/StarRocks/starrocks/pull/28831) #### 3.1.0[​](#310 "Direct link to 3.1.0") Release date: August 7, 2023 ##### New Features[​](#new-features-10 "Direct link to New Features") ###### Shared-data cluster[​](#shared-data-cluster "Direct link to Shared-data cluster") * Added support for Primary Key tables, on which persistent indexes cannot be enabled. * Supports the [AUTO\_INCREMENT](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/auto_increment/) column attribute, which enables a globally unique ID for each data row and thus simplifies data management. * Supports [automatically creating partitions during loading and using partitioning expressions to define partitioning rules](https://docs.starrocks.io/docs/table_design/data_distribution/expression_partitioning/), thereby making partition creation easier to use and more flexible. * Supports [abstraction of storage volumes](https://docs.starrocks.io/docs/deployment/shared_data/s3#use-your-shared-data-starrocks-cluster), in which users can configure storage location and authentication information, in shared-data StarRocks clusters. Users can directly reference an existing storage volume when creating a database or table, making authentication configuration easier. ###### Data Lake analytics[​](#data-lake-analytics "Direct link to Data Lake analytics") * Supports accessing views created on tables within [Hive catalogs](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog/). * Supports accessing Parquet-formatted Iceberg v2 tables. * Supports [sinking data to Parquet-formatted Iceberg tables](https://docs.starrocks.io/docs/data_source/catalog/iceberg_catalog#sink-data-to-an-iceberg-table). * \[Preview] Supports accessing data stored in Elasticsearch by using [Elasticsearch catalogs](https://docs.starrocks.io/docs/data_source/catalog/elasticsearch_catalog/). This simplifies the creation of Elasticsearch external tables. * \[Preview] Supports performing analytics on streaming data stored in Apache Paimon by using [Paimon catalogs](https://docs.starrocks.io/docs/data_source/catalog/paimon_catalog/). ###### Storage engine, data ingestion, and query[​](#storage-engine-data-ingestion-and-query "Direct link to Storage engine, data ingestion, and query") * Upgraded automatic partitioning to [expression partitioning](https://docs.starrocks.io/docs/table_design/expression_partitioning/). Users only need to use a simple partition expression (either a time function expression or a column expression) to specify a partitioning method at table creation, and StarRocks will automatically create partitions based on the data characteristics and the rule defined in the partition expression during data loading. This method of partition creation is suitable for most scenarios and is more flexible and user-friendly. * Supports [list partitioning](https://docs.starrocks.io/docs/table_design/list_partitioning/). Data is partitioned based on a list of values predefined for a particular column, which can accelerate queries and manage clearly categorized data more efficiently. * Added a new table named `loads` to the `Information_schema` database. Users can query the results of [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/BROKER_LOAD/) and [Insert](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/INSERT/) jobs from the `loads` table. * Supports logging the unqualified data rows that are filtered out by [Stream Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/STREAM_LOAD/), [Broker Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/BROKER_LOAD/), and [Spark Load](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/SPARK_LOAD/) jobs. Users can use the `log_rejected_record_num` parameter in their load job to specify the maximum number of data rows that can be logged. * Supports [random bucketing](https://docs.starrocks.io/docs/table_design/Data_distribution#how-to-choose-the-bucketing-columns). With this feature, users do not need to configure bucketing columns at table creation, and StarRocks will randomly distribute the data loaded into it to buckets. Using this feature together with the capability of automatically setting the number of buckets (`BUCKETS`) that StarRocks has provided since v2.5.7, users no longer need to consider bucket configurations, and table creation statements are greatly simplified. In big data and high performance-demanding scenarios, however, we recommend that users continue using hash bucketing, because this way they can use bucket pruning to accelerate queries. * Supports using the table function FILES() in [INSERT INTO](https://docs.starrocks.io/docs/loading/InsertInto/) to directly load the data of Parquet- or ORC-formatted data files stored in AWS S3. The FILES() function can automatically infer the table schema, which relieves the need to create external catalogs or file external tables before data loading and therefore greatly simplifies the data loading process. * Supports [generated columns](https://docs.starrocks.io/docs/sql-reference/sql-statements/generated_columns/). With the generated column feature, StarRocks can automatically generate and store the values of column expressions and automatically rewrite queries to improve query performance. * Supports loading data from Spark to StarRocks by using [Spark connector](https://docs.starrocks.io/docs/loading/Spark-connector-starrocks/). Compared to [Spark Load](https://docs.starrocks.io/docs/loading/SparkLoad/), the Spark connector provides more comprehensive capabilities. Users can define a Spark job to perform ETL operations on the data, and the Spark connector serves as the sink in the Spark job. * Supports loading data into columns of the [MAP](https://docs.starrocks.io/docs/sql-reference/data-types/semi_structured/Map/) and [STRUCT](https://docs.starrocks.io/docs/sql-reference/data-types/semi_structured/STRUCT/) data types, and supports nesting Fast Decimal values in ARRAY, MAP, and STRUCT. ###### SQL reference[​](#sql-reference "Direct link to SQL reference") * Added the following storage volume-related statements: [CREATE STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/Administration/CREATE_STORAGE_VOLUME/), [ALTER STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/Administration/ALTER_STORAGE_VOLUME/), [DROP STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/Administration/DROP_STORAGE_VOLUME/), [SET DEFAULT STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/Administration/SET_DEFAULT_STORAGE_VOLUME/), [DESC STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/Administration/DESC_STORAGE_VOLUME/), [SHOW STORAGE VOLUMES](https://docs.starrocks.io/docs/sql-reference/sql-statements/Administration/SHOW_STORAGE_VOLUMES/). * Supports altering table comments using [ALTER TABLE](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-definition/ALTER_TABLE/). [#21035](https://github.com/StarRocks/starrocks/pull/21035) * Added the following functions: * Struct functions: [struct (row)](https://docs.starrocks.io/docs/sql-reference/sql-functions/struct-functions/row/), [named\_struct](https://docs.starrocks.io/docs/sql-reference/sql-functions/struct-functions/named_struct/) * Map functions: [str\_to\_map](https://docs.starrocks.io/docs/sql-reference/sql-functions/string-functions/str_to_map/), [map\_concat](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_concat/), [map\_from\_arrays](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_from_arrays/), [element\_at](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/element_at/), [distinct\_map\_keys](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/distinct_map_keys/), [cardinality](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/cardinality/) * Higher-order Map functions: [map\_filter](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_filter/), [map\_apply](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/map_apply/), [transform\_keys](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/transform_keys/), [transform\_values](https://docs.starrocks.io/docs/sql-reference/sql-functions/map-functions/transform_values/) * Array functions: [array\_agg](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_agg/) supports `ORDER BY`, [array\_generate](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_generate/), [element\_at](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/element_at/), [cardinality](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/cardinality/) * Higher-order Array functions: [all\_match](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/all_match/), [any\_match](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/any_match/) * Aggregate functions: [min\_by](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/min_by/), [percentile\_disc](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/percentile_disc/) * Table functions: [FILES](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files/), [generate\_series](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/generate_series/) * Date functions: [next\_day](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/next_day/), [previous\_day](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/previous_day/), [last\_day](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/last_day/), [makedate](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/makedate/), [date\_diff](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/date_diff/) * Bitmap functions: [bitmap\_subset\_limit](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_subset_limit/), [bitmap\_subset\_in\_range](https://docs.starrocks.io/docs/sql-reference/sql-functions/bitmap-functions/bitmap_subset_in_range/) * Math functions: [cosine\_similarity](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/cos_similarity/), [cosine\_similarity\_norm](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/cos_similarity_norm/) ###### Privileges and security[​](#privileges-and-security "Direct link to Privileges and security") Added [privilege items](https://docs.starrocks.io/docs/administration/privilege_item#storage-volume) related to storage volumes and [privilege items](https://docs.starrocks.io/docs/administration/privilege_item#catalog) related to external catalogs, and supports using [GRANT](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/GRANT/) and [REVOKE](https://docs.starrocks.io/docs/sql-reference/sql-statements/account-management/REVOKE/) to grant and revoke these privileges. ##### Improvements[​](#improvements-14 "Direct link to Improvements") ###### Shared-data cluster[​](#shared-data-cluster-1 "Direct link to Shared-data cluster") Optimized the data cache in shared-data StarRocks clusters. The optimized data cache allows for specifying the range of hot data. It can also prevent queries against cold data from occupying the local disk cache, thereby ensuring the performance of queries against hot data. ###### Materialized view[​](#materialized-view "Direct link to Materialized view") * Optimized the creation of an asynchronous materialized view: * Supports random bucketing. If users do not specify bucketing columns, StarRocks adopts random bucketing by default. * Supports using `ORDER BY` to specify a sort key. * Supports specifying attributes such as `colocate_group`, `storage_medium`, and `storage_cooldown_time`. * Supports using session variables. Users can configure these variables by using the `properties("session." = "")` syntax to flexibly adjust view refreshing strategies. * Enables the spill feature for all asynchronous materialized views and implements a query timeout duration of 1 hour by default. * Supports creating materialized views based on views. This makes materialized views easier to use in data modeling scenarios, because users can flexibly use views and materialized views based on their varying needs to implement layered modeling. * Optimized query rewrite with asynchronous materialized views: * Supports Stale Rewrite, which allows materialized views that are not refreshed within a specified time interval to be used for query rewrite regardless of whether the base tables of the materialized views are updated. Users can specify the time interval by using the `mv_rewrite_staleness_second` property at materialized view creation. * Supports rewriting View Delta Join queries against materialized views that are created on Hive catalog tables (a primary key and a foreign key must be defined). * Optimized the mechanism for rewriting queries that contain union operations, and supports rewriting queries that contain joins or functions such as COUNT DISTINCT and time\_slice. * Optimized the refreshing of asynchronous materialized views: * Optimized the mechanism for refreshing materialized views that are created on Hive catalog tables. StarRocks now can perceive partition-level data changes, and refreshes only the partitions with data changes during each automatic refresh. * Supports using the `REFRESH MATERIALIZED VIEW WITH SYNC MODE` syntax to synchronously invoke materialized view refresh tasks. * Enhanced the use of asynchronous materialized views: * Supports using `ALTER MATERIALIZED VIEW {ACTIVE | INACTIVE}` to enable or disable a materialized view. Materialized views that are disabled (in the `INACTIVE` state) cannot be refreshed or used for query rewrite, but can be directly queried. * Supports using `ALTER MATERIALIZED VIEW SWAP WITH` to swap two materialized views. Users can create a new materialized view and then perform an atomic swap with an existing materialized view to implement schema changes on the existing materialized view. * Optimized synchronous materialized views: * Supports direct queries against synchronous materialized views using SQL hints `[_SYNC_MV_]`, allowing for walking around issues that some queries cannot be properly rewritten in rare circumstances. * Supports more expressions, such as `CASE-WHEN`, `CAST`, and mathematical operations, which make materialized views suitable for more business scenarios. ###### Data Lake analytics[​](#data-lake-analytics-1 "Direct link to Data Lake analytics") * Optimized metadata caching and access for Iceberg to improve Iceberg data query performance. * Optimized the data cache to further improve data lake analytics performance. ###### Storage engine, data ingestion, and query[​](#storage-engine-data-ingestion-and-query-1 "Direct link to Storage engine, data ingestion, and query") * Announced the general availability of the [spill](https://docs.starrocks.io/docs/3.1/administration/management/resource_management/spill_to_disk/) feature, which supports spilling the intermediate computation results of some blocking operators to disk. With the spill feature enabled, when a query contains aggregate, sort, or join operators, StarRocks can cache the intermediate computation results of the operators to disk to reduce memory consumption, thereby minimizing query failures caused by memory limits. * Supports pruning on cardinality-preserving joins. If users maintain a large number of tables which are organized in the star schema (for example, SSB) or the snowflake schema (for example, TCP-H) but they query only a small number of these tables, this feature helps prune unnecessary tables to improve the performance of joins. * Supports partial updates in column mode. Users can enable the column mode when they perform partial updates on Primary Key tables by using the [UPDATE](https://docs.starrocks.io/docs/sql-reference/sql-statements/data-manipulation/UPDATE/) statement. The column mode is suitable for updating a small number of columns but a large number of rows, and can improve the updating performance by up to 10 times. * Optimized the collection of statistics for the CBO. This reduces the impact of statistics collection on data ingestion and increases statistics collection performance. * Optimized the merge algorithm to increase the overall performance by up to 2 times in permutation scenarios. * Optimized the query logic to reduce dependency on database locks. * Dynamic partitioning further supports the partitioning unit to be year. [#28386](https://github.com/StarRocks/starrocks/pull/28386) ###### SQL reference[​](#sql-reference-1 "Direct link to SQL reference") * Conditional functions case, coalesce, if, ifnull, and nullif support the ARRAY, MAP, STRUCT, and JSON data types. * The following Array functions support nested types MAP, STRUCT, and ARRAY: * array\_agg * array\_contains, array\_contains\_all, array\_contains\_any * array\_slice, array\_concat * array\_length, array\_append, array\_remove, array\_position * reverse, array\_distinct, array\_intersect, arrays\_overlap * array\_sortby * The following Array functions support the Fast Decimal data type: * array\_agg * array\_append, array\_remove, array\_position, array\_contains * array\_length * array\_max, array\_min, array\_sum, array\_avg * arrays\_overlap, array\_difference * array\_slice, array\_distinct, array\_sort, reverse, array\_intersect, array\_concat * array\_sortby, array\_contains\_all, array\_contains\_any ##### Bug Fixes[​](#bug-fixes-17 "Direct link to Bug Fixes") Fixed the following issues: * Requests to reconnect to Kafka for Routine Load jobs cannot be properly processed. [#23477](https://github.com/StarRocks/starrocks/issues/23477) * For SQL queries that involve multiple tables and contain a `WHERE` clause, if these SQL queries have the same semantics but the order of the tables in each SQL query is different, some of these SQL queries may fail to be rewritten to benefit from the related materialized views. [#22875](https://github.com/StarRocks/starrocks/issues/22875) * Duplicate records are returned for queries that contain a `GROUP BY` clause. [#19640](https://github.com/StarRocks/starrocks/issues/19640) * Invoking the lead() or lag() function may cause BE crashes. [#22945](https://github.com/StarRocks/starrocks/issues/22945) * Rewriting partial partition queries based on materialized views that are created on external catalog tables fail. [#19011](https://github.com/StarRocks/starrocks/issues/19011) * SQL statements that contain both a backward slash (`\`) and a semicolon (`;`) cannot be properly parsed. [#16552](https://github.com/StarRocks/starrocks/issues/16552) * A table cannot be truncated if a materialized view created on the table is removed. [#19802](https://github.com/StarRocks/starrocks/issues/19802) ##### Behavior Change[​](#behavior-change-4 "Direct link to Behavior Change") * The `storage_cache_ttl` parameter is deleted from the table creation syntax used for shared-data StarRocks clusters. Now the data in the local cache is evicted based on the LRU algorithm. * The BE configuration items `disable_storage_page_cache` and `alter_tablet_worker_count` and the FE configuration item `lake_compaction_max_tasks` are changed from immutable parameters to mutable parameters. * The default value of the BE configuration item `block_cache_checksum_enable` is changed from `true` to `false`. * The default value of the BE configuration item `enable_new_load_on_memory_limit_exceeded` is changed from `false` to `true`. * The default value of the FE configuration item `max_running_txn_num_per_db` is changed from `100` to `1000`. * The default value of the FE configuration item `http_max_header_size` is changed from `8192` to `32768`. * The default value of the FE configuration item `tablet_create_timeout_second` is changed from `1` to `10`. * The default value of the FE configuration item `max_routine_load_task_num_per_be` is changed from `5` to `16`, and error information will be returned if a large number of Routine Load tasks are created. * The FE configuration item `quorom_publish_wait_time_ms` is renamed as `quorum_publish_wait_time_ms`, and the FE configuration item `async_load_task_pool_size` is renamed as `max_broker_load_job_concurrency`. * The BE configuration item `routine_load_thread_pool_size` is deprecated. Now the routine load thread pool size per BE node is controlled only by the FE configuration item `max_routine_load_task_num_per_be`. * The BE configuration item `txn_commit_rpc_timeout_ms` and the system variable `tx_visible_wait_timeout` are deprecated. * The FE configuration items `max_broker_concurrency` and `load_parallel_instance_num` are deprecated. * The FE configuration item `max_routine_load_job_num` is deprecated. Now StarRocks dynamically infers the maximum number of Routine Load tasks supported by each individual BE node based on the `max_routine_load_task_num_per_be` parameter and provides suggestions on task failures. * The CN configuration item `thrift_port` is renamed as `be_port`. * Two new Routine Load job properties, `task_consume_second` and `task_timeout_second`, are added to control the maximum amount of time to consume data and the timeout duration for individual load tasks within a Routine Load job, making job adjustment more flexible. If users do not specify these two properties in their Routine Load job, the FE configuration items `routine_load_task_consume_second` and `routine_load_task_timeout_second` prevail. * The session variable `enable_resource_group` is deprecated because the [Resource Group](https://docs.starrocks.io/docs/administration/resource_group/) feature is enabled by default since v3.1.0. * Two new reserved keywords, COMPACTION and TEXT, are added. --- ## Release 3.2 ### StarRocks version 3.2 #### 3.2.16[​](#3216 "Direct link to 3.2.16") Release Date: April 30, 2025 ##### Improvements[​](#improvements "Direct link to Improvements") * Stream Load task scheduling now supports BE node blacklist. Nodes in the blacklist will be excluded from task scheduling. [#57919](https://github.com/StarRocks/starrocks/pull/57919) ##### Bug Fixes[​](#bug-fixes "Direct link to Bug Fixes") Fixed the following issues: * Create tablet timeout. [#55808](https://github.com/StarRocks/starrocks/pull/55808) * Authentication information is lost when creating views via the `files()` function. [#56606](https://github.com/StarRocks/starrocks/pull/56606) * Optimizer failed to correctly handle constant comparisons when processing empty sets, leading to query failure. [#57735](https://github.com/StarRocks/starrocks/pull/57735) * Pre-aggregation strategy caused BE crashes when handling data overflow. [#58022](https://github.com/StarRocks/starrocks/pull/58022) * Attempting to delete associated materialized views after some partitions of the base table were deleted may cause exceptions, resulting in failure of the delete operation. [#58037](https://github.com/StarRocks/starrocks/pull/58037) * Defect in priority evaluation logic when loading Tablets for primary key tables, resulting in data loss due to incorrect version recognition. [#58404](https://github.com/StarRocks/starrocks/pull/58404) #### 3.2.15[​](#3215 "Direct link to 3.2.15") Release date: February 14, 2025 ##### New Features[​](#new-features "Direct link to New Features") * Window functions support `max_by` and `min_by`. [#54961](https://github.com/StarRocks/starrocks/pull/54961) ##### Improvements[​](#improvements-1 "Direct link to Improvements") * Added StarClient timeout parameters. [#54496](https://github.com/StarRocks/starrocks/pull/54496) * star\_client\_read\_timeout\_seconds * star\_client\_list\_timeout\_seconds * star\_client\_write\_timeout\_seconds * Tables with List partitioning strategies support partition pruning for DELETE statements. [#55400](https://github.com/StarRocks/starrocks/pull/55400) ##### Bug Fixes[​](#bug-fixes-1 "Direct link to Bug Fixes") Fixed the following issues: * Stream Load fails when a node with an Alive status of false was scheduled. [#55371](https://github.com/StarRocks/starrocks/pull/55371) * An error is returned during partial updates on Primary Key tables with Stream Load. [#53403](https://github.com/StarRocks/starrocks/pull/55430) * bRPC error persists after BE node restart. [#40229](https://github.com/StarRocks/starrocks/pull/40229) #### 3.2.14[​](#3214 "Direct link to 3.2.14") Release date: January 8, 2025 ##### Improvements[​](#improvements-2 "Direct link to Improvements") * Supports collecting statistics of Paimon tables. [#52858](https://github.com/StarRocks/starrocks/pull/52858) * Included node information and histogram metrics in JSON metrics. [#53735](https://github.com/StarRocks/starrocks/pull/53735) ##### Bug Fixes[​](#bug-fixes-2 "Direct link to Bug Fixes") Fixed the following issues: * The score of the Primary Key table index was not updated in the Commit phase. [#41737](https://github.com/StarRocks/starrocks/pull/41737) * Incorrect execution plans for `max(count(distinct))` when low-cardinality optimization is enabled. [#53403](https://github.com/StarRocks/starrocks/pull/53403) * When the List partition column has NULL values, queries against the Min/Max value of the partition column will lead to incorrect partition pruning. [#53235](https://github.com/StarRocks/starrocks/pull/53235) * Upload retries fail when backing up data to HDFS. [#53679](https://github.com/StarRocks/starrocks/pull/53679) #### 3.2.13[​](#3213 "Direct link to 3.2.13") Release date: December 13, 2024 ##### Improvements[​](#improvements-3 "Direct link to Improvements") * Supports setting a time range within which Base Compaction is forbidden for a specific table. [#50120](https://github.com/StarRocks/starrocks/pull/50120) ##### Bug Fixes[​](#bug-fixes-3 "Direct link to Bug Fixes") Fixed the following issues: * The `loadRowsRate` field returned `0` after executing SHOW ROUTINE LOAD. [#52151](https://github.com/StarRocks/starrocks/pull/52151) * The `Files()` function read columns that were not queried. [#52210](https://github.com/StarRocks/starrocks/pull/52210) * Prometheus failed to parse materialized view metrics with special characters in their names. (Now materialized view metrics support tags.) [#52782](https://github.com/StarRocks/starrocks/pull/52782) * The `array_map` function caused BE to crash. [#52909](https://github.com/StarRocks/starrocks/pull/52909) * Metadata Cache issues caused BE to crash. [#52968](https://github.com/StarRocks/starrocks/pull/52968) * Routine Load tasks were canceled due to expired transactions. (Now tasks are canceled only if the database or table no longer exists). [#50334](https://github.com/StarRocks/starrocks/pull/50334) * Stream Load failures when submitted using HTTP 1.0. [#53010](https://github.com/StarRocks/starrocks/pull/53010) [#53008](https://github.com/StarRocks/starrocks/pull/53008) * Issues related to Glue and S3 integration: [#48433](https://github.com/StarRocks/starrocks/pull/48433) * Some error messages did not display the root cause. * Error messages for writing to a Hive partitioned table with the partition column of type STRING when Glue was used as the metadata service. * Dropping Hive tables failed without proper error messages when the user lacked sufficient permissions. * The `storage_cooldown_time` property for materialized views did not take effect when set to `maximum`. [#52079](https://github.com/StarRocks/starrocks/pull/52079) #### 3.2.12[​](#3212 "Direct link to 3.2.12") Release date: October 23, 2024 ##### Improvements[​](#improvements-4 "Direct link to Improvements") * Optimized memory allocation and statistics in BE for certain complex query scenarios to avoid OOM. [#51382](https://github.com/StarRocks/starrocks/pull/51382) * Optimized memory usage in FE in Schema Change scenarios. [#50855](https://github.com/StarRocks/starrocks/pull/50855) * Optimized the job status display when querying the system-defined view `information_schema.routine_load_jobs` from Follower FE nodes. [#51763](https://github.com/StarRocks/starrocks/pull/51763) * Supports Backup and Restore of with the List partitioned tables. [#51993](https://github.com/StarRocks/starrocks/pull/51993) ##### Bug Fixes[​](#bug-fixes-4 "Direct link to Bug Fixes") Fixed the following issues: * The error message was lost after writing to Hive failed. [#33167](https://github.com/StarRocks/starrocks/pull/33167) * The `array_map` function causes a crash when excessive constant parameters are used. [#51244](https://github.com/StarRocks/starrocks/pull/51244) * Special characters in the PARTITION BY columns of expression partitioned tables cause FE CheckPoint failures. [#51677](https://github.com/StarRocks/starrocks/pull/51677) * Accessing the system-defined view `information_schema.fe_locks` causes a crash. [#51742](https://github.com/StarRocks/starrocks/pull/51742) * Querying generated columns causes an error. [#51755](https://github.com/StarRocks/starrocks/pull/51755) * Optimize Table fails when the table name contains special characters. [#51755](https://github.com/StarRocks/starrocks/pull/51755) * Tablets could not be balanced in certain scenarios. [#51828](https://github.com/StarRocks/starrocks/pull/51828) ##### Behavior Changes[​](#behavior-changes "Direct link to Behavior Changes") * Supports dynamic modification of Backup and Restore-related parameters.[#52111](https://github.com/StarRocks/starrocks/pull/52111) #### 3.2.11[​](#3211 "Direct link to 3.2.11") Release date: September 9, 2024 ##### Improvements[​](#improvements-5 "Direct link to Improvements") * Supports masking authentication information for Files() and PIPE. [#47629](https://github.com/StarRocks/starrocks/pull/47629) * Support automatic inference for the STRUCT type when reading Parquet files through Files(). [#50481](https://github.com/StarRocks/starrocks/pull/50481) ##### Bug Fixes[​](#bug-fixes-5 "Direct link to Bug Fixes") Fixed the following issues: * An error is returned for equi-join queries because they failed to be rewritten by the global dictionary. [#50690](https://github.com/StarRocks/starrocks/pull/50690) * The error "version has been compacted" caused by an infinite loop on the FE side during Tablet Clone. [#50561](https://github.com/StarRocks/starrocks/pull/50561) * Incorrect scheduling for unhealthy replica repairs after distributing data based on labels. [#50331](https://github.com/StarRocks/starrocks/pull/50331) * An error in the statistics collection log: "Unknown column '%s' in '%s." [#50785](https://github.com/StarRocks/starrocks/pull/50785) * Incorrect timezone usage when reading complex types like TIMESTAMP from Parquet files via Files(). [#50448](https://github.com/StarRocks/starrocks/pull/50448) ##### Behavior Changes[​](#behavior-changes-1 "Direct link to Behavior Changes") * When downgrading StarRocks from v3.3.x to v3.2.11, the system will ignore it if there is incompatible metadata. [#49636](https://github.com/StarRocks/starrocks/pull/49636) #### 3.2.10[​](#3210 "Direct link to 3.2.10") Release date: August 23, 2024 ##### Improvements[​](#improvements-6 "Direct link to Improvements") * Files() will automatically convert `BYTE_ARRAY` data with a `logical_type` of `JSON` in Parquet files to the JSON type in StarRocks. [#49385](https://github.com/StarRocks/starrocks/pull/49385) * Optimized error messages for Files() when Access Key ID and Secret Access Key are missing. [#49090](https://github.com/StarRocks/starrocks/pull/49090) * `information_schema.columns` supports the `GENERATION_EXPRESSION` field. [#49734](https://github.com/StarRocks/starrocks/pull/49734) ##### Bug Fixes[​](#bug-fixes-6 "Direct link to Bug Fixes") Fixed the following issues: * Downgrading a v3.3 shared-data cluster to v3.2 after setting the Primary Key table property `"persistent_index_type" = "CLOUD_NATIVE"` causes a crash. [#48149](https://github.com/StarRocks/starrocks/pull/48149) * Exporting data to CSV files using SELECT INTO OUTFILE may cause data inconsistency. [#48052](https://github.com/StarRocks/starrocks/pull/48052) * Queries encounter failures during concurrent query execution. [#48180](https://github.com/StarRocks/starrocks/pull/48180) * Queries would hang due to a timeout in the Plan phase without exiting. [#48405](https://github.com/StarRocks/starrocks/pull/48405) * After disabling index compression for Primary Key tables in older versions and then upgrading to v3.2.9, accessing `page_off` information causes an array out-of-bounds crash. [#48230](https://github.com/StarRocks/starrocks/pull/48230) * BE crash caused by concurrent execution of ADD/DROP COLUMN operations. [#49355](https://github.com/StarRocks/starrocks/pull/49355) * Queries against negative `TINYINT` values in ORC format files return `None` on the aarch64 architecture. [#49517](https://github.com/StarRocks/starrocks/pull/49517) * If the disk write operation fails, failures of `l0` snapshots for Primary Key Persistent Index may cause data loss. [#48045](https://github.com/StarRocks/starrocks/pull/48045) * Partial Update in Column mode for Primary Key tables fails under scenarios with large-volume data updates. [#49054](https://github.com/StarRocks/starrocks/pull/49054) * BE crash caused by Fast Schema Evolution when downgrading a v3.3.0 shared-data cluster to v3.2.9. [#42737](https://github.com/StarRocks/starrocks/pull/42737) * `partition_linve_nubmer` does not take effect. [#49213](https://github.com/StarRocks/starrocks/pull/49213) * The conflict between index persistence and compaction in Primary Key tables could cause clone failures. [#49341](https://github.com/StarRocks/starrocks/pull/49341) * Modifications of `partition_line_number` using ALTER TABLE do not take effect. [#49437](https://github.com/StarRocks/starrocks/pull/49437) * Rewrite of CTE distinct grouping sets generates an invalid plan. [#48765](https://github.com/StarRocks/starrocks/pull/48765) * RPC failures polluted the thread pool. [#49619](https://github.com/StarRocks/starrocks/pull/49619) * authentication failure issues when loading files from AWS S3 via PIPE. [#49837](https://github.com/StarRocks/starrocks/pull/49837) ##### Behavior Changes[​](#behavior-changes-2 "Direct link to Behavior Changes") * Added a check for the `meta` directory in the FE startup script. If the directory does not exist, it will be automatically created. [#48940](https://github.com/StarRocks/starrocks/pull/48940) * Added a memory limit parameter `load_process_max_memory_hard_limit_ratio` for data loading. If memory usage exceeds the limit, subsequent loading tasks will fail. [#48495](https://github.com/StarRocks/starrocks/pull/48495) #### 3.2.9[​](#329 "Direct link to 3.2.9") Release date: July 11, 2024 ##### New Features[​](#new-features-1 "Direct link to New Features") * Paimon tables now support DELETE Vectors. [#45866](https://github.com/StarRocks/starrocks/issues/45866) * Supports Column-level access control through Apache Ranger. [#47702](https://github.com/StarRocks/starrocks/pull/47702) * Stream Load can automatically convert JSON strings into STRUCT/MAP/ARRAY types during loading. [#45406](https://github.com/StarRocks/starrocks/pull/45406) * JDBC Catalog now supports Oracle and SQL Server. [#35691](https://github.com/StarRocks/starrocks/issues/35691) ##### Improvements[​](#improvements-7 "Direct link to Improvements") * Improved privilege management by restricting `user_admin` role users from resetting the password of the root user. [#47801](https://github.com/StarRocks/starrocks/pull/47801) * Stream Load now supports using `\t` and `\n` as row and column delimiters. Users do not need to convert them to their hexadecimal ASCII codes. [#47302](https://github.com/StarRocks/starrocks/pull/47302) * Optimized memory usage during data loading. [#47047](https://github.com/StarRocks/starrocks/pull/47047) * Supports masking authentication information for the Files() function in audit logs. [#46893](https://github.com/StarRocks/starrocks/pull/46893) * Hive tables now support the `skip.header.line.count` property. [#47001](https://github.com/StarRocks/starrocks/pull/47001) * JDBC Catalog supports more data types. [#47618](https://github.com/StarRocks/starrocks/pull/47618) ##### Behavior Changes[​](#behavior-changes-3 "Direct link to Behavior Changes") * Changed the value inheritance order of the `JAVA_OPTS` parameters. If versions other than JDK\_9 or JDK\_11 are used, users need to configure `JAVA_OPTS` directly. [#47495](https://github.com/StarRocks/starrocks/pull/47495) * When users create a non-partitioned table without specifying the bucket number, the minimum bucket number the system sets for the table is `16` (instead of `2` based on the formula `2*BE or CN count`). If users want to set a smaller bucket number when creating a small table, they must set it explicitly. [#47005](https://github.com/StarRocks/starrocks/pull/47005) * When users create a partitioned table without specifying the bucket number, if the number of partitions exceeds 5, the rule for setting the bucket count is changed to `max(2*BE or CN count, bucket number calculated based on the largest historical partition data volume)`. The previous rule was to calculate the bucket number based on the largest historical partition data volume. [#47949](https://github.com/StarRocks/starrocks/pull/47949) ##### Bug Fixes[​](#bug-fixes-7 "Direct link to Bug Fixes") Fixed the following issues: * BE crash caused by ALTER TABLE ADD COLUMN after upgrading a shared-data cluster from v3.2.x to v3.3.0 and then rolling it back. [#47826](https://github.com/StarRocks/starrocks/pull/47826) * Tasks initiated through SUBMIT TASK showed a Running status indefinitely in the QueryDetail interface. [#47619](https://github.com/StarRocks/starrocks/pull/47619) * Forwarding queries to the FE Leader node caused a null pointer exception. [#47559](https://github.com/StarRocks/starrocks/pull/47559) * SHOW MATERIALIZED VIEWS with WHERE conditions caused a null pointer exception. [#47811](https://github.com/StarRocks/starrocks/pull/47811) * Vertical Compaction fails for Primary Key tables in shared-data clusters. [#47192](https://github.com/StarRocks/starrocks/pull/47192) * Improper handling of I/O Error when sinking data to Hive or Iceberg tables. [#46979](https://github.com/StarRocks/starrocks/pull/46979) * Table properties do not take effect when whitespaces are added to their values. [#47119](https://github.com/StarRocks/starrocks/pull/47119) * BE crash caused by concurrent migration and Index Compaction operations on Primary Key tables. [#46675](https://github.com/StarRocks/starrocks/pull/46675) #### 3.2.8[​](#328 "Direct link to 3.2.8") Release date: June 7, 2024 ##### New Features[​](#new-features-2 "Direct link to New Features") * **[Supports adding labels on BEs](https://docs.starrocks.io/docs/3.2/administration/management/resource_management/be_label/)**: Supports adding labels on BEs based on information such as the racks and data centers where BEs are located. It ensures even data distribution among racks and data centers, and facilitates disaster recovery in case of power failures in certain racks or faults in data centers. [#38833](https://github.com/StarRocks/starrocks/pull/38833) ##### Bug Fixes[​](#bug-fixes-8 "Direct link to Bug Fixes") Fixed the following issues: * An error is returned when users DELETE data rows from tables that use the expression partitioning method with str2date. [#45939](https://github.com/StarRocks/starrocks/pull/45939) * BEs in the destination cluster crash when the StarRocks Cross-cluster Data Migration Tool fails to retrieve the Schema information from the source cluster. [#46068](https://github.com/StarRocks/starrocks/pull/46068) * The error `Multiple entries with same key` is returned to queries with non-deterministic functions. [#46602](https://github.com/StarRocks/starrocks/pull/46602) #### 3.2.7[​](#327 "Direct link to 3.2.7") Release date: May 24, 2024 ##### New Features[​](#new-features-3 "Direct link to New Features") * Stream Load supports data compression during transmission, reducing network bandwidth overhead. Users can specify different compression algorithms using parameters `compression` and `Content-Encoding`. Supported compression algorithms including GZIP, BZIP2, LZ4\_FRAME, and ZSTD. [#43732](https://github.com/StarRocks/starrocks/pull/43732) * Optimized the garbage collection (GC) mechanism in shared-data clusters. Supports manual compaction for tables or partitions stored in object storage. [#39532](https://github.com/StarRocks/starrocks/issues/39532) * Flink connector supports reading complex data types ARRAY, MAP, and STRUCT from StarRocks. [#42932](https://github.com/StarRocks/starrocks/pull/42932) [#347](https://github.com/StarRocks/starrocks-connector-for-apache-flink/pull/347) * Supports populating Data Cache asynchronously during queries, reducing the impact of populating cache on query performance. [#40489](https://github.com/StarRocks/starrocks/pull/40489) * ANALYZE TABLE supports collecting histograms for external tables, effectively addressing data skews. For more information, see [CBO statistics](https://docs.starrocks.io/docs/3.2/using_starrocks/Cost_based_optimizer/#collect-statistics-of-hiveiceberghudi-tables). [#42693](https://github.com/StarRocks/starrocks/pull/42693) * Lateral Join with [UNNEST](https://docs.starrocks.io/docs/3.2/sql-reference/sql-functions/array-functions/unnest/) supports LEFT JOIN. [#43973](https://github.com/StarRocks/starrocks/pull/43973) * Query Pool supports configuring memory usage threshold that triggers spilling via BE static parameter `query_pool_spill_mem_limit_threshold`. Once the threshold is reached, intermediate results of queries will be spilled to disks to reduce memory usage, thus avoiding OOM. * Supports creating asynchronous materialized views based on Hive views. ##### Improvements[​](#improvements-8 "Direct link to Improvements") * Optimized the error message returned for Broker Load tasks when there is no data under the specified HDFS paths. [#43839](https://github.com/StarRocks/starrocks/pull/43839) * Optimized the error message returned when the Files function is used to read data from AWS S3 without Access Key and Secret Key specified. [#42450](https://github.com/StarRocks/starrocks/pull/42450) * Optimized the error message returned for Broker Load tasks that load no data to any partitions. [#44292](https://github.com/StarRocks/starrocks/pull/44292) * Optimized the error message returned for INSERT INTO SELECT tasks when the column count of the destination table does not match that in the SELECT statement. [#44331](https://github.com/StarRocks/starrocks/pull/44331) ##### Bug Fixes[​](#bug-fixes-9 "Direct link to Bug Fixes") Fixed the following issues: * Concurrent read or write of the BITMAP-type data may cause BE to crash. [#44167](https://github.com/StarRocks/starrocks/pull/44167) * Primary key indexes may cause BE to crash. [#43793](https://github.com/StarRocks/starrocks/pull/43793) [#43569](https://github.com/StarRocks/starrocks/pull/43569) [#44034](https://github.com/StarRocks/starrocks/pull/44034) * Under high query concurrency scenarios, the str\_to\_map function may cause BE to crash. [#43901](https://github.com/StarRocks/starrocks/pull/43901) * When the Masking policy of Apache Ranger is used, an error is returned when table aliases are specified in queries. [#44445](https://github.com/StarRocks/starrocks/pull/44445) * In shared-data clusters, query execution cannot be routed to a backup node when the current node encounters exceptions. The corresponding error message is optimized for this issue. [#43489](https://github.com/StarRocks/starrocks/pull/43489) * Memory information is incorrect in the container environment. [#43225](https://github.com/StarRocks/starrocks/issues/43225) * An exception is thrown when INSERT tasks are canceled. [#44239](https://github.com/StarRocks/starrocks/pull/44239) * Expression-based dynamic partitions cannot be automatically created. [#44163](https://github.com/StarRocks/starrocks/pull/44163) * Creating partitions may cause FE deadlock. [#44974](https://github.com/StarRocks/starrocks/pull/44974) #### 3.2.6[​](#326 "Direct link to 3.2.6") Release date: April 18, 2024 ##### Bug Fixes[​](#bug-fixes-10 "Direct link to Bug Fixes") Fixed the following issue: * The privileges of external tables cannot be found due to incompatibility issues. [#44030](https://github.com/StarRocks/starrocks/pull/44030) #### 3.2.5 (Yanked)[​](#325-yanked "Direct link to 3.2.5 (Yanked)") Release date: April 12, 2024 tip This version has been taken offline due to privilege issues in querying external tables in external catalogs such as Hive and Iceberg. * **Problem**: When a user queries data from an external table in an external catalog, access to this table is denied even when the user has the SELECT privilege on this table. SHOW GRANTS also shows that the user has this privilege. * **Impact scope**: This problem only affects queries on external tables in external catalogs. Other queries are not affected. * **Temporary workaround**: The query succeeds after the SELECT privilege on this table is granted to the user again. But `SHOW GRANTS` will return duplicate privilege entries. After an upgrade to v3.2.6, users can run `REVOKE` to remove one of the privilege entries. ##### New Features[​](#new-features-4 "Direct link to New Features") * Supports the [dict\_mapping](https://docs.starrocks.io/docs/3.2/sql-reference/sql-functions/dict-functions/dict_mapping/) column property, which can significantly facilitate the loading process during the construction of a global dictionary, accelerating the exact COUNT DISTINCT calculation. ##### Behavior Changes[​](#behavior-changes-4 "Direct link to Behavior Changes") * When null values in JSON data are evaluated based on the `IS NULL` operator, they are considered NULL values following SQL language. For example, `true` is returned for `SELECT parse_json('{"a": null}') -> 'a' IS NULL` (before this behavior change, `false` is returned). [#42765](https://github.com/StarRocks/starrocks/pull/42765) ##### Improvements[​](#improvements-9 "Direct link to Improvements") * Optimized the column type unionization rules for automatic schema detection in the FILES table function. When columns with the same name but different types exist in separate files, FILES will attempt to merge them by selecting the type with the larger granularity as the final type. For example, if there are columns with the same name but of types FLOAT and INT respectively, FILES will return DOUBLE as the final type. [#40959](https://github.com/StarRocks/starrocks/pull/40959) * Primary Key tables support Size-tiered Compaction to reduce the I/O amplification. [#41130](https://github.com/StarRocks/starrocks/pull/41130) * When Broker Load is used to load data from ORC files that contain TIMESTAMP-type data, StarRocks supports retaining microseconds in the timestamps when converting the timestamps to match its own DATETIME data type. [#42179](https://github.com/StarRocks/starrocks/pull/42179) * Optimized the error messages for Routine Load. [#41306](https://github.com/StarRocks/starrocks/pull/41306) * Optimized the error messages when the FILES table function is used to convert invalid data types. [#42717](https://github.com/StarRocks/starrocks/pull/42717) ##### Bug Fixes[​](#bug-fixes-11 "Direct link to Bug Fixes") Fixed the following issues: * FEs fail to start after system-defined views are dropped. Dropping system-defined views is now prohibited. [#43552](https://github.com/StarRocks/starrocks/pull/43552) * BEs crash when duplicate sort key columns exist in Primary Key tables. Duplicate sort key columns are now prohibited. [#43206](https://github.com/StarRocks/starrocks/pull/43206) * An error, instead of NULL, is returned when the input value of the to\_json() function is NULL. [#42171](https://github.com/StarRocks/starrocks/pull/42171) * In shared-data mode, the garbage collection and thread eviction mechanisms for handling persistent indexes created on Primary Key tables cannot take effect on CN nodes. As a result, obsolete data cannot be deleted. [#41955](https://github.com/StarRocks/starrocks/pull/41955) * In shared-data mode, an error is returned when users modify the `enable_persistent_index` property of a Primary Key table. [#42890](https://github.com/StarRocks/starrocks/pull/42890) * In shared-data mode, NULL values are given to columns that are not supposed to be changed when users update a Primary Key table with partial updates in column mode. [#42355](https://github.com/StarRocks/starrocks/pull/42355) * Queries cannot be rewritten with asynchronous materialized views created on logical views. [#42173](https://github.com/StarRocks/starrocks/pull/42173) * CNs crash when the Cross-cluster Data Migration Tool is used to migrate Primary Key tables to a shared-data cluster. [#42260](https://github.com/StarRocks/starrocks/pull/42260) * The partition ranges of the external catalog-based asynchronous materialized views are not consecutive. [#41957](https://github.com/StarRocks/starrocks/pull/41957) #### 3.2.4 (Yanked)[​](#324-yanked "Direct link to 3.2.4 (Yanked)") Release date: March 12, 2024 tip This version has been taken offline due to privilege issues in querying external tables in external catalogs such as Hive and Iceberg. * **Problem**: When a user queries data from an external table in an external catalog, access to this table is denied even when the user has the SELECT privilege on this table. SHOW GRANTS also shows that the user has this privilege. * **Impact scope**: This problem only affects queries on external tables in external catalogs. Other queries are not affected. * **Temporary workaround**: The query succeeds after the SELECT privilege on this table is granted to the user again. But `SHOW GRANTS` will return duplicate privilege entries. After an upgrade to v3.2.6, users can run `REVOKE` to remove one of the privilege entries. ##### New Features[​](#new-features-5 "Direct link to New Features") * Cloud-native Primary Key tables in shared-data clusters support Size-tiered Compaction to reduce the write I/O amplification. [#41034](https://github.com/StarRocks/starrocks/pull/41034) * Added the date function `milliseconds_diff`. [#38171](https://github.com/StarRocks/starrocks/pull/38171) * Added the session variable `catalog`, which specifies the catalog to which the session belongs. [#41329](https://github.com/StarRocks/starrocks/pull/41329) * Supports [setting user-defined variables in hints](https://docs.starrocks.io/docs/3.2/administration/Query_planning/#user-defined-variable-hint). [#40746](https://github.com/StarRocks/starrocks/pull/40746) * Supports CREATE TABLE LIKE in Hive catalogs. [#37685](https://github.com/StarRocks/starrocks/pull/37685) * Added the view `information_schema.partitions_meta`, which records detailed metadata of partitions. [#39265](https://github.com/StarRocks/starrocks/pull/39265) * Added the view `sys.fe_memory_usage`, which records the memory usage for StarRocks. [#40464](https://github.com/StarRocks/starrocks/pull/40464) ##### Behavior Changes[​](#behavior-changes-5 "Direct link to Behavior Changes") * `cbo_decimal_cast_string_strict` is used to control how CBO converts data from the DECIMAL type to the STRING type. The default value `true` indicates that the logic built in v2.5.x and later versions prevails and the system implements strict conversion (namely, the system truncates the generated string and fills 0s based on the scale length). The DECIMAL type is not strictly filled in earlier versions, causing different results when comparing the DECIMAL type and the STRING type. [#40619](https://github.com/StarRocks/starrocks/pull/40619) * The default value of the Iceberg Catalog parameter `enable_iceberg_metadata_cache` has been changed to `false`. From v3.2.1 to v3.2.3, this parameter is set to `true` by default, regardless of what metastore service is used. In v3.2.4 and later, if the Iceberg cluster uses AWS Glue as metastore, this parameter still defaults to `true`. However, if the Iceberg cluster uses other metastore service such as Hive metastore, this parameter defaults to `false`. [#41826](https://github.com/StarRocks/starrocks/pull/41826) * The user who can refresh materialized views is changed from the `root` user to the user who creates the materialized views. This change does not affect existing materialized views. [#40670](https://github.com/StarRocks/starrocks/pull/40670) * By default, when comparing columns of constant and string types, StarRocks compares them as strings. Users can use the session variable `cbo_eq_base_type` to adjust the rule used for the comparison. For example, users can set `cbo_eq_base_type` to `decimal`, and StarRocks then compares the columns as numeric values. [#40619](https://github.com/StarRocks/starrocks/pull/40619) ##### Improvements[​](#improvements-10 "Direct link to Improvements") * Shared-data StarRocks clusters support the Partitioned Prefix feature for S3-compatible object storage systems. When this feature is enabled, StarRocks stores the data into multiple, uniformly prefixed partitions (sub-paths) under the bucket. This improves the read and write efficiency on data files in S3-compatible object storages. [#41627](https://github.com/StarRocks/starrocks/pull/41627) * StarRocks supports using the parameter `s3_compatible_fs_list` to specify which S3-compatible object storage can be accessed via AWS SDK, and supports using the parameter `fallback_to_hadoop_fs_list` to specify non-S3-compatible object storages that require access via HDFS Schema (this method requires the use of vendor-provided JAR packages). [#41123](https://github.com/StarRocks/starrocks/pull/41123) * Optimized compatibility with Trino. Supports syntax conversion from the following Trino functions: current\_catalog, current\_schema, to\_char, from\_hex, to\_date, to\_timestamp, and index. [#41217](https://github.com/StarRocks/starrocks/pull/41217) [#41319](https://github.com/StarRocks/starrocks/pull/41319) [#40803](https://github.com/StarRocks/starrocks/pull/40803) * Optimized the query rewrite logic of materialized views. StarRocks can rewrite queries with materialized views created upon logical views. [#42173](https://github.com/StarRocks/starrocks/pull/42173) * Improved the efficiency of converting the STRING type to the DATETIME type by 35% to 40%. [#41464](https://github.com/StarRocks/starrocks/pull/41464) * The `agg_type` of BITMAP-type columns in an Aggregate table can be set to `replace_if_not_null` in order to support updates only to a few columns of the table. [#42034](https://github.com/StarRocks/starrocks/pull/42034) * Improved the Broker Load performance when loading small ORC files. [#41765](https://github.com/StarRocks/starrocks/pull/41765) * The tables with hybrid row-column storage support Schema Change. [#40851](https://github.com/StarRocks/starrocks/pull/40851) * The tables with hybrid row-column storage support complex types including BITMAP, HLL, JSON, ARRAY, MAP, and STRUCT. [#41476](https://github.com/StarRocks/starrocks/pull/41476) * A new internal SQL log file is added to record log data related to statistics and materialized views. [#40453](https://github.com/StarRocks/starrocks/pull/40453) ##### Bug Fixes[​](#bug-fixes-12 "Direct link to Bug Fixes") Fixed the following issues: * "Analyze Error" is thrown if inconsistent letter cases are assigned to the names or aliases of tables or views queried in the creation of a Hive view. [#40921](https://github.com/StarRocks/starrocks/pull/40921) * I/O usage reaches the upper limit if persistent indexes are created on Primary Key tables. [#39959](https://github.com/StarRocks/starrocks/pull/39959) * In shared-data clusters, primary key index directories are deleted every 5 hours. [#40745](https://github.com/StarRocks/starrocks/pull/40745) * After users execute ALTER TABLE COMPACT by hand, the memory usage statistics for compaction operations are abnormal. [#41150](https://github.com/StarRocks/starrocks/pull/41150) * Retries of the Publish phase may hang for Primary Key tables. [#39890](https://github.com/StarRocks/starrocks/pull/39890) #### 3.2.3[​](#323 "Direct link to 3.2.3") Release date: February 8, 2024 ##### New Features[​](#new-features-6 "Direct link to New Features") * \[Preview] Supports hybrid row-column storage for tables. It allows better performance for high concurrency, low-latency point lookups against Primary Key tables and partial data updates. Currently, this feature does not support modification via ALTER TABLE, changing Sort Key, and partial updates in column mode. * Supports backing up and restoring asynchronous materialized views. * Broker Load supports loading JSON-type data. * Supports query rewrite using asynchronous materialized views created upon views. Queries against a view can be rewritten based on materialized views that are created upon that view. * Supports CREATE OR REPLACE PIPE. [#37658](https://github.com/StarRocks/starrocks/pull/37658) ##### Behavior Changes[​](#behavior-changes-6 "Direct link to Behavior Changes") * Added the session variable `enable_strict_order_by`. When this variable is set to the default value `TRUE`, an error is reported for such a query pattern: Duplicate alias is used in different expressions of the query and this alias is also a sorting field in ORDER BY, for example, `select distinct t1.* from tbl1 t1 order by t1.k1;`. The logic is the same as that in v2.3 and earlier. When this variable is set to `FALSE`, a loose deduplication mechanism is used, which processes such queries as valid SQL queries. [#37910](https://github.com/StarRocks/starrocks/pull/37910) * Added the session variable `enable_materialized_view_for_insert`, which controls whether materialized views rewrite the queries in INSERT INTO SELECT statements. The default value is `false`. [#37505](https://github.com/StarRocks/starrocks/pull/37505) * When a single query is executed within the Pipeline framework, its memory limit is now constrained by the variable `query_mem_limit` instead of `exec_mem_limit`. Setting the value of `query_mem_limit` to `0` indicates no limit. [#34120](https://github.com/StarRocks/starrocks/pull/34120) ##### Parameter Changes[​](#parameter-changes "Direct link to Parameter Changes") * Added the FE configuration item `http_worker_threads_num`, which specifies the number of threads for HTTP server to deal with HTTP requests. The default value is `0`. If the value for this parameter is set to a negative value or `0`, the actual thread number is twice the number of CPU cores. [#37530](https://github.com/StarRocks/starrocks/pull/37530) * Added the BE configuration item `lake_pk_compaction_max_input_rowsets`, which controls the maximum number of input rowsets allowed in a Primary Key table compaction task in a shared-data StarRocks cluster. This helps optimize resource consumption for compaction tasks. [#39611](https://github.com/StarRocks/starrocks/pull/39611) * Added the session variable `connector_sink_compression_codec`, which specifies the compression algorithm used for writing data into Hive tables or Iceberg tables, or exporting data with Files(). Valid algorithms include GZIP, BROTLI, ZSTD, and LZ4. [#37912](https://github.com/StarRocks/starrocks/pull/37912) * Added the FE configuration item `routine_load_unstable_threshold_second`. [#36222](https://github.com/StarRocks/starrocks/pull/36222) * Added the BE configuration item `pindex_major_compaction_limit_per_disk` to configure the maximum concurrency of compaction on a disk. This addresses the issue of uneven I/O across disks due to compaction. This issue can cause excessively high I/O for certain disks. The default value is `1`. [#36681](https://github.com/StarRocks/starrocks/pull/36681) * Added the BE configuration item `enable_lazy_delta_column_compaction`. The default value is `true`, indicating that StarRocks does not perform frequent compaction operations on delta columns. [#36654](https://github.com/StarRocks/starrocks/pull/36654) * Added the FE configuration item `default_mv_refresh_immediate`, which specifies whether to immediately refresh the materialized view after the materialized view is created. The default value is `true`. [#37093](https://github.com/StarRocks/starrocks/pull/37093) * Changed the default value of the FE configuration item `default_mv_refresh_partition_num`to `1`. This indicates that when multiple partitions need to be updated during a materialized view refresh, the task will be split in batches, refreshing only one partition at a time. This helps reduce resource consumption during each refresh. [#36560](https://github.com/StarRocks/starrocks/pull/36560) * Changed the default value of the BE/CN configuration item `starlet_use_star_cache` to `true`. This indicates that Data Cache is enabled by default in shared-data clusters. If, before upgrading, you have manually configured the BE/CN configuration item `starlet_cache_evict_high_water` to `X`, you must configure the BE/CN configuration item `starlet_star_cache_disk_size_percent` to `(1.0 - X) * 100`. For example, if you have set `starlet_cache_evict_high_water` to `0.3` before upgrading, you must set `starlet_star_cache_disk_size_percent` to `70`. This ensures that both file data cache and Data Cache will not exceed the disk capacity limit. [#38200](https://github.com/StarRocks/starrocks/pull/38200) ##### Improvements[​](#improvements-11 "Direct link to Improvements") * Added date formats `yyyy-MM-ddTHH:mm` and `yyyy-MM-dd HH:mm` to support TIMESTAMP partition fields in Apache Iceberg tables. [#39986](https://github.com/StarRocks/starrocks/pull/39986) * Added Data Cache-related metrics to the monitoring API. [#40375](https://github.com/StarRocks/starrocks/pull/40375) * Optimized BE log printing to prevent too many irrelevant logs. [#22820](https://github.com/StarRocks/starrocks/pull/22820) [#36187](https://github.com/StarRocks/starrocks/pull/36187) * Added the field `storage_medium` to the view `information_schema.be_tablets`. [#37070](https://github.com/StarRocks/starrocks/pull/37070) * Supports `SET_VAR` in multiple sub-queries. [#36871](https://github.com/StarRocks/starrocks/pull/36871) * A new field `LatestSourcePosition` is added to the return result of SHOW ROUTINE LOAD to record the position of the latest message in each partition of the Kafka topic, helping check the latencies of data loading. [#38298](https://github.com/StarRocks/starrocks/pull/38298) * When the string on the right side of the LIKE operator within the WHERE clause does not include `%` or `_`, the LIKE operator is converted into the `=` operator. [#37515](https://github.com/StarRocks/starrocks/pull/37515) * The default retention period of trash files is changed to 1 day from the original 3 days. [#37113](https://github.com/StarRocks/starrocks/pull/37113) * Supports collecting statistics from Iceberg tables with Partition Transform. [#39907](https://github.com/StarRocks/starrocks/pull/39907) * The scheduling policy for Routine Load is optimized, so that slow tasks do not block the execution of the other normal tasks. [#37638](https://github.com/StarRocks/starrocks/pull/37638) ##### Bug Fixes[​](#bug-fixes-13 "Direct link to Bug Fixes") Fixed the following issues: * The execution of ANALYZE TABLE gets stuck occasionally. [#36836](https://github.com/StarRocks/starrocks/pull/36836) * The memory consumption by PageCache exceeds the threshold specified by the BE dynamic parameter `storage_page_cache_limit` in certain circumstances. [#37740](https://github.com/StarRocks/starrocks/pull/37740) * Hive metadata in Hive catalogs is not automatically refreshed when new fields are added to Hive tables. [#37549](https://github.com/StarRocks/starrocks/pull/37549) * In some cases, `bitmap_to_string` may return incorrect results due to data type overflow. [#37405](https://github.com/StarRocks/starrocks/pull/37405) * When `SELECT ... FROM ... INTO OUTFILE` is executed to export data into CSV files, the error "Unmatched number of columns" is reported if the FROM clause contains multiple constants. [#38045](https://github.com/StarRocks/starrocks/pull/38045) * In some cases, querying semi-structured data in tables may cause BEs to crash. [#40208](https://github.com/StarRocks/starrocks/pull/40208) #### 3.2.2[​](#322 "Direct link to 3.2.2") Release date: December 30, 2023 ##### Bug Fixes[​](#bug-fixes-14 "Direct link to Bug Fixes") Fixed the following issue: * When StarRocks is upgraded from v3.1.2 or earlier to v3.2, FEs may fail to restart. [#38172](https://github.com/StarRocks/starrocks/pull/38172) #### 3.2.1[​](#321 "Direct link to 3.2.1") Release date: December 21, 2023 ##### New Features[​](#new-features-7 "Direct link to New Features") ###### Data Lake Analytics[​](#data-lake-analytics "Direct link to Data Lake Analytics") * Supports reading [Hive Catalog](https://docs.starrocks.io/docs/3.2/data_source/catalog/hive_catalog/) tables and file external tables in Avro, SequenceFile, and RCFile formats through Java Native Interface (JNI). ###### Materialized View[​](#materialized-view "Direct link to Materialized View") * Added a view `object_dependencies` to the database `sys`. It contains the lineage information of asynchronous materialized views. [#35060](https://github.com/StarRocks/starrocks/pull/35060) * Supports creating synchronous materialized views with the WHERE clause. * Supports partition-level incremental refresh for asynchronous materialized views created upon Iceberg catalogs. * \[Preview] Supports creating asynchronous materialized views based on tables in a Paimon catalog with partition-level refresh. ###### Query and SQL functions[​](#query-and-sql-functions "Direct link to Query and SQL functions") * Supports the prepared statement. It allows better performance for processing high concurrency point lookup queries. It also prevents SQL injection effectively. * Supports the following Bitmap functions: [subdivide\_bitmap](https://docs.starrocks.io/docs/3.2/sql-reference/sql-functions/bitmap-functions/subdivide_bitmap/), [bitmap\_from\_binary](https://docs.starrocks.io/docs/3.2/sql-reference/sql-functions/bitmap-functions/bitmap_from_binary/), and [bitmap\_to\_binary](https://docs.starrocks.io/docs/3.2/sql-reference/sql-functions/bitmap-functions/bitmap_to_binary/). * Supports the Array function [array\_unique\_agg](https://docs.starrocks.io/docs/3.2/sql-reference/sql-functions/array-functions/array_unique_agg/). ###### Monitoring and alerts[​](#monitoring-and-alerts "Direct link to Monitoring and alerts") * Added a new metric `max_tablet_rowset_num` for setting the maximum allowed number of rowsets. This metric helps detect possible compaction issues and thus reduces the occurrences of the error "too many versions". [#36539](https://github.com/StarRocks/starrocks/pull/36539) ##### Parameter change[​](#parameter-change "Direct link to Parameter change") * A new BE configuration item `enable_stream_load_verbose_log` is added. The default value is `false`. With this parameter set to `true`, StarRocks can record the HTTP requests and responses for Stream Load jobs, making troubleshooting easier. [#36113](https://github.com/StarRocks/starrocks/pull/36113) ##### Improvements[​](#improvements-12 "Direct link to Improvements") * Upgraded the default GC algorithm in JDK8 to G1. [#37268](https://github.com/StarRocks/starrocks/pull/37268) * A new value option `GROUP_CONCAT_LEGACY` is added to the session variable [sql\_mode](https://docs.starrocks.io/docs/3.2/sql-reference/System_variable/#sql_mode) to provide compatibility with the implementation logic of the [group\_concat](https://docs.starrocks.io/docs/3.2/sql-reference/sql-functions/string-functions/group_concat/) function in versions earlier than v2.5. [#36150](https://github.com/StarRocks/starrocks/pull/36150) * The authentication information `aws.s3.access_key` and `aws.s3.access_secret` for [AWS S3 in Broker Load jobs](https://docs.starrocks.io/docs/3.2/loading/s3/) are hidden in audit logs. [#36571](https://github.com/StarRocks/starrocks/pull/36571) * The `be_tablets` view in the `information_schema` database provides a new field `INDEX_DISK`, which records the disk usage (measured in bytes) of persistent indexes. [#35615](https://github.com/StarRocks/starrocks/pull/35615) * The result returned by the [SHOW ROUTINE LOAD](https://docs.starrocks.io/docs/3.2/sql-reference/sql-statements/data-manipulation/SHOW_ROUTINE_LOAD/) statement provides a new field `OtherMsg`, which shows information about the last failed task. [#35806](https://github.com/StarRocks/starrocks/pull/35806) ##### Bug Fixes[​](#bug-fixes-15 "Direct link to Bug Fixes") Fixed the following issues: * The BEs crash if users create persistent indexes in the event of data corruption.[#30841](https://github.com/StarRocks/starrocks/pull/30841) * The [array\_distinct](https://docs.starrocks.io/docs/3.2/sql-reference/sql-functions/array-functions/array_distinct/) function occasionally causes the BEs to crash. [#36377](https://github.com/StarRocks/starrocks/pull/36377) * After the DISTINCT window operator pushdown feature is enabled, errors are reported if SELECT DISTINCT operations are performed on the complex expressions of the columns computed by window functions. [#36357](https://github.com/StarRocks/starrocks/pull/36357) * Some S3-compatible object storage returns duplicate files, causing the BEs to crash. [#36103](https://github.com/StarRocks/starrocks/pull/36103) #### 3.2.0[​](#320 "Direct link to 3.2.0") Release date: December 1, 2023 ##### New Features[​](#new-features-8 "Direct link to New Features") ###### Shared-data cluster[​](#shared-data-cluster "Direct link to Shared-data cluster") * Supports persisting indexes of [Primary Key tables](https://docs.starrocks.io/docs/3.2/table_design/table_types/primary_key_table/) to local disks. * Supports even distribution of Data Cache among multiple local disks. ###### Materialized View[​](#materialized-view-1 "Direct link to Materialized View") **Asynchronous materialized view** * The Query Dump file can include information of asynchronous materialized views. * The Spill to Disk feature is enabled by default for the refresh tasks of asynchronous materialized views, reducing memory consumption. ###### Data Lake Analytics[​](#data-lake-analytics-1 "Direct link to Data Lake Analytics") * Supports creating and dropping databases and managed tables in [Hive catalogs](https://docs.starrocks.io/docs/3.2/data_source/catalog/hive_catalog/), and supports exporting data to Hive's managed tables using INSERT or INSERT OVERWRITE. * Supports [Unified Catalog](https://docs.starrocks.io/docs/3.2/data_source/catalog/unified_catalog/), with which users can access different table formats (Hive, Iceberg, Hudi, and Delta Lake) that share a common metastore like Hive metastore or AWS Glue. * Supports collecting statistics of Hive and Iceberg tables using ANALYZE TABLE, and storing the statistics in StarRocks, thus facilitating optimization of query plans and accelerating subsequent queries. * Supports Information Schema for external tables, providing additional convenience for interactions between external systems (such as BI tools) and StarRocks. ###### Storage engine, data ingestion, and export[​](#storage-engine-data-ingestion-and-export "Direct link to Storage engine, data ingestion, and export") * Added the following features of loading with the table function [FILES()](https://docs.starrocks.io/docs/3.2/sql-reference/sql-functions/table-functions/files/): * Loading Parquet and ORC format data from Azure or GCP. * Extracting the value of a key/value pair from the file path as the value of a column using the parameter `columns_from_path`. * Loading complex data types including ARRAY, JSON, MAP, and STRUCT. * Supports unloading data from StarRocks to Parquet-formatted files stored in AWS S3 or HDFS by using INSERT INTO FILES. For detailed instructions, see [Unload data using INSERT INTO FILES](https://docs.starrocks.io/docs/3.2/unloading/unload_using_insert_into_files/). * Supports [manual optimization of table structure and data distribution strategy](https://docs.starrocks.io/docs/3.2/table_design/Data_distribution#optimize-data-distribution-after-table-creation-since-32) used in an existing table to optimize the query and loading performance. You can set a new bucket key, bucket number, or sort key for a table. You can also set a different bucket number for specific partitions. * Supports continuous data loading from [AWS S3](https://docs.starrocks.io/docs/3.2/loading/s3/#use-pipe) or [HDFS](https://docs.starrocks.io/docs/3.2/loading/hdfs_load/#use-pipe) using the PIPE method. * When PIPE detects new or modifications in a remote storage directory, it can automatically load the new or modified data into the destination table in StarRocks. While loading data, PIPE automatically splits a large loading task into smaller, serialized tasks, enhancing stability in large-scale data ingestion scenarios and reducing the cost of error retries. ###### Query[​](#query "Direct link to Query") * Supports [HTTP SQL API](https://docs.starrocks.io/docs/3.2/reference/HTTP_API/SQL/), enabling users to access StarRocks data via HTTP and execute SELECT, SHOW, EXPLAIN, or KILL operations. * Supports Runtime Profile and text-based Profile analysis commands (SHOW PROFILELIST, ANALYZE PROFILE, EXPLAIN ANALYZE) to allow users to directly analyze profiles via MySQL clients, facilitating bottleneck identification and discovery of optimization opportunities. ###### SQL reference[​](#sql-reference "Direct link to SQL reference") Added the following functions: * String functions: substring\_index, url\_extract\_parameter, url\_encode, url\_decode, and translate * Date functions: dayofweek\_iso, week\_iso, quarters\_add, quarters\_sub, milliseconds\_add, milliseconds\_sub, date\_diff, jodatime\_format, str\_to\_jodatime, to\_iso8601, to\_tera\_date, and to\_tera\_timestamp * Pattern matching function: regexp\_extract\_all * hash function: xx\_hash3\_64 * Aggregate functions: approx\_top\_k * Window functions: cume\_dist, percent\_rank and session\_number * Utility functions: get\_query\_profile and is\_role\_in\_session ###### Privileges and security[​](#privileges-and-security "Direct link to Privileges and security") StarRocks supports access control through [Apache Ranger](https://docs.starrocks.io/docs/3.2/administration/ranger_plugin/), providing a higher level of data security and allowing the reuse of existing services of external data sources. After integrating with Apache Ranger, StarRocks enables the following access control methods: * When accessing internal tables, external tables, or other objects in StarRocks, access control can be enforced based on the access policies configured for the StarRocks Service in Ranger. * When accessing an external catalog, access control can also leverage the corresponding Ranger service of the original data source (such as Hive Service) to control access (currently, access control for exporting data to Hive is not yet supported). For more information, see [Manage permissions with Apache Ranger](https://docs.starrocks.io/docs/3.2/administration/ranger_plugin/). ##### Improvements[​](#improvements-13 "Direct link to Improvements") ###### Data Lake Analytics[​](#data-lake-analytics-2 "Direct link to Data Lake Analytics") * Optimized ORC Reader: * Optimized the ORC Column Reader, resulting in nearly a two-fold performance improvement for VARCHAR and CHAR data reading. * Optimized the decompression performance of ORC files in Zlib compression format. * Optimized Parquet Reader: * Supports adaptive I/O merging, allowing adaptive merging of columns with and without predicates based on filtering effects, thus reducing I/O. * Optimized Dict Filter for faster predicate rewriting. Supports STRUCT sub-columns, and on-demand dictionary column decoding. * Optimized Dict Decode performance. * Optimized late materialization performance. * Supports caching file footers to avoid repeated computation overhead. * Supports decompression of Parquet files in lzo compression format. * Optimized CSV Reader: * Optimized the Reader performance. * Supports decompression of CSV files in Snappy and lzo compression formats. * Optimized the performance of the count calculation. * Optimized Iceberg Catalog capabilities: * Supports collecting column statistics from Manifest files to accelerate queries. * Supports collecting NDV (number of distinct values) from Puffin files to accelerate queries. * Supports partition pruning. * Reduced Iceberg metadata memory consumption to enhance stability in scenarios with large metadata volume or high query concurrency. ###### Materialized View[​](#materialized-view-2 "Direct link to Materialized View") **Asynchronous materialized view** * Supports automatic refresh for an asynchronous materialized view created upon views or materialized views when schema changes occur on the views, materialized views, or their base tables. * Data consistency: * Added the property `query_rewrite_consistency` for asynchronous materialized view creation. This property defines the query rewrite rules based on the consistency check. * Add the property `force_external_table_query_rewrite` for external catalog-based asynchronous materialized view creation. This property defines whether to allow force query rewrite for asynchronous materialized views created upon external catalogs. * For detailed information, see [CREATE MATERIALIZED VIEW](https://docs.starrocks.io/docs/3.2/sql-reference/sql-statements/data-definition/CREATE_MATERIALIZED_VIEW/). * Added a consistency check for materialized views' partitioning key. * When users create an asynchronous materialized view with window functions that include a PARTITION BY expression, the partitioning column of the window function must match that of the materialized view. ###### Storage engine, data ingestion, and export[​](#storage-engine-data-ingestion-and-export-1 "Direct link to Storage engine, data ingestion, and export") * Optimized the persistent index for Primary Key tables by improving memory usage logic while reducing I/O read and write amplification. [#24875](https://github.com/StarRocks/starrocks/pull/24875) [#27577](https://github.com/StarRocks/starrocks/pull/27577) [#28769](https://github.com/StarRocks/starrocks/pull/28769) * Supports data re-distribution across local disks for Primary Key tables. * Partitioned tables support automatic cooldown based on the partition time range and cooldown time. Compared to the original cooldown logic, it is more convenient to perform hot and cold data management on the partition level. For more information, see [Specify initial storage medium, automatic storage cooldown time, replica number](https://docs.starrocks.io/docs/3.2/sql-reference/sql-statements/data-definition/CREATE_TABLE#specify-initial-storage-medium-automatic-storage-cooldown-time-replica-number). * The Publish phase of a load job that writes data into a Primary Key table is changed from asynchronous mode to synchronous mode. As such, the data loaded can be queried immediately after the load job finishes. For more information, see [enable\_sync\_publish](https://docs.starrocks.io/docs/3.2/administration/FE_configuration#enable_sync_publish). * Supports Fast Schema Evolution, which is controlled by the table property [`fast_schema_evolution`](https://docs.starrocks.io/docs/3.2/sql-reference/sql-statements/data-definition/CREATE_TABLE#set-fast-schema-evolution). After this feature is enabled, the execution efficiency of adding or dropping columns is significantly improved. This mode is disabled by default (Default value is `false`). You cannot modify this property for existing tables using ALTER TABLE. * [Supports dynamically adjusting the number of tablets to create](https://docs.starrocks.io/docs/3.2/table_design/Data_distribution#set-the-number-of-buckets) according to cluster information and the size of the data for **Duplicate Key** tables created with the Radom Bucketing strategy. ###### Query[​](#query-1 "Direct link to Query") * Optimized StarRocks' compatibility with Metabase and Superset. Supports integrating them with external catalogs. ###### SQL Reference[​](#sql-reference-1 "Direct link to SQL Reference") * [array\_agg](https://docs.starrocks.io/docs/3.2/sql-reference/sql-functions/array-functions/array_agg/) supports the keyword DISTINCT. * INSERT, UPDATE, and DELETE operations now support `SET_VAR`. [#35283](https://github.com/StarRocks/starrocks/pull/35283) ###### Others[​](#others "Direct link to Others") * Added the session variable `large_decimal_underlying_type = "panic"|"double"|"decimal"` to set the rules to deal with DECIMAL type overflow. `panic` indicates returning an error immediately, `double` indicates converting the data to DOUBLE type, and `decimal` indicates converting the data to DECIMAL(38,s). ##### Developer tools[​](#developer-tools "Direct link to Developer tools") * Supports Trace Query Profile for asynchronous materialized views, which can be used to analyze its transparent rewrite. ##### Behavior Change[​](#behavior-change "Direct link to Behavior Change") To be updated. ##### Parameter Change[​](#parameter-change-1 "Direct link to Parameter Change") ###### FE Parameters[​](#fe-parameters "Direct link to FE Parameters") * Added the following FE configuration items: * `catalog_metadata_cache_size` * `enable_backup_materialized_view` * `enable_colocate_mv_index` * `enable_fast_schema_evolution` * `json_file_size_limit` * `lake_enable_ingest_slowdown` * `lake_ingest_slowdown_threshold` * `lake_ingest_slowdown_ratio` * `lake_compaction_score_upper_bound` * `mv_auto_analyze_async` * `primary_key_disk_schedule_time` * `statistic_auto_collect_small_table_rows` * `stream_load_task_keep_max_num` * `stream_load_task_keep_max_second` * Removed FE configuration item `enable_pipeline_load`. * Default value modifications: * The default value of `enable_sync_publish` is changed from `false` to `true`. * The default value of `enable_persistent_index_by_default` is changed from `false` to `true`. ###### BE Parameters[​](#be-parameters "Direct link to BE Parameters") * Data Cache-related configuration changes. * Added `datacache_enable` to replace `block_cache_enable`. * Added `datacache_mem_size` to replace `block_cache_mem_size`. * Added `datacache_disk_size` to replace `block_cache_disk_size`. * Added `datacache_disk_path` to replace `block_cache_disk_path`. * Added `datacache_meta_path` to replace `block_cache_meta_path`. * Added `datacache_block_size` to replace `block_cache_block_size`. * Added `datacache_checksum_enable` to replace `block_cache_checksum_enable`. * Added `datacache_direct_io_enable` to replace `block_cache_direct_io_enable`. * Added `datacache_max_concurrent_inserts` to replace `block_cache_max_concurrent_inserts`. * Added `datacache_max_flying_memory_mb`. * Added `datacache_engine` to replace `block_cache_engine`. * Removed `block_cache_max_parcel_memory_mb`. * Removed `block_cache_report_stats`. * Removed `block_cache_lru_insertion_point`. After renaming Block Cache to Data Cache, StarRocks has introduced a new set of BE parameters prefixed with `datacache` to replace the original parameters prefixed with `block_cache`. After upgrade to v3.2, the original parameters will still be effective. Once enabled, the new parameters will override the original ones. The mixed usage of new and original parameters is not supported, as it may result in some configurations not taking effect. In the future, StarRocks plans to deprecate the original parameters with the `block_cache` prefix, so we recommend you use the new parameters with the `datacache` prefix. * Added the following BE configuration items: * `spill_max_dir_bytes_ratio` * `streaming_agg_limited_memory_size` * `streaming_agg_chunk_buffer_size` * Removed the following BE configuration items: * Dynamic parameter `tc_use_memory_min` * Dynamic parameter `tc_free_memory_rate` * Dynamic parameter `tc_gc_period` * Static parameter `tc_max_total_thread_cache_byte` * Default value modifications: * The default value of `disable_column_pool` is changed from `false` to `true`. * The default value of `thrift_port` is changed from `9060` to `0`. * The default value of `enable_load_colocate_mv` is changed from `false` to `true`. * The default value of `enable_pindex_minor_compaction` is changed from `false` to `true`. ###### System Variables[​](#system-variables "Direct link to System Variables") * Added the following session variables: * `enable_per_bucket_optimize` * `enable_write_hive_external_table` * `hive_temp_staging_dir` * `spill_revocable_max_bytes` * `thrift_plan_protocol` * Removed the following session variables: * `enable_pipeline_query_statistic` * `enable_deliver_batch_fragments` * Renamed the following session variables: * `enable_scan_block_cache` is renamed as `enable_scan_datacache`. * `enable_populate_block_cache` is renamed as `enable_populate_datacache`. ###### Reserved Keywords[​](#reserved-keywords "Direct link to Reserved Keywords") Added reserved keywords `OPTIMIZE` and `PREPARE`. ##### Bug Fixes[​](#bug-fixes-16 "Direct link to Bug Fixes") Fixed the following issues: * BEs crash when libcurl is invoked. [#31667](https://github.com/StarRocks/starrocks/pull/31667) * Schema Change may fail if it takes an excessively long period of time, because the specified tablet version is handled by garbage collection. [#31376](https://github.com/StarRocks/starrocks/pull/31376) * Failed to access the Parquet files in MinIO via file external tables. [#29873](https://github.com/StarRocks/starrocks/pull/29873) * The ARRAY, MAP, and STRUCT type columns are not correctly displayed in `information_schema.columns`. [#33431](https://github.com/StarRocks/starrocks/pull/33431) * An error is reported if specific path formats are used during data loading via Broker Load: `msg:Fail to parse columnsFromPath, expected: [rec_dt]`. [#32720](https://github.com/StarRocks/starrocks/pull/32720) * `DATA_TYPE` and `COLUMN_TYPE` for BINARY or VARBINARY data types are displayed as `unknown` in the `information_schema.columns` view. [#32678](https://github.com/StarRocks/starrocks/pull/32678) * Complex queries that involve many unions, expressions, and SELECT columns can result in a sudden surge in the bandwidth or CPU usage within an FE node. * The refresh of asynchronous materialized view may occasionally encounter deadlock. [#35736](https://github.com/StarRocks/starrocks/pull/35736) ##### Upgrade Notes[​](#upgrade-notes "Direct link to Upgrade Notes") * Optimization on **Random Bucketing** is disabled by default. To enable it, you need to add the property `bucket_size` when creating tables. This allows the system to dynamically adjust the number of tablets based on cluster information and the size of loaded data. Please note that once this optimization is enabled, if you need to roll back your cluster to v3.1 or earlier, you must delete tables with this optimization enabled and manually execute a metadata checkpoint (by executing `ALTER SYSTEM CREATE IMAGE`). Otherwise, the rollback will fail. * Starting from v3.2.0, StarRocks has disabled non-Pipeline queries. Therefore, before upgrading your cluster to v3.2, you need to globally enable the Pipeline engine (by adding the configuration `enable_pipeline_engine=true` in the FE configuration file **fe.conf**). Failure to do so will result in errors for non-Pipeline queries. --- ## Release 3.3 ### StarRocks version 3.3 warning * After upgrading StarRocks to v3.3, DO NOT downgrade it directly to v3.2.0, v3.2.1, or v3.2.2, otherwise it will cause metadata loss. You must downgrade the cluster to v3.2.3 or later to prevent the issue. * After upgrading StarRocks to v3.3.9, you can only downgrade it to v3.2.11 or later. #### 3.3.22[​](#3322 "Direct link to 3.3.22") Release Date: January 27, 2026 ##### Bug Fixes[​](#bug-fixes "Direct link to Bug Fixes") The following issues have been fixed: * CVE-2025-27818. [#67335](https://github.com/StarRocks/starrocks/pull/67335) * SIGSEGV crash in CN when querying non-partitioned Iceberg tables with DATE/TIME predicates on ARM64/Graviton architectures. [#66864](https://github.com/StarRocks/starrocks/pull/66864) * Deadlock caused by lock ordering issues when closing `LocalTabletsChannel` and `LakeTabletsChannel`. [#66748](https://github.com/StarRocks/starrocks/pull/66748) * Potential BE crash when executing `CACHE SELECT` queries with filter conditions. [#67375](https://github.com/StarRocks/starrocks/pull/67375) * Issue where the Multicast Sink Operator could get stuck in the `OUTPUT_FULL` state if an upstream operator (for example, LIMIT) finished early, causing the query to hang. [#67153](https://github.com/StarRocks/starrocks/pull/67153) * Potential Segfault caused by failure to invalidate cache pointers when `ObjectColumn` is resized or moved. [#66957](https://github.com/StarRocks/starrocks/pull/66957) * Potential errors or OOM issues when Java UDFs handle Nullable columns containing all NULLs. [#67025](https://github.com/StarRocks/starrocks/pull/67025) * BE crash caused by the optimization logic of Ranking Window Functions when `PARTITION BY` and `ORDER BY` are missing. [#67081](https://github.com/StarRocks/starrocks/pull/67081) * Incorrect result issue where `COUNT(DISTINCT)` was not correctly rewritten to `multi_distinct_count` when queried alongside non-distinct aggregations (like SUM) on a single-bucket table. [#66767](https://github.com/StarRocks/starrocks/pull/66767) * Incorrect results when `regexp_replace` processes multiple rows with `enable_hyperscan_vec` enabled. [#67380](https://github.com/StarRocks/starrocks/pull/67380) * Potential incorrect results with Sorted Streaming Aggregate in shared-data mode. [#67376](https://github.com/StarRocks/starrocks/pull/67376) * Query failure where the optimizer generated access paths using the old column name after the column was renamed. [#67533](https://github.com/StarRocks/starrocks/pull/67533) * Incorrect Bitmap column type propagation in the rewrite rule from `bitmap_to_array` to `unnest_bitmap`. [#66855](https://github.com/StarRocks/starrocks/pull/66855) * Dependency derivation error in Low Cardinality optimization logic by adopting the Union-Find algorithm to correctly handle column relationships. [#66724](https://github.com/StarRocks/starrocks/pull/66724) * "Compute node not found" error in Short-circuit Read under shared-data clusters by adding a fallback mechanism to non-short-circuit mode. [#67323](https://github.com/StarRocks/starrocks/pull/67323) * "Version not found" error during Replication publishing caused by FE Replicas not updating the minimum readable version. [#67538](https://github.com/StarRocks/starrocks/pull/67538) * Logic error in physical partition comparison during replication transactions to ensure deterministic comparison using ID order. [#67616](https://github.com/StarRocks/starrocks/pull/67616) * Inaccurate statistics for counters like scan rows in Cloud Native Tables. [#67307](https://github.com/StarRocks/starrocks/pull/67307) * Issue where expired Tablets were not cleaned up from the scheduler, causing the scheduling queue to pile up. [#66718](https://github.com/StarRocks/starrocks/pull/66718) * Inaccurate SQL statements displayed in the Profile when multiple statements are submitted. [#67097](https://github.com/StarRocks/starrocks/pull/67097) * Issue where the transaction ID was empty in `publish_version` logs in the new FE. [#66732](https://github.com/StarRocks/starrocks/pull/66732) * Performance issue caused by unnecessary Protobuf message copying after `set_allocated`. [#67844](https://github.com/StarRocks/starrocks/pull/67844) #### 3.3.21[​](#3321 "Direct link to 3.3.21") Release Date: December 25, 2025 ##### Bug Fixes[​](#bug-fixes-1 "Direct link to Bug Fixes") The following issues have been fixed: * Logic errors when the `trim` function handles specific Unicode whitespace characters (for example, `\u1680`) and performance issues caused by reserved memory calculation. [#66428](https://github.com/StarRocks/starrocks/pull/66428) [#66477](https://github.com/StarRocks/starrocks/pull/66477) * Foreign key constraints are lost after FE restart due to table loading order. [#66474](https://github.com/StarRocks/starrocks/pull/66474) * Security vulnerabilities CVE-2025-66566 and CVE-2025-12183 in `lz4-java`, and potential crashes in upstream. [#66453](https://github.com/StarRocks/starrocks/pull/66453) [#66362](https://github.com/StarRocks/starrocks/pull/66362) [#67075](https://github.com/StarRocks/starrocks/pull/67075) * Incorrect results when Join is used with window functions in Group Execution mode. [#66441](https://github.com/StarRocks/starrocks/pull/66441) * System continues attempting to fetch metadata for a deleted warehouse, causing `SHOW LOAD` or SQL execution failures. [#66436](https://github.com/StarRocks/starrocks/pull/66436) * `PartitionColumnMinMaxRewriteRule` optimization incorrectly returns an empty set instead of NULL when the Scan input for aggregation is fully filtered. [#66356](https://github.com/StarRocks/starrocks/pull/66356) * Rowset IDs are not properly released when Rowset Commit or Compaction fails, preventing disk space reclamation. [#66301](https://github.com/StarRocks/starrocks/pull/66301) * Missing scan statistics (for example, scanned rows/bytes) in audit logs when a high-selectivity filter causes the Scan to end early (EOS). [#66280](https://github.com/StarRocks/starrocks/pull/66280) * BE continues to respond to heartbeats as Alive after entering the crash handling process (for example, SIGSEGV), causing the FE to continue dispatching queries and reporting errors [#66212](https://github.com/StarRocks/starrocks/pull/66212) * BE crash caused by multiple calls to `set_collector` of the Runtime Filter due to Local TopN pushdown optimization. [#66199](https://github.com/StarRocks/starrocks/pull/66199) * Load task failures caused by un-initialized `rssid` when column-mode Partial Update is used with Conditional Update. [#66139](https://github.com/StarRocks/starrocks/pull/66139) * Race condition when submitting drivers in Colocate Execution Group, leading to BE crashes. [#66099](https://github.com/StarRocks/starrocks/pull/66099) * `MemoryScratchSinkOperator` remains in a pending state and cannot be cancelled after `RecordBatchQueue` is closed (For example, triggered by SparkSQL Limit), causing queries to hang. [#66041](https://github.com/StarRocks/starrocks/pull/66041) * Use-after-free issue caused by a race condition when accessing `_num_pipelines` in the `ExecutionGroup` countdown logic. [#65940](https://github.com/StarRocks/starrocks/pull/65940) * Incorrect calculation logic for query error rate monitoring metrics (Internal/Analysis/Timeout error rate), resulting in negative values. [#65891](https://github.com/StarRocks/starrocks/pull/65891) * Null Pointer Exception (NPE) caused by unset `ConnectContext` when executing tasks as an LDAP user. [#65843](https://github.com/StarRocks/starrocks/pull/65843) * Performance issues where Filesystem Cache lookups for the same Key fail because object reference comparison is used instead of value comparison. [#65823](https://github.com/StarRocks/starrocks/pull/65823) * Tablet-related files are not correctly cleaned up after a snapshot load failure due to incorrect status variable checking. [#65709](https://github.com/StarRocks/starrocks/pull/65709) * Compression type and level configured in FE are not correctly propagated to BE or persisted during table creation or Schema Change, resulting in default compression settings. [#65673](https://github.com/StarRocks/starrocks/pull/65673) * `COM_STMT_EXECUTE` has no audit logs by default, and Profile information is incorrectly merged into the Prepare stage. [#65448](https://github.com/StarRocks/starrocks/pull/65448) * Delete Vector CRC32 check failures during cluster upgrade/downgrade scenarios due to version incompatibility. [#65442](https://github.com/StarRocks/starrocks/pull/65442) * BE crash caused by improper handling of Nullable properties when `UnionConstSourceOperator` merges Union to Values. [#65429](https://github.com/StarRocks/starrocks/pull/65429) * Concurrent load tasks fail during the Commit phase if the target tablet is dropped during an ALTER TABLE operation. [#65396](https://github.com/StarRocks/starrocks/pull/65396) * Inaccurate error log information when authentication fails due to incorrect context setting if users are switched in the HTTP SQL interface. [#65371](https://github.com/StarRocks/starrocks/pull/65371) * Statistics collection issues after `INSERT OVERWRITE`, including failure to collect statistics for temporary partitions and inaccurate row count statistics at transaction commit preventing collection trigger. [#65327](https://github.com/StarRocks/starrocks/pull/65327), [#65298](https://github.com/StarRocks/starrocks/pull/65298), [#65225](https://github.com/StarRocks/starrocks/pull/65225) * `HttpConnectContext` related to SQL is not released in TCP connection reuse scenarios because subsequent non-SQL HTTP requests overwrite the context. [#65203](https://github.com/StarRocks/starrocks/pull/65203) * Potential version loss caused by incomplete tablet loading when RocksDB iteration times out during BE startup metadata loading. [#65146](https://github.com/StarRocks/starrocks/pull/65146) * BE crash caused by out-of-bounds access when processing compression parameters in the `percentile_approx_weighted` function. [#64838](https://github.com/StarRocks/starrocks/pull/64838) * Missing Query Profile logs for queries forwarded from Follower to Leader. [#64395](https://github.com/StarRocks/starrocks/pull/64395) * BE crash caused by lax size checks during LZ4 encoding of large string columns during spilling. [#61495](https://github.com/StarRocks/starrocks/pull/61495) * `MERGING-EXCHANGE` operator crash caused by Ranking window function optimization generating an empty `ORDER BY` when there is no `PARTITION BY` and `GROUP BY`. [#67081](https://github.com/StarRocks/starrocks/pull/67081) * Unstable results or errors in the Low Cardinality column rewrite logic due to dependence on the Set iteration order. [#66724](https://github.com/StarRocks/starrocks/pull/66724) #### 3.3.20[​](#3320 "Direct link to 3.3.20") Release Date: November 18, 2025 ##### Bug Fixes[​](#bug-fixes-2 "Direct link to Bug Fixes") The following issues have been fixed: * CVE-2024-47561. [#64193](https://github.com/StarRocks/starrocks/pull/64193) * CVE-2025-59419. [#64142](https://github.com/StarRocks/starrocks/pull/64142) * Incorrect row count for lake Primary Key tables. [#64007](https://github.com/StarRocks/starrocks/pull/64007) * Window function with IGNORE NULLS flags can not be consolidated with its counterpart without IGNORE NULLS flag. [#63958](https://github.com/StarRocks/starrocks/pull/63958) * ASAN error in `PartitionedSpillerWriter::_remove_partition`. [#63903](https://github.com/StarRocks/starrocks/pull/63903) * Wrong results for sorted aggregation in shared-data clusters. [#63849](https://github.com/StarRocks/starrocks/pull/63849) * NPE when creating a partitioned materialized view. [#63830](https://github.com/StarRocks/starrocks/pull/63830) * Partitioned Spill crash when removing partitions. [#63825](https://github.com/StarRocks/starrocks/pull/63825) * NPE when removing expired load jobs in FE. [#63820](https://github.com/StarRocks/starrocks/pull/63820) * A potential deadlock during initialization of `ExceptionStackContext`. [#63776](https://github.com/StarRocks/starrocks/pull/63776) * Degraded scan performance caused by the profitless simplification of CASE WHEN with complex functions. [#63732](https://github.com/StarRocks/starrocks/pull/63732) * Materialized view rewrite failures caused by type mismatch. [#63659](https://github.com/StarRocks/starrocks/pull/63659) * Materialized view rewrite throws `IllegalStateException` under certain plans. [#63655](https://github.com/StarRocks/starrocks/pull/63655) * LZ4 compression and decompression errors cannot be perceived. [#63629](https://github.com/StarRocks/starrocks/pull/63629) * Stability issue caused by incorrect overflow detection when casting LARGEINT to DECIMAL128 at sign-edge cases (for example, INT128\_MIN) [#63559](https://github.com/StarRocks/starrocks/pull/63559) * `date_trunc` partition pruning with combined predicates that mistakenly produced EMPTYSET. [#63464](https://github.com/StarRocks/starrocks/pull/63464) * Incomplete `Left Join` results caused by ARRAY low-cardinality optimization. [#63419](https://github.com/StarRocks/starrocks/pull/63419) * An issue caused by the aggregate intermediate type uses `ARRAY`. [#63371](https://github.com/StarRocks/starrocks/pull/63371) * Metadata inconsistency in partial updates based on auto-increment columns. [#63370](https://github.com/StarRocks/starrocks/pull/63370) * Incompatible Bitmap index reuse for Fast Schema Evolution in shared-data clusters. [#63315](https://github.com/StarRocks/starrocks/pull/63315) * Unnecessary CN deregistration during pod restart/upgrade. [#63085](https://github.com/StarRocks/starrocks/pull/63085) * Profiles showing SQL as `omit` for returns of the PREPARE/EXECUTE statements. [#62988](https://github.com/StarRocks/starrocks/pull/62988) #### 3.3.19[​](#3319 "Direct link to 3.3.19") Release Date: October 14, 2025 ##### Bug Fixes[​](#bug-fixes-3 "Direct link to Bug Fixes") The following issues have been fixed: * `UserProperty` had lower priority than Session Variables. [#63173](https://github.com/StarRocks/starrocks/pull/63173) * Materialized view refresh failures that could occur when the Hive base table was dropped and recreated. [#63072](https://github.com/StarRocks/starrocks/pull/63072) * Issues with the aggregation pushdown rewrite rule. [#63060](https://github.com/StarRocks/starrocks/pull/63060) * Inconsistencies between null columns and data columns in Boolean extraction functions for JSON. [#63054](https://github.com/StarRocks/starrocks/pull/63054) * Issues when getting partition columns in Delta Lake format tables. [#62953](https://github.com/StarRocks/starrocks/pull/62953) * Lack of colocation support for materialized views in shared-data clusters. [#62941](https://github.com/StarRocks/starrocks/pull/62941) * Projection mapping errors in view-based materialized view rewrite. [#62918](https://github.com/StarRocks/starrocks/pull/62918) * SQL syntax errors in histogram statistics when Most Common Values (MCV) contained single quotes. [#62853](https://github.com/StarRocks/starrocks/pull/62853) * `KILL ANALYZE` did not work. [#62842](https://github.com/StarRocks/starrocks/pull/62842) * CVE-2025-58056 vulnerability. [#62801](https://github.com/StarRocks/starrocks/pull/62801) * Executing `SHOW CREATE ROUTINE LOAD` without specifying a database causes wrong results. [#62745](https://github.com/StarRocks/starrocks/pull/62745) * Data loss caused by incorrectly skipping CSV headers in `files()`. [#62719](https://github.com/StarRocks/starrocks/pull/62719) * Version check failures when Replication and Compaction transactions were committed together. [#62663](https://github.com/StarRocks/starrocks/pull/62663) * Materialized view refresh is skipped because the materialized view version map is not cleared after a failed restore job. [#62634](https://github.com/StarRocks/starrocks/pull/62634) * Issues caused by case-sensitive partition column validation in the materialized view analyzer. [#62598](https://github.com/StarRocks/starrocks/pull/62598) #### 3.3.18[​](#3318 "Direct link to 3.3.18") Release Date: August 28, 2025 ##### Bug Fixes[​](#bug-fixes-4 "Direct link to Bug Fixes") The following issues have been fixed: * BE crashes when `LakePersistentIndex` initialization failed due to cleanup of `_memtable`. [#62279](https://github.com/StarRocks/starrocks/pull/62279) * A concurrency issue caused by missing locks when retrieving the maximum Tablet version in the replication transaction manager. [#62238](https://github.com/StarRocks/starrocks/pull/62238) * A hang issue in the phased scheduler, which waited indefinitely during synchronous Profile collection (after the fix, the system correctly terminates Profile collection when scheduling errors occur). [#62140](https://github.com/StarRocks/starrocks/pull/62140) * Exception handling issues in low-cardinality optimization under the `ALLOW_THROW_EXCEPTION` mode (after the fix, exceptions in expression evaluation are properly caught and returned). [#62098](https://github.com/StarRocks/starrocks/pull/62098) * FThe system failed to compute nested CTE statistics outside of the memo during table pruning when `enable_rbo_table_prune` was set to `false`. [#62070](https://github.com/StarRocks/starrocks/pull/62070) * CVE-2025-55163 issue. [#62041](https://github.com/StarRocks/starrocks/pull/62041) * An issue where `split_morsel_queue` nested inside `partition_morsel_queue` failed to correctly receive the Tablet Schema. [#62034](https://github.com/StarRocks/starrocks/pull/62034) * Incorrect handling of `NULL` arrays during Parquet writes, which could cause data inconsistency or crashes (after the fix, the system ensures the `split` function can correctly handle `NULL` input strings). [#61999](https://github.com/StarRocks/starrocks/pull/61999) * Failure when creating materialized views using `CASE WHEN` expressions due to incompatible return types of VARCHAR (after the fix, the system ensures consistency before and after refresh). [#61996](https://github.com/StarRocks/starrocks/pull/61996) * A concurrency safety issue caused by long operations holding shard-level locks while calculating compression scores. [#61899](https://github.com/StarRocks/starrocks/pull/61899) * An incomplete table pruning issue in CBO caused by pruning logic not considering all relevant predicates. [#61881](https://github.com/StarRocks/starrocks/pull/61881) #### 3.3.17[​](#3317 "Direct link to 3.3.17") Release Date: July 30, 2025 ##### Bug Fixes[​](#bug-fixes-5 "Direct link to Bug Fixes") The following issues have been fixed: * Upgraded HttpClient5 to 5.4.3. [#61298](https://github.com/StarRocks/starrocks/pull/61298) * Incorrect `cpu_core_used_permille` limit in resource groups. [#61177](https://github.com/StarRocks/starrocks/pull/61177) * Conflict between ALTER jobs and partition creation tasks. [#61167](https://github.com/StarRocks/starrocks/pull/61167) * NPE caused by missing `globalStateMgr` in `ConnectContext`. [#60880](https://github.com/StarRocks/starrocks/pull/60880) * Partition creation failed when partition names matched case-insensitively but had different values. [#60909](https://github.com/StarRocks/starrocks/pull/60909) * Lock competition caused by synchronous access to partition statistics. [#61041](https://github.com/StarRocks/starrocks/pull/61041) * ANALYZE tasks stuck in `pending` state after FE restart. [#61113](https://github.com/StarRocks/starrocks/pull/61113) * Issue with JIT (Just-In-Time) compilation in BE. [#61060](https://github.com/StarRocks/starrocks/pull/61060) * Leader address issue in Starmgr. [#61016](https://github.com/StarRocks/starrocks/pull/61016) * CVE vulnerabilities in Broker. [#60908](https://github.com/StarRocks/starrocks/pull/60908) * Actual number of JDBC connections exceeded `jdbc_connection_pool_size` limit. [#61004](https://github.com/StarRocks/starrocks/pull/61004) * CVE-2022-41404 vulnerability. [#59689](https://github.com/StarRocks/starrocks/pull/59689) * CVEs related to Parquet and HttpClient5. [#58750](https://github.com/StarRocks/starrocks/pull/58750) * Partition not removed from `_partition_map` when physical partition ID was empty. [#60842](https://github.com/StarRocks/starrocks/pull/60842) * Missing version check in shared-data clusters. [#59422](https://github.com/StarRocks/starrocks/pull/59422) * Transaction log missing when publishing logs in batches in shared-data clusters. [#60949](https://github.com/StarRocks/starrocks/pull/60949) * Concurrent publishing of the same transaction when Batch Publish is enabled in shared-data clusters. [#57574](https://github.com/StarRocks/starrocks/pull/57574) * Statistics overwrite issue caused by lack of semi-synchronous mode. [#60897](https://github.com/StarRocks/starrocks/pull/60897) * Inaccurate `maxInstantTime` used for filtering Hudi files when retrieving latest merged file slices. [#60927](https://github.com/StarRocks/starrocks/pull/60927) * TaskRun state incompatible with earlier versions. [#60438](https://github.com/StarRocks/starrocks/pull/60438) * CVE-2025-52999 vulnerability. [#60795](https://github.com/StarRocks/starrocks/pull/60795) * Vulnerability caused by `log4j-1.2.17-cloudera6` in Broker. [#59579](https://github.com/StarRocks/starrocks/pull/59579) * BE crash when loading OOM partitions. [#60778](https://github.com/StarRocks/starrocks/pull/60778) * Base Compaction tasks blocking other compaction tasks. [#60711](https://github.com/StarRocks/starrocks/pull/60711) * Inefficient handling of error string truncation. [#60878](https://github.com/StarRocks/starrocks/pull/60878) * Materialized view rewrite failed in multi-FE environments. [#60841](https://github.com/StarRocks/starrocks/pull/60841) * INSERT OVERWRITE failed on manually created partitions. [#60750](https://github.com/StarRocks/starrocks/pull/60750) * Issue caused by using random distribution in aggregate keys. [#60702](https://github.com/StarRocks/starrocks/pull/60702) * Crash caused by low cardinality rewrite in `multi_distinct_count`. [#60664](https://github.com/StarRocks/starrocks/pull/60664) * Issue with Pivot resolving fields. [#60748](https://github.com/StarRocks/starrocks/pull/60748) * Upgraded `hudi-common` to 1.0.2. [#59501](https://github.com/StarRocks/starrocks/pull/59501) * BE crash when CLONE and DROP TABLE run concurrently. [#61359](https://github.com/StarRocks/starrocks/pull/61359) #### 3.3.16[​](#3316 "Direct link to 3.3.16") Release Date: July 4, 2025 ##### Improvements[​](#improvements "Direct link to Improvements") * Optimized error logs when creating Hive tables with duplicate names. [#60076](https://github.com/StarRocks/starrocks/pull/60076) * Added the FE parameter `slow_lock_print_stack` to prevent process stalls in large clusters when printing thread stacks. [#59967](https://github.com/StarRocks/starrocks/pull/59967) * Reduced unnecessary locks during tablet scheduling. [#59744](https://github.com/StarRocks/starrocks/pull/59744) ##### Bug Fixes[​](#bug-fixes-6 "Direct link to Bug Fixes") Fixed the following issues: * SplitOR fails to prune scan columns. [#60223](https://github.com/StarRocks/starrocks/pull/60223) * Incorrect query plan for null-aware left anti joins. [#60119](https://github.com/StarRocks/starrocks/pull/60119) * Incorrect query results when rewriting queries with materialized views due to missing NULL partitions. [#60087](https://github.com/StarRocks/starrocks/pull/60087) * Partition pruning errors when tables contain empty partitions. [#60162](https://github.com/StarRocks/starrocks/pull/60162) * Refresh errors on Iceberg external tables when using partition expressions based on `str2date`. [#60089](https://github.com/StarRocks/starrocks/pull/60089) * Unexpected behavior caused by materialized view schema changes. [#60079](https://github.com/StarRocks/starrocks/pull/60079) * Issues related to low-cardinality global dictionaries in UNION operators. [#60075](https://github.com/StarRocks/starrocks/pull/60075) * Incorrect partition ranges for temporary partitions created using the START END syntax. [#60014](https://github.com/StarRocks/starrocks/pull/60014) * Lock issues with SUBMIT TASK. [#60026](https://github.com/StarRocks/starrocks/pull/60026) * Partial updates fail on Primary Key tables under certain conditions. [#60052](https://github.com/StarRocks/starrocks/pull/60052) * Crashes caused by BE failing to create directories due to a lack of permissions to access storage paths. [#60028](https://github.com/StarRocks/starrocks/pull/60028) * Cache failures due to cache key duplication in concurrent scenarios. [#60053](https://github.com/StarRocks/starrocks/pull/60053) * Hive table metadata background refresh failure in Unified Catalog. [#55215](https://github.com/StarRocks/starrocks/pull/55215) * Query failures caused by incorrect return types of CASE WHEN. [#59972](https://github.com/StarRocks/starrocks/pull/59972) * Query failures when Delta Lake tables UNION themselves. [#60030](https://github.com/StarRocks/starrocks/pull/60030) * Partition creation failure when writing to multiple tables within the same transaction. [#59954](https://github.com/StarRocks/starrocks/pull/59954) * Queries could return empty results instead of errors when tablet versions were updated during execution. [#53060](https://github.com/StarRocks/starrocks/pull/53060) * Queries against modified columns in a table return null after upgrading to v3.4. [#59941](https://github.com/StarRocks/starrocks/pull/59941) * Authentication information is printed in logs. [#59907](https://github.com/StarRocks/starrocks/pull/59907) * Metadata refresh failures for external tables in Hive Catalog. [#54596](https://github.com/StarRocks/starrocks/pull/54596) * CACHE SELECT failures for tables after schema changes. [#59812](https://github.com/StarRocks/starrocks/pull/59812) * Broker Load could not recover after FE Leader shifts. [#59732](https://github.com/StarRocks/starrocks/pull/59732) * Stream Load failures when the target table name contains Chinese characters. [#59722](https://github.com/StarRocks/starrocks/pull/59722) * Incorrect query results in external tables due to search key hash collisions (affecting Iceberg/Delta/Paimon). [#59781](https://github.com/StarRocks/starrocks/pull/59781) #### 3.3.15[​](#3315 "Direct link to 3.3.15") Release Date: Jun 20, 2025 ##### Bug Fixes[​](#bug-fixes-7 "Direct link to Bug Fixes") Fixed the following issues: * Missing double quotes for string parameters in statistics INSERT statements. [#59713](https://github.com/StarRocks/starrocks/pull/59713) * Downgrade failure caused by Rollup tasks. [#59735](https://github.com/StarRocks/starrocks/pull/59735) * Incorrect function parameters in the result of `SHOW CREATE VIEW`. [#59714](https://github.com/StarRocks/starrocks/pull/59714) * A security issue where SQL statements with syntax errors exposed sensitive information in the Audit Log. [#59442](https://github.com/StarRocks/starrocks/pull/59442) * Error "Query version not found". [#59194](https://github.com/StarRocks/starrocks/pull/59194) * Failure to change data distribution using the `ALTER TABLE` statement. [#59360](https://github.com/StarRocks/starrocks/pull/59360) * An issue where root user processes were still visible when admin protection was enabled. [#59435](https://github.com/StarRocks/starrocks/pull/59435) * Failure of `INSERT OVERWRITE` into Hive. [#59469](https://github.com/StarRocks/starrocks/pull/59469) * Missing Tablet ID in the `max_tablet_rowset_num` log item. [#59467](https://github.com/StarRocks/starrocks/pull/59467) * An error caused by misconfigured Persistent Index parameters on a Duplicate table. [#56040](https://github.com/StarRocks/starrocks/pull/56040) * TaskRun history being archived on FE Follower nodes. [#59393](https://github.com/StarRocks/starrocks/pull/59393) * External catalog-based materialized view refresh errors. [#59369](https://github.com/StarRocks/starrocks/pull/59369) * Missing minimum version in Tablet information on shared-data clusters. [#59373](https://github.com/StarRocks/starrocks/pull/59373) * Abnormal maximum column unique ID in native tables of shared-data clusters due to version compatibility logic errors. [#59190](https://github.com/StarRocks/starrocks/pull/59190) * Materialized view refresh failure on Iceberg catalogs when the source Iceberg table is dropped and recreated, and manual refresh also fails after the materialized view is set to active. [#59287](https://github.com/StarRocks/starrocks/pull/59287) * Contamination of parameters in materialized view refresh tasks. [#59052](https://github.com/StarRocks/starrocks/pull/59052) * Data loss caused by Persistent Index when loading snapshot fails. [#59247](https://github.com/StarRocks/starrocks/pull/59247) * Issues caused when subcolumns of STRUCT appear in multiple predicates. [#59216](https://github.com/StarRocks/starrocks/pull/59216) * Query failure after renaming columns. [#59178](https://github.com/StarRocks/starrocks/pull/59178) * Loading failure due to multiple Stream Load requests. [#59181](https://github.com/StarRocks/starrocks/pull/59181) * Inability to refresh Hive table-based materialized views at the partition level in Unified Catalog. [#59139](https://github.com/StarRocks/starrocks/pull/59139) * Incorrect UNION plan causing FE out-of-memory (OOM). [#59030](https://github.com/StarRocks/starrocks/pull/59030) * Version loss during data loading. [#59006](https://github.com/StarRocks/starrocks/pull/59006) * Predicate loss when queries are rewritten to synchronous materialized views. [#58831](https://github.com/StarRocks/starrocks/pull/58831) * Issues with BITMAP/HLL/PERCENTILE data types in window functions. [#58776](https://github.com/StarRocks/starrocks/pull/58776) * Metadata changes to the external tables in Hive Catalog cannot be refreshed. [#54596](https://github.com/StarRocks/starrocks/pull/54596) ##### Behavior Changes[​](#behavior-changes "Direct link to Behavior Changes") * Introduced FE configuration parameter `task_runs_max_history_number` to control the number of historical TaskRuns retained in the `information_schema.task_runs` view, reducing memory usage. [#59161](https://github.com/StarRocks/starrocks/pull/59161) #### 3.3.14[​](#3314 "Direct link to 3.3.14") Release Date: May 14, 2025 ##### Improvements[​](#improvements-1 "Direct link to Improvements") * Optimized error messages for regex parsing failures. [#57904](https://github.com/StarRocks/starrocks/pull/57904) * Fixed security vulnerabilities [SNYK-JAVA-ORGJSON-5488379](https://security.snyk.io/vuln/SNYK-JAVA-ORGJSON-5488379) and [SNYK-JAVA-ORGJSON-5962464](https://security.snyk.io/vuln/SNYK-JAVA-ORGJSON-5962464). [#58425](https://github.com/StarRocks/starrocks/pull/58425) ##### Bug Fixes[​](#bug-fixes-8 "Direct link to Bug Fixes") Fixed the following issues: * Issues with the JSON data type in `first_value`/`last_value`/`lead`/`lag` window functions. [#58697](https://github.com/StarRocks/starrocks/pull/58697) * Deadlock caused by table-level locks from base tables during materialized view writes (after the bug fix, DB-level locks are used). [#58615](https://github.com/StarRocks/starrocks/pull/58615) * INSERT tasks hang when the target table is deleted. [#58603](https://github.com/StarRocks/starrocks/pull/58603) * Failure to change active/inactive state of materialized views with List partitions. [#58575](https://github.com/StarRocks/starrocks/pull/58575) * Incorrect `streaming_load_current_processing` metric. [#58565](https://github.com/StarRocks/starrocks/pull/58565) * Data version update errors caused by continuous loading and replica clone tasks. [#58513](https://github.com/StarRocks/starrocks/pull/58513) * Failed to refresh materialized views on external tables. [#58506](https://github.com/StarRocks/starrocks/pull/58506) * Incorrect `if()` results on ARM architecture. [#58455](https://github.com/StarRocks/starrocks/pull/58455) * Materialized view rewriting generated incorrect query plans. [#58487](https://github.com/StarRocks/starrocks/pull/58487) * Iceberg table metadata did not refresh automatically. [#58490](https://github.com/StarRocks/starrocks/pull/58490) * Incorrect query plan generated by `group_concat`. [#57908](https://github.com/StarRocks/starrocks/pull/57908) * Mass Tablet load failures caused by unhandled exceptions during loading. [#58393](https://github.com/StarRocks/starrocks/pull/58393) * Constant folding failed due to type mismatches while pruning List partitions with generated columns (after the bug fix, an implicit cast rule was added). [#54543](https://github.com/StarRocks/starrocks/pull/54543) * Mismatch between aggregate function return type and original column type (after the bug fix, the column type is `cast` to the function output type). [#58407](https://github.com/StarRocks/starrocks/pull/58407) * `broadcast_row_limit` set to 0 or below failed to prevent BROADCAST JOIN generation. [#58307](https://github.com/StarRocks/starrocks/pull/58307) * Broker Load used BE nodes that had already been blacklisted. [#58350](https://github.com/StarRocks/starrocks/pull/58350) * Asynchronous tasks persist in the background and cannot be dropped after manually cancelling materialized view refresh tasks. [#58310](https://github.com/StarRocks/starrocks/pull/58310) * Failed to create expression partitions with month or year granularity. [#58182](https://github.com/StarRocks/starrocks/pull/58182) * `ngram_search` generated invalid query plans. [#58190](https://github.com/StarRocks/starrocks/pull/58190) #### 3.3.13[​](#3313 "Direct link to 3.3.13") Release Date: April 22, 2025 ##### Improvements[​](#improvements-2 "Direct link to Improvements") * Added memory consumption metrics for queries in FE in audit logs and the QueryDetail interface. [#57731](https://github.com/StarRocks/starrocks/pull/57731) * Optimized the strategy for concurrent creation of expression partitions. [#57899](https://github.com/StarRocks/starrocks/pull/57899) * Added monitoring metrics for the number of active FE nodes. [#57857](https://github.com/StarRocks/starrocks/pull/57857) * The `information_schema.task_runs` view supports pushdown of the LIMIT clause. [#57404](https://github.com/StarRocks/starrocks/pull/57404) * Fixed several CVE issues. [#57705](https://github.com/StarRocks/starrocks/pull/57705) [#57620](https://github.com/StarRocks/starrocks/pull/57620) * Primary Key tables support retry during the PUBLISH stage, enhancing system disaster recovery capabilities. [#57354](https://github.com/StarRocks/starrocks/pull/57354) * Reduced memory consumption of Flat JSON. [#57357](https://github.com/StarRocks/starrocks/pull/57357) * The `information_schema.routine_load_jobs` view adds the `timestamp_progress` column, consistent with the SHOW ROUTINE LOAD statement return. [#57123](https://github.com/StarRocks/starrocks/pull/57123) * Disallowed unauthorized behaviors from StarRocks to LDAP. [#57131](https://github.com/StarRocks/starrocks/pull/57131) * Supports returning an error when the schema of an AVRO file does not match the schema of the Hive table. [#57296](https://github.com/StarRocks/starrocks/pull/57296) * Materialized views support the `excluded_refresh_tables` property. [#56428](https://github.com/StarRocks/starrocks/pull/56428) ##### Bug Fixes[​](#bug-fixes-9 "Direct link to Bug Fixes") Fixed the following issues: * Flat JSON does not support the `get_json_bool` function. [#58077](https://github.com/StarRocks/starrocks/pull/58077) * SHOW AUTHENTICATION statement returns the password. [#58072](https://github.com/StarRocks/starrocks/pull/58072) * The `percentile_count` function returns incorrect values. [#58038](https://github.com/StarRocks/starrocks/pull/58038) * Issues caused by spilling strategies. [#58022](https://github.com/StarRocks/starrocks/pull/58022) * After a BE is blacklisted, Stream Load still dispatches tasks to the BE, causing task failures. [#57919](https://github.com/StarRocks/starrocks/pull/57919) * Issues when using the `cast` function with semi-structured data types. [#57804](https://github.com/StarRocks/starrocks/pull/57804) * The `array_map` function returns incorrect values. [#57756](https://github.com/StarRocks/starrocks/pull/57756) * In the scenario of a single tablet, using multiple `distinct` functions on the same column with a single-column GROUP BY clause leads to incorrect query results. [#57690](https://github.com/StarRocks/starrocks/pull/57690) * MIN/MAX values in the profiles of big queries are inaccurate. [#57655](https://github.com/StarRocks/starrocks/pull/57655) * Non-partitioned materialized views based on Delta Lake data cannot rewrite queries. [#57686](https://github.com/StarRocks/starrocks/pull/57686) * A Routine Load deadlock issue. [#57430](https://github.com/StarRocks/starrocks/pull/57430) * Predicate pushdown issues with DATE/DATETIME columns. [#57576](https://github.com/StarRocks/starrocks/pull/57576) * An issue when the `percentile_disc` function has an empty input. [#57572](https://github.com/StarRocks/starrocks/pull/57572) * When modifying the bucket distribution of a table with the statement `ALTER TABLE {table} PARTITIONS (p1, p1) DISTRIBUTED BY ...`, specifying duplicate partition names could result in failure to delete internally generated temporary partitions. [#57005](https://github.com/StarRocks/starrocks/pull/57005) * ALTER TABLE MODIFY COLUMN fails with expression partitioned tables based on `str2date` function. [#57487](https://github.com/StarRocks/starrocks/pull/57487) * CACHE SELECT issue with semi-structured columns. [#57448](https://github.com/StarRocks/starrocks/pull/57448) * Upgrade compatibility issue caused by `hadoop-lib`. [#57436](https://github.com/StarRocks/starrocks/pull/57436) * Case sensitivity error issues when creating partitions. [#54867](https://github.com/StarRocks/starrocks/pull/54867) * Some columns generate incorrect sort keys during updates. [#57375](https://github.com/StarRocks/starrocks/pull/57375) * Unknown issues caused by nested window functions . [#57216](https://github.com/StarRocks/starrocks/pull/57216) #### 3.3.12[​](#3312 "Direct link to 3.3.12") Release date: April 3, 2025 ##### New Features[​](#new-features "Direct link to New Features") * Supports the `percentile_approx_weighted` function. [#56654](https://github.com/StarRocks/starrocks/pull/56654) * Supports modifying properties of Hive Catalog and Hudi Catalog. [#56212](https://github.com/StarRocks/starrocks/pull/56212) * Paimon Catalog supports manifest cache. [#55788](https://github.com/StarRocks/starrocks/pull/55788) * Supports `SHOW PARTITIONS` for tables in Paimon Catalog. [#55785](https://github.com/StarRocks/starrocks/pull/55785) * Supports statistics collection for Paimon Catalog. [#55757](https://github.com/StarRocks/starrocks/pull/55757) ##### Improvements[​](#improvements-3 "Direct link to Improvements") * Various improvements and bug fixes related to statistics. [#57147](https://github.com/StarRocks/starrocks/pull/57147) [#57238](https://github.com/StarRocks/starrocks/pull/57238) [#57170](https://github.com/StarRocks/starrocks/pull/57170) [#57154](https://github.com/StarRocks/starrocks/pull/57154) [#57124](https://github.com/StarRocks/starrocks/pull/57124) [#57047](https://github.com/StarRocks/starrocks/pull/57047) [#56956](https://github.com/StarRocks/starrocks/pull/56956) [#57031](https://github.com/StarRocks/starrocks/pull/57031) [#56904](https://github.com/StarRocks/starrocks/pull/56904) [#56950](https://github.com/StarRocks/starrocks/pull/56950) [#56671](https://github.com/StarRocks/starrocks/pull/56671) [#55922](https://github.com/StarRocks/starrocks/pull/55922) * Optimized error messages when table creation fails. [#57055](https://github.com/StarRocks/starrocks/pull/57055) * Enhanced retry mechanism for Broker Load. [#56987](https://github.com/StarRocks/starrocks/pull/56987) * Improved performance of `array_generate`. [#57252](https://github.com/StarRocks/starrocks/pull/57252) * Aborted ongoing Compaction tasks for deleted partitions. [#56943](https://github.com/StarRocks/starrocks/pull/56943) * Optimized error messages when `ALTER TABLE` fails. [#57054](https://github.com/StarRocks/starrocks/pull/57054) * Removed unnecessary reverse step from `array_agg()` to improve performance. [#56958](https://github.com/StarRocks/starrocks/pull/56958) * Added checksum verification for replicas in Primary Key tables. [#56519](https://github.com/StarRocks/starrocks/pull/56519) * Masked sensitive information in the `FILES` function output. [#56684](https://github.com/StarRocks/starrocks/pull/56684) * Reduced noisy logs related to materialized views. [#56672](https://github.com/StarRocks/starrocks/pull/56672) * Upgraded Iceberg version to 1.7.1. [#55271](https://github.com/StarRocks/starrocks/pull/55271) ##### Bug Fixes[​](#bug-fixes-10 "Direct link to Bug Fixes") * `INSERT INTO FILES` did not support CSV delimiter conversion. [#57126](https://github.com/StarRocks/starrocks/pull/57126) * Issues with Iceberg REST Catalog. [#55416](https://github.com/StarRocks/starrocks/pull/55416) * Predicate was lost during rewrite for view-based materialized views. [#57153](https://github.com/StarRocks/starrocks/pull/57153) * Paimon Catalog failed to read tables with schema changes. [#56796](https://github.com/StarRocks/starrocks/pull/56796) * Timezone conversion issue in Paimon Catalog. [#56879](https://github.com/StarRocks/starrocks/pull/56879) * `SHOW MATERIALIZED VIEWS` did not display `default_catalog` information. [#56362](https://github.com/StarRocks/starrocks/pull/56362) * In Trino dialect mode, time strings containing 'T' were not accepted. (Solution: replaced `parse_datetime` with `str_to_jodatime`.) [#56565](https://github.com/StarRocks/starrocks/pull/56565) * Incorrect result of `first_value` function. [#56467](https://github.com/StarRocks/starrocks/pull/56467) * Incorrect result of `concat_ws` function. [#56384](https://github.com/StarRocks/starrocks/pull/56384) ##### Behavior Changes[​](#behavior-changes-1 "Direct link to Behavior Changes") * Added authentication to the FE Profile interface. [#56914](https://github.com/StarRocks/starrocks/pull/56914) * Changed default value of session variable `big_query_profile_threshold` from `0` to `30`. [#56520](https://github.com/StarRocks/starrocks/pull/56520) #### 3.3.11[​](#3311 "Direct link to 3.3.11") Release date: March 7, 2025 ##### New Features[​](#new-features-1 "Direct link to New Features") * Window functions support `max_by` and `min_by`. [#54961](https://github.com/StarRocks/starrocks/pull/54961) ##### Improvements[​](#improvements-4 "Direct link to Improvements") * `Files` supports exporting JSON type data into Parquet files. [#56406](https://github.com/StarRocks/starrocks/pull/56406) * Optimized Data Cache WarmUp performance for cloud-native tables in shared-data clusters. [#56190](https://github.com/StarRocks/starrocks/pull/56190) * Supports parsing `AT TIME ZONE` expressions and the `from_iso8601_timestamp` function in Trino. [#56311](https://github.com/StarRocks/starrocks/pull/56311) [#55573](https://github.com/StarRocks/starrocks/pull/55573) * Partial Updates for Primary Key tables within shared-data clusters supports Condition Updates. [#56132](https://github.com/StarRocks/starrocks/pull/56132) * Extended support for statistics collection across all types of SQL statements. [#56257](https://github.com/StarRocks/starrocks/pull/56257) * Supports configuring the maximum number of returned rows for `SHOW PROC '/transaction'`. [#55933](https://github.com/StarRocks/starrocks/pull/55933) * Supports creating asynchronous materialized views on Oracle-type JDBC Catalog tables. [#55372](https://github.com/StarRocks/starrocks/pull/55372) * MemTracker on BE WebUI supports pagination with 25 rows per page. [#56206](https://github.com/StarRocks/starrocks/pull/56206) * Supports pushdown for subfields of complex types in table functions. [#55425](https://github.com/StarRocks/starrocks/pull/55425) * Supports LDAP login for MariaDB clients. [#55720](https://github.com/StarRocks/starrocks/pull/55720) * Upgraded Paimon version to 1.0.1. [#54796](https://github.com/StarRocks/starrocks/pull/54796) [#55760](https://github.com/StarRocks/starrocks/pull/55760) * Eliminates unnecessary `unnest` computations during query execution to reduce overhead. [#55431](https://github.com/StarRocks/starrocks/pull/55431) * Supports enabling Compaction for source clusters that are in the shared-data mode during cross-cluster synchronization. [#54787](https://github.com/StarRocks/starrocks/pull/54787) * Brings high-cost operations like DECIMAL division forward in topN computations to reduce overhead. [#55417](https://github.com/StarRocks/starrocks/pull/55417) * Optimized performance under ARM architecture. [#55072](https://github.com/StarRocks/starrocks/pull/55072) [#55510](https://github.com/StarRocks/starrocks/pull/55510) * For Hive table-based materialized views, StarRocks will perform checks and refreshes on the updated partitions only instead of full table refreshes if the base table was dropped and recreated. [#45118](https://github.com/StarRocks/starrocks/pull/45118) * DELETE operations support partition pruning. [#55400](https://github.com/StarRocks/starrocks/pull/55400) * Optimized priority strategy for collecting internal table statistics to improve efficiency when there are excessive tables. [#55446](https://github.com/StarRocks/starrocks/pull/55446) * When data loading involves multiple partitions, StarRocks merges transaction logs to improve loading performance. [#55143](https://github.com/StarRocks/starrocks/pull/55143) * Optimized error messages for SQL Translation. [#55327](https://github.com/StarRocks/starrocks/pull/55327) * Added a session variable `parallel_merge_late_materialization_mode` to control parallel merge behavior. [#55082](https://github.com/StarRocks/starrocks/pull/55082) * Optimized error messages for generated columns. [#54949](https://github.com/StarRocks/starrocks/pull/54949) * Optimized performance of `SHOW MATERIALIZED VIEWS`. [#54374](https://github.com/StarRocks/starrocks/pull/54374) ##### Bug Fixes[​](#bug-fixes-11 "Direct link to Bug Fixes") Fixed the following issues: * FE does not support casting constant TIME data types into DATETIME. [#55804](https://github.com/StarRocks/starrocks/pull/55804) * Stream Load transaction interface does not support the `starrocks_fe_table_load_rows` and `starrocks_fe_table_load_bytes` metrics. [#44991](https://github.com/StarRocks/starrocks/pull/44991) * Changes to automatic statistics collection do not take effect. [#56173](https://github.com/StarRocks/starrocks/pull/56173) * Materialized views in abnormal states caused issues with `SHOW MATERIALIZED VIEWS`. [#55995](https://github.com/StarRocks/starrocks/pull/55995) * Text-based materialized view rewrite does not work across different databases. [#56001](https://github.com/StarRocks/starrocks/pull/56001) * Metadata compatibility issues in JDBC Catalogs. [#55993](https://github.com/StarRocks/starrocks/pull/55993) * Issues of handling the JSON data type in JDBC Catalogs. [#56008](https://github.com/StarRocks/starrocks/pull/56008) * Incorrect Sort Key settings during Schema Change. [#55902](https://github.com/StarRocks/starrocks/pull/55902) * Credential information leak issue in Broker Load. [#55358](https://github.com/StarRocks/starrocks/pull/55358) * An error caused by pushing down LIMIT before predicates in CTE. [#55768](https://github.com/StarRocks/starrocks/pull/55768) * An error caused by table schema changes in Stream Load. [#55773](https://github.com/StarRocks/starrocks/pull/55773) * A privilege issue due to the execution plan of DELETE statements containing SELECT. [#55695](https://github.com/StarRocks/starrocks/pull/55695) * An issue caused by not aborting compaction tasks when shutting down CN. [#55503](https://github.com/StarRocks/starrocks/pull/55503) * Follower FE nodes unable to fetch updated loading statistics. [#55758](https://github.com/StarRocks/starrocks/pull/55758) * Incorrect capacity statistics for the spill directory. [#55703](https://github.com/StarRocks/starrocks/pull/55703) * Failed to create materialized views due to lack of sufficient partition checks for base tables with list partitions. [#55673](https://github.com/StarRocks/starrocks/pull/55673) * An issue caused by missing metadata locks in ALTER TABLE. [#55605](https://github.com/StarRocks/starrocks/pull/55605) * An error in `SHOW CREATE TABLE` caused by constraints. [#55592](https://github.com/StarRocks/starrocks/pull/55592) * OOM due to large ARRAY in Nestloop Join. [#55603](https://github.com/StarRocks/starrocks/pull/55603) * Lock issue with DROP PARTITION. [#55549](https://github.com/StarRocks/starrocks/pull/55549) * An issue with min/max window functions due to not supporting string types. [#55537](https://github.com/StarRocks/starrocks/pull/55537) * Parser performance degraded. [#54830](https://github.com/StarRocks/starrocks/pull/54830) * Column name case sensitivity issue during partial updates. [#55442](https://github.com/StarRocks/starrocks/pull/55442) * Import failures when Stream Load is scheduled on nodes with an "Alive" state of false. [#55371](https://github.com/StarRocks/starrocks/pull/55371) * Incorrect output column order in materialized views containing ORDER BY. [#55355](https://github.com/StarRocks/starrocks/pull/55355) * BE crashes due to disk failure. [#55042](https://github.com/StarRocks/starrocks/pull/55042) * Incorrect query results caused by Query Cache. [#55287](https://github.com/StarRocks/starrocks/pull/55287) * Parquet Writer fails to convert time zone when writing TIMESTAMP type with time zones. [#55194](https://github.com/StarRocks/starrocks/pull/55194) * Loading tasks hang due to ALTER job timeout. [#55207](https://github.com/StarRocks/starrocks/pull/55207) * An error caused by `date_format` function when input is in milliseconds. [#54854](https://github.com/StarRocks/starrocks/pull/54854) * Materialized view rewrite failure caused by Partition Key being of DATE type. [#54804](https://github.com/StarRocks/starrocks/pull/54804) ##### Behavior Changes[​](#behavior-changes-2 "Direct link to Behavior Changes") * Added authentication to the `query_detail` interface in FE. [#55919](https://github.com/StarRocks/starrocks/pull/55919) * The UUID type in Iceberg now maps to BINARY. [#54978](https://github.com/StarRocks/starrocks/pull/54978) * Uses changed row count instead of the visible time of partitions to determine if statistics need to be recollected. [#55373](https://github.com/StarRocks/starrocks/pull/55373) #### 3.3.10 (Yanked)[​](#3310-yanked "Direct link to 3.3.10 (Yanked)") Release date: February 21, 2025 tip This version has been taken offline due to metadata loss issues in **shared-data clusters**. * **Problem**: When there are committed compaction transactions that are not yet been published during a shift of Leader FE node in a shared-data cluster, metadata loss may occur after the shift. * **Impact scope**: This problem only affects shared-data clusters. Shared-nothing clusters are unaffected. * **Temporary workaround**: When the Publish task is returned with an error, you can execute `SHOW PROC 'compactions'` to check if there are any partitions that have two compaction transactions with empty `FinishTime`. You can execute `ALTER TABLE DROP PARTITION FORCE` to drop the partitions to avoid Publish tasks getting hang. #### 3.3.9[​](#339 "Direct link to 3.3.9") Release date: January 12, 2025 ##### New Features[​](#new-features-2 "Direct link to New Features") * Supports the translation of Trino SQL into StarRocks SQL. [#54185](https://github.com/StarRocks/starrocks/pull/54185) ##### Improvements[​](#improvements-5 "Direct link to Improvements") * Corrected FE node names starting with `bdbje_reset_election_group` to enhance clarity. [#54399](https://github.com/StarRocks/starrocks/pull/54399) * Implemented vectorization for the `IF` function on ARM architectures. [#53093](https://github.com/StarRocks/starrocks/pull/53093) * `ALTER SYSTEM CREATE IMAGE` supports creating an image for StarManager. [#54370](https://github.com/StarRocks/starrocks/pull/54370) * Supports deleting cloud-native indexes of Primary Key tables in shared-data clusters. [#53971](https://github.com/StarRocks/starrocks/pull/53971) * Enforced the refresh of materialized views when the `FORCE` keyword is specified. [#52081](https://github.com/StarRocks/starrocks/pull/52081) * Supports specifying hints in `CACHE SELECT`. [#54697](https://github.com/StarRocks/starrocks/pull/54697) * Supports loading compressed CSV files using the `FILES()` function. Supported compression formats include gzip, bz2, lz4, deflate, and zstd. [#54626](https://github.com/StarRocks/starrocks/pull/54626) * Supports assigning multiple values to the same column in an `UPDATE` statement. [#54534](https://github.com/StarRocks/starrocks/pull/54534) ##### Bug Fixes[​](#bug-fixes-12 "Direct link to Bug Fixes") Fixed the following issues: * Unexpected errors when refreshing materialized views built on JDBC catalogs. [#54487](https://github.com/StarRocks/starrocks/pull/54487) * Instability in results when a Delta Lake table joins itself. [#54473](https://github.com/StarRocks/starrocks/pull/54473) * Upload retries fail when backing up data to HDFS. [#53679](https://github.com/StarRocks/starrocks/pull/53679) * BFD initialization errors on the aarch64 architecture. [#54372](https://github.com/StarRocks/starrocks/pull/54372) * Sensitive information recorded in BE logs. [#54677](https://github.com/StarRocks/starrocks/pull/54677) * Errors in Compaction-related metrics in profiles. [#54678](https://github.com/StarRocks/starrocks/pull/54678) * BE crashes caused by creating tables with nested `TIME` types. [#54601](https://github.com/StarRocks/starrocks/pull/54601) * Query plan errors for `LIMIT` queries with subquery TOP-N. [#54507](https://github.com/StarRocks/starrocks/pull/54507) ##### Downgrade notes[​](#downgrade-notes "Direct link to Downgrade notes") * Clusters can be downgraded from v3.3.9 only to v3.2.11 and later. #### 3.3.8[​](#338 "Direct link to 3.3.8") Release date: January 3, 2025 ##### Improvements[​](#improvements-6 "Direct link to Improvements") * Added a cluster idle API to assist in determining cluster status. [#53850](https://github.com/StarRocks/starrocks/pull/53850) * Included node information and histogram metrics in JSON metrics. [#53735](https://github.com/StarRocks/starrocks/pull/53735) * Optimized the MemTable for Primary Key tables in shared-data clusters. [#54178](https://github.com/StarRocks/starrocks/pull/54178) * Optimized memory usage and statistics for Primary Key tables in shared-data clusters. [#54358](https://github.com/StarRocks/starrocks/pull/54358) * Introduced a limit on the number of partitions scanned per node for queries requiring full-table or large-scale partition scans, enhancing system stability by reducing scanning pressure on individual BE or CN nodes. [#53747](https://github.com/StarRocks/starrocks/pull/53747) * Supports collecting statistics of Paimon tables. [#52858](https://github.com/StarRocks/starrocks/pull/52858) * Supports configuration of S3 client request timeout for shared-data clusters. [#54211](https://github.com/StarRocks/starrocks/pull/54211) ##### Bug Fixes[​](#bug-fixes-13 "Direct link to Bug Fixes") Fixed the following issues: * BE crashes caused by inconsistencies in the DelVec of Primary Key tables. [#53460](https://github.com/StarRocks/starrocks/pull/53460) * Issues with lock release of Primary Key tables in shared-data clusters. [#53878](https://github.com/StarRocks/starrocks/pull/53878) * Errors of UDFs nested in functions are not returned in query failures. [#44297](https://github.com/StarRocks/starrocks/pull/44297) * Transactions are blocked at the Decommission phase because they depend on the original replicas. [#49349](https://github.com/StarRocks/starrocks/pull/49349) * Queries against Delta Lake tables use relative paths instead of filenames for file retrieval. [#53949](https://github.com/StarRocks/starrocks/pull/53949) * An error is returned when querying Delta Lake Shallow Clone tables. [#54044](https://github.com/StarRocks/starrocks/pull/54044) * Case sensitivity issues when reading Paimon using JNI. [#54041](https://github.com/StarRocks/starrocks/pull/54041) * An error is returned during `INSERT OVERWRITE` operations on Hive tables created in Hive. [#53792](https://github.com/StarRocks/starrocks/pull/53792) * `SHOW TABLE STATUS` command does not validate view privileges. [#53811](https://github.com/StarRocks/starrocks/pull/53811) * Missing FE metrics. [#53058](https://github.com/StarRocks/starrocks/pull/53058) * Memory leaks in `INSERT` tasks. [#53809](https://github.com/StarRocks/starrocks/pull/53809) * Concurrency issues caused by missing write locks in replication tasks. [#54061](https://github.com/StarRocks/starrocks/pull/54061) * `partition_ttl` of tables in the `statistics` database does not take effect. [#54398](https://github.com/StarRocks/starrocks/pull/54398) * Query Cache-related issues: * Crashes when Query Cache is enabled with Group Execution. [#54363](https://github.com/StarRocks/starrocks/pull/54363) * Runtime Filter crashes. [#54305](https://github.com/StarRocks/starrocks/pull/54305) * Issues with materialized view Union Rewrite. [#54293](https://github.com/StarRocks/starrocks/pull/54293) * Missing padding in string updates for partial updates in Primary Key tables. [#54182](https://github.com/StarRocks/starrocks/pull/54182) * Incorrect execution plans for `max(count(distinct))` when low-cardinality optimization is enabled. [#53403](https://github.com/StarRocks/starrocks/pull/53403) * Issues with changing the `excluded_refresh_tables` parameter of materialized views. [#53394](https://github.com/StarRocks/starrocks/pull/53394) ##### Behavior Changes[​](#behavior-changes-3 "Direct link to Behavior Changes") * Changed the default value of `persistent_index_type` for Primary Key tables in shared-data clusters to `CLOUD_NATIVE`, that is, enabled Persistent Index by default. [#52209](https://github.com/StarRocks/starrocks/pull/52209) #### 3.3.7[​](#337 "Direct link to 3.3.7") Release date: November 29, 2024 ##### New Features[​](#new-features-3 "Direct link to New Features") * Added a new Materialized View parameter, `excluded_refresh_tables`, exclude tables that need to be refreshed. [#50926](https://github.com/StarRocks/starrocks/pull/50926) ##### Improvements[​](#improvements-7 "Direct link to Improvements") * Rewrote `unnest(bitmap_to_array)` as `unnest_bitmap` to improve performance. [#52870](https://github.com/StarRocks/starrocks/pull/52870) * Reduced the write and delete operations of Txn logs. [#42542](https://github.com/StarRocks/starrocks/pull/42542) ##### Bug Fixes[​](#bug-fixes-14 "Direct link to Bug Fixes") Fixed the following issues: * Failure to connect Power BI to external tables. [#52977](https://github.com/StarRocks/starrocks/pull/52977) * Misleading FE Thrift RPC failure messages in logs. [#52706](https://github.com/StarRocks/starrocks/pull/52706) * Routine Load tasks were canceled due to expired transactions (now tasks are canceled only if the database or table no longer exists). [#50334](https://github.com/StarRocks/starrocks/pull/50334) * Stream Load failures when submitted using HTTP 1.0. [#53010](https://github.com/StarRocks/starrocks/pull/53010) [#53008](https://github.com/StarRocks/starrocks/pull/53008) * Integer overflow of partition IDs. [#52965](https://github.com/StarRocks/starrocks/pull/52965) * Hive Text Reader failed to recognize the last empty element. [#52990](https://github.com/StarRocks/starrocks/pull/52990) * Issues caused by `array_map` in Join conditions. [#52911](https://github.com/StarRocks/starrocks/pull/52911) * Metadata cache issues under high concurrency scenarios. [#52968](https://github.com/StarRocks/starrocks/pull/52968) * The whole materialized view was refreshed when a partition was dropped from the base table. [#52740](https://github.com/StarRocks/starrocks/pull/52740) #### 3.3.6[​](#336 "Direct link to 3.3.6") Release date: November 18, 2024 ##### Improvements[​](#improvements-8 "Direct link to Improvements") * Optimized internal repair logic for Primary Key tables. [#52707](https://github.com/StarRocks/starrocks/pull/52707) * Optimized the internal implementation of histograms of statistics. [#52400](https://github.com/StarRocks/starrocks/pull/52400) * Supports adjusting log level via the FE configuration item `sys_log_warn_modules` to reduce Hudi Catalog logging. [#52709](https://github.com/StarRocks/starrocks/pull/52709) * Supports constant folding in the `yearweek` function. [#52714](https://github.com/StarRocks/starrocks/pull/52714) * Avoided push-down for Lambda functions. [#52655](https://github.com/StarRocks/starrocks/pull/52655) * Divided the Query Error metric into three: Internal Error Rate, Analysis Error Rate, and Timeout Rate. [#52646](https://github.com/StarRocks/starrocks/pull/52646) * Avoided constant expressions being extracted as common expressions within `array_map`. [#52541](https://github.com/StarRocks/starrocks/pull/52541) * Optimized the Text-based Rewrite of materialized views. [#52498](https://github.com/StarRocks/starrocks/pull/52498) ##### Bug Fixes[​](#bug-fixes-15 "Direct link to Bug Fixes") Fixed the following issues: * The `unique_constraints` and `foreign_constraints` parameters were incomplete in SHOW CREATE TABLE for cloud-native tables in shared-data clusters. [#52804](https://github.com/StarRocks/starrocks/pull/52804) * Some materialized views were activated even when `enable_mv_automatic_active_check` was set to `false`. [#52799](https://github.com/StarRocks/starrocks/pull/52799) * Memory usage is not reducing after stale memory flush. [#52613](https://github.com/StarRocks/starrocks/pull/52613) * Resource leak caused by Hudi file-system views. [#52738](https://github.com/StarRocks/starrocks/pull/52738) * Concurrent Publish and Update operations on Primary Key tables may cause issues. [#52687](https://github.com/StarRocks/starrocks/pull/52687) * Failures to terminate queries on clients. [#52185](https://github.com/StarRocks/starrocks/pull/52185) * Multi-column List partitions cannot be pushed down. [#51036](https://github.com/StarRocks/starrocks/pull/51036) * Incorrect result due to the lack of `hasnull` property in ORC files. [#52555](https://github.com/StarRocks/starrocks/pull/52555) * An issue caused by using uppercase column names in ORDER BY during table creation. [#52513](https://github.com/StarRocks/starrocks/pull/52513) * An error was returned after running `ALTER TABLE PARTITION (*) SET ("storage_cooldown_ttl" = "xxx")`. [#52482](https://github.com/StarRocks/starrocks/pull/52482) ##### Behavior Changes[​](#behavior-changes-4 "Direct link to Behavior Changes") * In earlier versions, scale-in operations would fail if there were insufficient replicas for views in the `_statistics_` database. Starting from v3.3.6, if nodes are scaled in to 3 or more, view replicas are set to 3; if there is only 1 node after the scale-in, view replicas are set to 1, allowing for successful scale-in. [#51799](https://github.com/StarRocks/starrocks/pull/51799) Affected views include: * `column_statistics` * `histogram_statistics` * `table_statistic_v1` * `external_column_statistics` * `external_histogram_statistics` * `pipe_file_list` * `loads_history` * `task_run_history` * New Primary Key tables no longer allow `__op` as a column name, even if `allow_system_reserved_names` is set to `true`. Existing tables are unaffected. [#52621](https://github.com/StarRocks/starrocks/pull/52621) * Expression-partitioned tables cannot have partition names modified. [#52557](https://github.com/StarRocks/starrocks/pull/52557) * Deprecated FE parameters `heartbeat_mgr_blocking_queue_size` and `profile_process_threads_num`. [#52236](https://github.com/StarRocks/starrocks/pull/52236) * Enabled persistent index on object storage by default for Primary Key tables in shared-data clusters. [#52209](https://github.com/StarRocks/starrocks/pull/52209) * Disallowed manual changes to bucketing methods for tables with the random bucketing method. [#52120](https://github.com/StarRocks/starrocks/pull/52120) * Backup and Restore-related parameter changes: [#52111](https://github.com/StarRocks/starrocks/pull/52111) * `make_snapshot_worker_count` supports dynamic configuration. * `release_snapshot_worker_count` supports dynamic configuration. * `upload_worker_count` supports dynamic configuration. Its default value is changed from `1` to the number of CPU cores on the machine where the BE resides. * `download_worker_count` supports dynamic configuration. Its default value is changed from `1` to the number of CPU cores on the machine where the BE resides. * The return type of `SELECT @@autocommit` has changed from BOOLEAN to BIGINT. [#51946](https://github.com/StarRocks/starrocks/pull/51946) * Added a new FE configuration item, `max_bucket_number_per_partition`, to control the maximum number of buckets per partition. [#47852](https://github.com/StarRocks/starrocks/pull/47852) * Enabled memory usage checks by default for Primary Key tables. [#52393](https://github.com/StarRocks/starrocks/pull/52393) * Optimized loading strategy to reduce loading speed when Compaction tasks cannot be completed on time. [#52269](https://github.com/StarRocks/starrocks/pull/52269) #### 3.3.5[​](#335 "Direct link to 3.3.5") Release date: October 23, 2024 ##### New Features[​](#new-features-4 "Direct link to New Features") * Supports millisecond and microsecond precision in the DATETIME type. * Resource groups support CPU hard isolation. ##### Improvements[​](#improvements-9 "Direct link to Improvements") * Optimized performance and extraction strategy for Flat JSON. [#50696](https://github.com/StarRocks/starrocks/pull/50696) * Reduced memory usage for the following ARRAY functions: * array\_contains/array\_position [#50912](https://github.com/StarRocks/starrocks/pull/50912) * array\_filter [#51363](https://github.com/StarRocks/starrocks/pull/51363) * array\_match [#51377](https://github.com/StarRocks/starrocks/pull/51377) * array\_map [#51244](https://github.com/StarRocks/starrocks/pull/51244) * Optimized error messages when loading `Null` values into List partition keys with the `Not Null` attribute. [#51086](https://github.com/StarRocks/starrocks/pull/51086) * Optimized error messages for Files() when authentication fails in the Files function. [#51697](https://github.com/StarRocks/starrocks/pull/51697) * Optimized internal statistics for `INSERT OVERWRITE`. [#50417](https://github.com/StarRocks/starrocks/pull/50417) * Shared-data clusters support garbage collection (GC) for persistent index files. [#51684](https://github.com/StarRocks/starrocks/pull/51684) * Added FE logs to help diagnose FE out-of-memory (OOM) issues. [#51528](https://github.com/StarRocks/starrocks/pull/51528) * Supports recovering metadata from the metadata directory of FE. [#51040](https://github.com/StarRocks/starrocks/pull/51040) ##### Bug Fixes[​](#bug-fixes-16 "Direct link to Bug Fixes") Fixed the following issues: * A deadlock issue caused by PIPE exceptions. [#50841](https://github.com/StarRocks/starrocks/pull/50841) * Dynamic partition creation failures block subsequent partition creation. [#51440](https://github.com/StarRocks/starrocks/pull/51440) * An error is returned for `UNION ALL` queries with `ORDER BY`. [#51647](https://github.com/StarRocks/starrocks/pull/51647) * CTE in UPDATE statements causes hints to be ignored. [#51458](https://github.com/StarRocks/starrocks/pull/51458) * The `load_finish_time` field in the system-defined view `statistics.loads_history` does not update as expected after a loading task is completed. [#51174](https://github.com/StarRocks/starrocks/pull/51174) * UDTF mishandles multibyte UTF-8 characters. [#51232](https://github.com/StarRocks/starrocks/pull/51232) ##### Behavior Changes[​](#behavior-changes-5 "Direct link to Behavior Changes") * Modified the return content of the `EXPLAIN` statement. After the change, the return content is equivalent to `EXPLAIN COST`. You can configure the level of details returned by `EXPLAIN` using the dynamic FE parameter `query_detail_explain_level`. The default value is `COSTS`, with other valid values being `NORMAL` and `VERBOSE`. [#51439](https://github.com/StarRocks/starrocks/pull/51439) #### 3.3.4[​](#334 "Direct link to 3.3.4") Release date: September 30, 2024 ##### New Features[​](#new-features-5 "Direct link to New Features") * Supports creating asynchronous materialized views on List Partition tables. [#46680](https://github.com/StarRocks/starrocks/pull/46680) [#46808](https://github.com/StarRocks/starrocks/pull/46808/files) * List Partition tables now support Nullable partition columns. [#47797](https://github.com/StarRocks/starrocks/pull/47797) * Supports viewing external file schema information using `DESC FILES()`. [#50527](https://github.com/StarRocks/starrocks/pull/50527) * Supports viewing replication task metrics via `SHOW PROC '/replications'`. [#50483](https://github.com/StarRocks/starrocks/pull/50483) ##### Improvements[​](#improvements-10 "Direct link to Improvements") * Optimized data recycling performance for `TRUNCATE TABLE` in shared-data clusters. [#49975](https://github.com/StarRocks/starrocks/pull/49975) * Supports intermediate result spilling for CTE operators. [#47982](https://github.com/StarRocks/starrocks/pull/47982) * Supports adaptive phased scheduling to alleviate OOM issues caused by complex queries. [#47868](https://github.com/StarRocks/starrocks/pull/47868) * Supports predicate pushdown for STRING-type date or datatime columns in specific scenarios. [#50643](https://github.com/StarRocks/starrocks/pull/50643) * Supports COUNT DISTINCT computation on constant semi-structured data. [#48273](https://github.com/StarRocks/starrocks/pull/48273) * Added a new FE parameter `lake_enable_balance_tablets_between_workers` to enable tablet balancing for tables in shared-date clusters. [#50843](https://github.com/StarRocks/starrocks/pull/50843) * Enhanced query rewrite capabilities for generated columns. [#50398](https://github.com/StarRocks/starrocks/pull/50398) * Partial Update now supports automatically populating columns with default values of `CURRENT_TIMESTAMP`. [#50287](https://github.com/StarRocks/starrocks/pull/50287) ##### Bug Fixes[​](#bug-fixes-17 "Direct link to Bug Fixes") Fixed the following issues: * The error "version has been compacted" caused by an infinite loop on the FE side during Tablet Clone. [#50561](https://github.com/StarRocks/starrocks/pull/50561) * ISO- formatted DATETIME types cannot be pushed down. [#49358](https://github.com/StarRocks/starrocks/pull/49358) * In concurrent scenarios, data still existed after the tablet was deleted. [#50382](https://github.com/StarRocks/starrocks/pull/50382) * Incorrect results returned by the `yearweek` function. [#51065](https://github.com/StarRocks/starrocks/pull/51065) * An issue with low cardinality dictionaries in ARRAY during CTE queries. [#51148](https://github.com/StarRocks/starrocks/pull/51148) * After FE restarts, partition TTL-related parameters were lost for materialized views. [#51028](https://github.com/StarRocks/starrocks/pull/51028) * Data loss in columns defined with `CURRENT_TIMESTAMP` after upgrading. [#50911](https://github.com/StarRocks/starrocks/pull/50911) * A stack overflow caused by the `array_distinct` function. [#51017](https://github.com/StarRocks/starrocks/pull/51017) * Activation failures for materialized views after upgrading due to changes in default field lengths. You can avoid such issues by setting `enable_active_materialized_view_schema_strict_check` to `false`. [#50869](https://github.com/StarRocks/starrocks/pull/50869) * Resource group property `cpu_weight` can be set to a negative value. [#51005](https://github.com/StarRocks/starrocks/pull/51005) * Incorrect statistics for disk capacity information. [#50669](https://github.com/StarRocks/starrocks/pull/50669) * Constant fold in the `replace` function. [#50828](https://github.com/StarRocks/starrocks/pull/50828) ##### Behavior Changes[​](#behavior-changes-6 "Direct link to Behavior Changes") * Changed the default replica number for external catalog-based materialized views from `1` to the value of the FE parameter `default_replication_num` (Default value: `3`). [#50931](https://github.com/StarRocks/starrocks/pull/50931) #### 3.3.3[​](#333 "Direct link to 3.3.3") Release date: September 5, 2024 ##### New Features[​](#new-features-6 "Direct link to New Features") * Supports user-level variables. [#48477](https://github.com/StarRocks/starrocks/pull/48477) * Supports Delta Lake Catalog metadata cache with manual and periodic refresh strategies. [#46526](https://github.com/StarRocks/starrocks/pull/46526) [#49069](https://github.com/StarRocks/starrocks/pull/49069) * Supports loading JSON types from Parquet files. [#49385](https://github.com/StarRocks/starrocks/pull/49385) * JDBC SQL Server Catalog supports queries with LIMIT. [#48248](https://github.com/StarRocks/starrocks/pull/48248) * Shared-data clusters support Partial Updates with INSERT INTO. [#49336](https://github.com/StarRocks/starrocks/pull/49336) ##### Improvements[​](#improvements-11 "Direct link to Improvements") * Optimized error messages for loading: * When memory limits are reached during loading, the IP of the corresponding BE node is returned for easier troubleshooting. [#49335](https://github.com/StarRocks/starrocks/pull/49335) * Detailed messages are provided when CSV data is loaded to target table columns that are not long enough. [#49713](https://github.com/StarRocks/starrocks/pull/49713) * Specific node information is provided when Kerberos authentication fails in Broker Load. [#46085](https://github.com/StarRocks/starrocks/pull/46085) * Optimized the partitioning mechanism during data loading to reduce memory usage in the initial stage. [#47976](https://github.com/StarRocks/starrocks/pull/47976) * Optimized memory usage for shared-nothing clusters by limiting metadata memory usage to avoid issues when there are too many Tablets or Segment files. [#49170](https://github.com/StarRocks/starrocks/pull/49170) * Optimized the performance of queries using `max(partition_column)`. [#49391](https://github.com/StarRocks/starrocks/pull/49391) * Partition pruning is used to optimize query performance when the partition column is a generated column (a column that is calculated based on a native column in the table), and the query predicate filter condition includes the native column. [#48692](https://github.com/StarRocks/starrocks/pull/48692) * Supports masking authentication information for Files() and PIPE. [#47629](https://github.com/StarRocks/starrocks/pull/47629) * Introduced a new statement `show proc '/global_current_queries'` to view queries running on all FE nodes. `show proc '/current_queries'` only shows queries running on the current FE node. [#49826](https://github.com/StarRocks/starrocks/pull/49826) ##### Bug Fixes[​](#bug-fixes-18 "Direct link to Bug Fixes") Fixed the following issues: * The source cluster's BE nodes were mistakenly added to the current cluster when exporting data to the destination cluster via StarRocks external tables. [#49323](https://github.com/StarRocks/starrocks/pull/49323) * TINYINT data type returned NULL when StarRocks reads ORC files using `select * from files` from clusters deployed on aarch64 machines. [#49517](https://github.com/StarRocks/starrocks/pull/49517) * Stream Load fails when loading JSON files containing large Integer types. [#49927](https://github.com/StarRocks/starrocks/pull/49927) * Incorrect schema is returned due to improper handling of invisible characters when users load CSV files with Files(). [#49718](https://github.com/StarRocks/starrocks/pull/49718) * An issue with temporary partition replacement in tables with multiple partition columns. [#49764](https://github.com/StarRocks/starrocks/pull/49764) ##### Behavior Changes[​](#behavior-changes-7 "Direct link to Behavior Changes") * Introduced a new parameter `object_storage_rename_file_request_timeout_ms` to better accommodate backup scenarios with cloud object storage. This parameter will be used as the backup timeout, with a default value of 30 seconds. [#49706](https://github.com/StarRocks/starrocks/pull/49706) * `to_json`, `CAST(AS MAP)`, and `STRUCT AS JSON` will return NULL instead of throwing an error by default when the conversion fails. You can allow errors by setting the system variable `sql_mode` to `ALLOW_THROW_EXCEPTION`. [#50157](https://github.com/StarRocks/starrocks/pull/50157) #### 3.3.2[​](#332 "Direct link to 3.3.2") Release date: August 8, 2024 ##### New Features[​](#new-features-7 "Direct link to New Features") * Supports renaming columns within StarRocks internal tables. [#47851](https://github.com/StarRocks/starrocks/pull/47851) * Supports reading Iceberg views. Currently, only Iceberg views created through StarRocks are supported. [#46273](https://github.com/StarRocks/starrocks/issues/46273) * \[Experimental] Supports adding and removing fields of STRUCT-type data. [#46452](https://github.com/StarRocks/starrocks/issues/46452) * Supports specifying the compression level for ZSTD compression format during table creation. [#46839](https://github.com/StarRocks/starrocks/issues/46839) * Added the following FE dynamic parameters to limit table boundaries. [#47896](https://github.com/StarRocks/starrocks/pull/47869) Including: * `auto_partition_max_creation_number_per_load` * `max_partition_number_per_table` * `max_bucket_number_per_partition` * `max_column_number_per_table` * Supports runtime optimization of table data distribution, ensuring optimization tasks do not conflict with DML operations on the table. [#43747](https://github.com/StarRocks/starrocks/pull/43747) * Added an observability interface for the global hit rate of Data Cache. [#48450](https://github.com/StarRocks/starrocks/pull/48450) * Added the SQL function array\_repeat. [#47862](https://github.com/StarRocks/starrocks/pull/47862) ##### Improvements[​](#improvements-12 "Direct link to Improvements") * Optimized the error messages for Routine Load failures due to Kafka authentication failures. [#46136](https://github.com/StarRocks/starrocks/pull/46136) [#47649](https://github.com/StarRocks/starrocks/pull/47649) * Stream Load supports using `\t` and `\n` as row and column delimiters. Users do not need to convert them to their hexadecimal ASCII codes. [#47302](https://github.com/StarRocks/starrocks/pull/47302) * Optimized the asynchronous statistics collection method for write operators, addressing the issue of increased latency when there are many import tasks. [#48162](https://github.com/StarRocks/starrocks/pull/48162) * Added the following BE dynamic parameters to control resource hard limits during loading, reducing the impact on BE stability when writing a large number of tablets. [#48495](https://github.com/StarRocks/starrocks/pull/48495) Including: * `load_process_max_memory_hard_limit_ratio` * `enable_new_load_on_memory_limit_exceeded` * Added consistency checks for Column IDs within the same table to prevent Compaction errors. [#48498](https://github.com/StarRocks/starrocks/pull/48628) * Supports persisting PIPE metadata to prevent metadata loss due to FE restarts. [#48852](https://github.com/StarRocks/starrocks/pull/48852) ##### Bug Fixes[​](#bug-fixes-19 "Direct link to Bug Fixes") Fixed the following issues: * The process could not end when creating a dictionary from an FE Follower. [#47802](https://github.com/StarRocks/starrocks/pull/47802) * Inconsistent information returned by the SHOW PARTITIONS command in shared-data clusters and shared-nothing clusters. [#48647](https://github.com/StarRocks/starrocks/pull/48647) * Data errors caused by incorrect type handling when loading data from JSON fields to `ARRAY` columns. [#48387](https://github.com/StarRocks/starrocks/pull/48387) * The `query_id` column in `information_schema.task_runs` cannot be queried. [#48876](https://github.com/StarRocks/starrocks/pull/48879) * During Backup, multiple requests for the same operation are submitted to different Brokers, causing request errors. [#48856](https://github.com/StarRocks/starrocks/pull/48856) * Downgrading to versions earlier than v3.1.11 or v3.2.4 causes Primary Key table index decompression failures, leading to query errors. [#48659](https://github.com/StarRocks/starrocks/pull/48659) ##### Downgrade Notes[​](#downgrade-notes-1 "Direct link to Downgrade Notes") If you have used the renaming column feature, you must rename the columns to their original names before downgrading your cluster to an earlier version. You can check the audit log of your cluster after upgrading to identify any `ALTER TABLE RENAME COLUMN` operations and the original names of the columns. #### 3.3.1 (Yanked)[​](#331-yanked "Direct link to 3.3.1 (Yanked)") Release date: July 18, 2024 tip This version has been taken offline due to compatibility issues in Primary Key tables. * **Problem**: After the cluster is upgraded from versions earlier than v3.1.11 and v3.2.4 to v3.3.1, index decompression failures will lead to failures of queries against Primary Key tables. * **Impact scope**: This problem only affects queries against Primary Key tables. * **Temporary workaround**: You can downgrade the cluster to v3.3.0 or earlier to avoid this issue. It will be fixed in v3.3.2. ##### New Features[​](#new-features-8 "Direct link to New Features") * \[Preview] Supports temporary tables. * \[Preview] JDBC Catalog supports Oracle and SQL Server. * \[Preview] Unified Catalog supports Kudu. * INSERT INTO on Primary Key tables supports Partial Updates by specifying the column list. * User-defined variables support the ARRAY type. [#42631](https://github.com/StarRocks/starrocks/pull/42613) * Stream Load supports converting JSON-type data and loading it into columns of STRUCT/MAP/ARRAY types. [#45406](https://github.com/StarRocks/starrocks/pull/45406) * Supports global dictionary cache. * Supports deleting partitions in batch. [#44744](https://github.com/StarRocks/starrocks/issues/44744) * Supports managing column-level permissions in Apache Ranger. (Column-level permissions for materialized views and views must be set under the table object.) [#47702](https://github.com/StarRocks/starrocks/pull/47702) * Supports Partial Updates in Column mode For Primary Key tables in shared-data clusters. [#46516](https://github.com/StarRocks/starrocks/issues/46516) * Stream Load supports data compression during transmission, reducing network bandwidth overhead. Users can specify different compression algorithms using parameters `compression` and `Content-Encoding`. Supported compression algorithms including GZIP, BZIP2, LZ4\_FRAME, and ZSTD. [#43732](https://github.com/StarRocks/starrocks/pull/43732) ##### Improvements[​](#improvements-13 "Direct link to Improvements") * Optimized the IdChain hashcode implementation to reduce the FE restart time. [#47599](https://github.com/StarRocks/starrocks/pull/47599) * Improved error messages for the `csv.trim_space` parameter in the FILES() function, checking for illegal characters and providing reasonable prompts. [#44740](https://github.com/StarRocks/starrocks/pull/44740) * Stream Load supports using `\t` and `\n` as row and column delimiters. Users do not need to convert them to their hexadecimal ASCII codes. [#47302](https://github.com/StarRocks/starrocks/pull/47302) ##### Bug Fixes[​](#bug-fixes-20 "Direct link to Bug Fixes") Fixed the following issues: * Schema Change failures due to file location changes caused by Tablet migration during the Schema Change process. [#45517](https://github.com/StarRocks/starrocks/pull/45517) * Cross-cluster Data Migration Tool fails to create tables in the target cluster due to control characters such as `\`, `\r` in the default values of fields. [#47861](https://github.com/StarRocks/starrocks/pull/47861) * Persistent bRPC failures after BE restarts. [#40229](https://github.com/StarRocks/starrocks/pull/40229) * The `user_admin` role can change the root password using the ALTER USER command. [#47801](https://github.com/StarRocks/starrocks/pull/47801) * Primary key index write failures cause data write errors. [#48045](https://github.com/StarRocks/starrocks/pull/48045) ##### Behavior Changes[​](#behavior-changes-8 "Direct link to Behavior Changes") * Intermediate result spilling is enabled by default when sinking data to Hive and Iceberg. [#47118](https://github.com/StarRocks/starrocks/pull/47118) * Changed the default value of the BE configuration item `max_cumulative_compaction_num_singleton_deltas` to `500`. [#47621](https://github.com/StarRocks/starrocks/pull/47621) * When users create a partitioned table without specifying the bucket number, if the number of partitions exceeds 5, the rule for setting the bucket count is changed to `max(2*BE or CN count, bucket number calculated based on the largest historical partition data volume)`. The previous rule was to calculate the bucket number based on the largest historical partition data volume). [#47949](https://github.com/StarRocks/starrocks/pull/47949) * Specifying a column list in the INSERT INTO statement on a Primary Key table will perform Partial Updates instead of Full Upsert in earlier versions. ##### Downgrade notes[​](#downgrade-notes-2 "Direct link to Downgrade notes") To downgrade a cluster from v3.3.1 or later to v3.2, users must clean all temporary tables in the cluster by following these steps: 1. Disallow users to create new temporary tables: ```sql ADMIN SET FRONTEND CONFIG("enable_experimental_temporary_table"="false"); ``` 2. Check if there are any temporary tables in the cluster: ```sql SELECT * FROM information_schema.temp_tables; ``` 3. If there are temporary tables in the system, clean them up using the following command (the SYSTEM-level OPERATE privilege is required): ```sql CLEAN TEMPORARY TABLE ON SESSION 'session'; ``` #### 3.3.0[​](#330 "Direct link to 3.3.0") Release date: June 21, 2024 ##### New Features and Improvements[​](#new-features-and-improvements "Direct link to New Features and Improvements") ###### Shared-data Cluster[​](#shared-data-cluster "Direct link to Shared-data Cluster") * Optimized the performance of Schema Evolution in shared-data clusters, reducing the time consumption of DDL changes to a sub-second level. For more information, see [Schema Evolution](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE/#set-fast-schema-evolution). * To satisfy the requirement for data migration from shared-nothing clusters to shared-data clusters, the community officially released the [StarRocks Data Migration Tool](https://docs.starrocks.io/docs/administration/data_migration_tool/). It can also be used for data synchronization and disaster recovery between shared-nothing clusters. * \[Preview] AWS Express One Zone Storage can be used as storage volumes, significantly improving read and write performance. For more information, see [CREATE STORAGE VOLUME](https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME/#properties). * Optimized the garbage collection (GC) mechanism in shared-data clusters. Supports manual compaction for data in object storage. For more information, see [Manual Compaction](https://docs.starrocks.io/docs/sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE/#manual-compaction-from-31). * Optimized the Publish execution of Compaction transactions for Primary Key tables in shared-data clusters, reducing I/O and memory overhead by avoiding reading primary key indexes. * Supports Internal Parallel Scan within tablets. This optimizes query performance in scenarios where there are very few buckets in the table, which limits query parallelism to the number of tablets. Users can enable the Parallel Scan feature by setting the following system variables: ```sql SET GLOBAL enable_lake_tablet_internal_parallel = true; SET GLOBAL tablet_internal_parallel_mode = "force_split"; ``` ###### Data Lake Analytics[​](#data-lake-analytics "Direct link to Data Lake Analytics") * **Data Cache enhancements** * Added the [Data Cache Warmup](https://docs.starrocks.io/docs/data_source/data_cache_warmup/) command CACHE SELECT to fetch hotspot data from data lakes, which speeds up queries and minimizes resource usage. CACHE SELECT can work with SUBMIT TASK to achieve periodic cache warmup. This feature supports both tables in external catalogs and internal tables in shared-data clusters. * Added metrics and monitoring methods to enhance the [observability of Data Cache](https://docs.starrocks.io/docs/data_source/data_cache_observe/). * **Parquet reader performance enhancements** * Optimized Page Index, significantly reducing the data scan size. * Reduced the occurrence of reading unnecessary pages when Page Index is used. * Uses SIMD to accelerate the computation to determine whether data rows are empty. * **ORC reader performance enhancements** * Uses column ID for predicate pushdown to read ORC files after Schema Change. * Optimized the processing logic for ORC tiny stripes. * **Iceberg table format enhancements** * Significantly improved the metadata access performance of the Iceberg Catalog by refactoring the parallel Scan logic. Resolved the single-threaded I/O bottleneck in the native Iceberg SDK when handling large volumes of metadata files. As a result, queries with metadata bottlenecks now experience more than a 10-fold performance increase. * Queries on Parquet-formatted Iceberg v2 tables support [equality deletes](https://docs.starrocks.io/docs/data_source/catalog/iceberg/iceberg_catalog/#usage-notes). * **\[Experimental] Paimon Catalog enhancements** * Materialized views created based on the Paimon external tables now support automatic query rewriting. * Optimized Scan Range scheduling for queries against the Paimon Catalog, improving I/O concurrency. * Support for querying Paimon system tables. * Paimon external tables now support DELETE Vectors, enhancing query efficiency in update and delete scenarios. * **[Enhancements in collecting external table statistics](https://docs.starrocks.io/docs/using_starrocks/Cost_based_optimizer/#collect-statistics-of-hiveiceberghudi-tables)** * ANALYZE TABLE can be used to collect histograms of external tables, which helps prevent data skews. * Supports collecting statistics of STRUCT subfields. * **Table sink enhancements** * The performance of the Sink operator is doubled compared to Trino. * Data can be sunk to Textfile- and ORC-formatted tables in [Hive catalogs](https://docs.starrocks.io/docs/data_source/catalog/hive_catalog/) and storage systems such as HDFS and cloud storage like AWS S3. * \[Preview] Supports Alibaba Cloud [MaxCompute catalogs](https://docs.starrocks.io/docs/data_source/catalog/maxcompute_catalog/), with which you can query data from MaxCompute without ingestion and directly transform and load the data from MaxCompute by using INSERT INTO. * \[Experimental] Supports ClickHouse Catalog. * \[Experimental] Supports [Kudu Catalog](https://docs.starrocks.io/docs/data_source/catalog/kudu_catalog/). ###### Performance Improvement and Query Optimization[​](#performance-improvement-and-query-optimization "Direct link to Performance Improvement and Query Optimization") * **Optimized performance on ARM.** * Significantly optimized performance for ARM architecture instruction sets. Performance tests under AWS Graviton instances showed that the ARM architecture was 11% faster than the x86 architecture in the SSB 100G test, 39% faster in the Clickbench test, 13% faster in the TPC-H 100G test, and 35% faster in the TPC-DS 100G test. * **Spill to Disk is in GA.** Optimized the memory usage of complex queries and improved spill scheduling, allowing large queries to run stably without OOM. * \[Preview] Supports [spilling intermediate results to object storage](https://docs.starrocks.io/docs/administration/management/resource_management/spill_to_disk/#preview-spill-intermediate-result-to-object-storage). * **Supports more indexes.** * \[Preview] Supports [full-text inverted index](https://docs.starrocks.io/docs/table_design/indexes/inverted_index/) to accelerate full-text searches. * \[Preview] Supports [N-Gram bloom filter index](https://docs.starrocks.io/docs/table_design/indexes/Ngram_Bloom_Filter_Index/) to speed up `LIKE` queries and the computation speed of `ngram_search` and `ngram_search_case_insensitive` functions. * Improved the performance and memory usage of Bitmap functions. Added the capability to export Bitmap data to Hive by using [Hive Bitmap UDFs](https://docs.starrocks.io/docs/sql-reference/sql-functions/hive_bitmap_udf/). * **\[Preview] Supports [Flat JSON](https://docs.starrocks.io/docs/using_starrocks/Flat_json/).** This feature automatically detects JSON data during data loading, extracts common fields from the JSON data, and stores these fields in a columnar manner. This improves JSON query performance, comparable to querying STRUCT data. * **\[Preview] Optimized global dictionary.** Provides a dictionary object to store the mapping of key-value pairs from a dictionary table in the BE memory. A new `dictionary_get()` function is now used to directly query the dictionary object in the BE memory, accelerating the speed of querying the dictionary table compared to using the `dict_mapping()` function. Furthermore, the dictionary object can also serve as a dimension table. Dimension values can be obtained by directly querying the dictionary object using `dictionary_get()`, resulting in faster query speeds than the original method of performing JOIN operations on the dimension table to obtain dimension values. * \[Preview] Supports Colocate Group Execution. Significantly reduces memory usage for executing Join and Agg operators on the colocate tables, which ensures that large queries can be executed more stably. * Optimized the performance of CodeGen. JIT is enabled by default, which achieves a 5X performance improvement for complex expression calculations. * Supports using vectorization technology to implement regular expression matching, which reduces the CPU consumption of the `regexp_replace` function. * Optimized Broadcast Join so that the Broadcast Join operation can be terminated in advance when the right table is empty. * Optimized Shuffle Join in scenarios of data skew to prevent OOM. * When an aggregate query contains `Limit`, multiple Pipeline threads can share the `Limit` condition to prevent compute resource consumption. ###### Storage Optimization and Cluster Management[​](#storage-optimization-and-cluster-management "Direct link to Storage Optimization and Cluster Management") * **[Enhanced flexibility of range partitioning](https://docs.starrocks.io/docs/table_design/Data_distribution/#range-partitioning).** Three time functions can be used as partitioning columns. These functions convert timestamps or strings in the partitioning columns into date values and then the data can be partitioned based on the converted date values. * **FE memory observability.** Provides detailed memory usage metrics for each module within the FE to better manage resources. * **[Optimized metadata locks in FE](https://docs.starrocks.io/docs/administration/management/FE_configuration/#lock_manager_enabled).** Provides Lock manager to achieve centralized management for metadata locks in FE. For example, it can refine the granularity of metadata lock from the database level to the table level, which improves load and query concurrency. In a scenario of 100 concurrent load jobs on a small dataset, the load time can be reduced by 35%. * **[Supports adding labels on BEs](https://docs.starrocks.io/docs/administration/management/resource_management/be_label/).** Supports adding labels on BEs based on information such as the racks and data centers where BEs are located. It ensures even data distribution among racks and data centers, and facilitates disaster recovery in case of power failures in certain racks or faults in data centers. * **[Optimized the sort key](https://docs.starrocks.io/docs/table_design/indexes/Prefix_index_sort_key/#usage-notes).** Duplicate Key tables, Aggregate tables, and Unique Key tables all support specifying sort keys through the `ORDER BY` clause. * **\[Experimental] Optimized the storage efficiency of non-string scalar data.** This type of data supports dictionary encoding, reducing storage space usage by 12%. * **Supports size-tiered compaction for Primary Key tables.** Reduces write I/O and memory overhead during compaction. This improvement is supported in both shared-data and shared-nothing clusters. You can use the BE configuration item `enable_pk_size_tiered_compaction_strategy` to control whether to enable this feature (enabled by default). * **Optimized read I/O for persistent indexes in Primary Key tables.** Supports reading persistent indexes by a smaller granularity (page) and improves the persistent index's bloom filter. This improvement is supported in both shared-data and shared-nothing clusters. * Supports for IPv6. StarRocks now supports deployment on IPv6 networks. ###### Materialized Views[​](#materialized-views "Direct link to Materialized Views") * **Supports view-based query rewrite.** With this feature enabled, queries against views can be rewritten to materialized views created upon those views. For more information, see [View-based materialized view rewrite](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views/#view-based-materialized-view-rewrite). * **Supports text-based query rewrite.** With this feature enabled, queries (or their sub-queries) that have the same abstract syntax trees (AST) as the materialized views can be transparently rewritten. For more information, see [Text-based materialized view rewrite](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views/#text-based-materialized-view-rewrite). * **\[Preview] Supports setting transparent rewrite mode for queries directly against the materialized view.** When the `transparent_mv_rewrite_mode` property is enabled, StarRocks will automatically rewrite queries to materialized views. It will merge data from refreshed materialized view partitions with the raw data corresponding to the unrefreshed partitions using an automatic UNION operation. This mode is suitable for modeling scenarios where data consistency must be maintained while also aiming to control refresh frequency and reduce refresh costs. For more information, see [CREATE MATERIALIZED VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/CREATE_MATERIALIZED_VIEW/#parameters-1). * Supports aggregation pushdown for materialized view query rewrite: When the `enable_materialized_view_agg_pushdown_rewrite` variable is enabled, users can use single-table asynchronous materialized views with [Aggregation Rollup](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views/#aggregation-rollup-rewrite) to accelerate multi-table join scenarios. Aggregate functions will be pushed down to the Scan Operator during query execution and rewritten by the materialized view before the Join Operator is executed, significantly improving query efficiency. For more information, see [Aggregation pushdown](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views/#aggregation-pushdown). * **Supports a new property to control materialized view rewrite.** Users can set the `enable_query_rewrite` property to `false` to disable query rewrite based on a specific materialized view, reducing query rewrite overhead. If a materialized view is used only for direct query after modeling and not for query rewrite, users can disable query rewrite for this materialized view. For more information, see [CREATE MATERIALIZED VIEW](https://docs.starrocks.io/docs/sql-reference/sql-statements/materialized_view/CREATE_MATERIALIZED_VIEW/#parameters-1). * **Optimized the cost of materialized view rewrite.** Supports specifying the number of candidate materialized views and enhanced the filter algorithms. Introduced materialized view plan cache to reduce the time consumption of the Optimizer at the query rewrite phase. For more information, see `cbo_materialized_view_rewrite_related_mvs_limit`. * **Optimized materialized views created upon Iceberg catalogs.** Materialized views based on Iceberg catalogs now support incremental refresh triggered by partition updates and partition alignment for Iceberg tables using Partition Transforms. For more information, see [Data lake query acceleration with materialized views](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/data_lake_query_acceleration_with_materialized_views/#choose-a-suitable-refresh-strategy). * **Enhanced the observability of materialized views.** Improved the monitoring and management of materialized views for better system insights. For more information, see [Metrics for asynchronous materialized views](https://docs.starrocks.io/docs/administration/management/monitoring/metrics/#metrics-for-asynchronous-materialized-views). * **Improved the efficiency of large-scale materialized view refresh.** Supports global FIFO scheduling, optimized the cascading refresh strategy for nested materialized views, and fixed some issues that occur in high-frequency refresh scenarios. * **Supports refresh triggered by multiple fact tables.** Materialized views created upon multiple fact tables now support partition-level incremental refresh when data in any of the fact tables is updated, increasing data management flexibility. For more information, see [Align partitions with multiple base tables](https://docs.starrocks.io/docs/using_starrocks/async_mv/use_cases/create_partitioned_materialized_view/#align-partitions-with-multiple-base-tables). ###### SQL Functions[​](#sql-functions "Direct link to SQL Functions") * DATETIME fields support microsecond precision. The new time unit is supported in related time functions and during data loading. * Added the following functions: * [String functions](https://docs.starrocks.io/docs/category/string-1/): crc32, url\_extract\_host, ngram\_search * Array functions: [array\_contains\_seq](https://docs.starrocks.io/docs/sql-reference/sql-functions/array-functions/array_contains_seq/) * Date and time functions: [yearweek](https://docs.starrocks.io/docs/sql-reference/sql-functions/date-time-functions/yearweek/) * Math functions: [cbrt](https://docs.starrocks.io/docs/sql-reference/sql-functions/math-functions/cbrt/) ###### Ecosystem Support[​](#ecosystem-support "Direct link to Ecosystem Support") * \[Experimental] Provides [ClickHouse SQL Rewriter](https://github.com/StarRocks/SQLTransformer), a new tool for converting the syntax in ClickHouse to the syntax in StarRocks. * The Flink connector v1.2.9 provided by StarRocks is integrated with the Flink CDC 3.0 framework, which can build a streaming ELT pipeline from CDC data sources to StarRocks. The pipeline can synchronize the entire database, sharded tables, and schema changes in the sources to StarRocks. For more information, see [Synchronize data with Flink CDC 3.0 (with schema change supported)](https://docs.starrocks.io/docs/loading/Flink-connector-starrocks/#synchronize-data-with-flink-cdc-30-with-schema-change-supported). ##### Behavior and Parameter Changes[​](#behavior-and-parameter-changes "Direct link to Behavior and Parameter Changes") ###### Table Creation and Data Distribution[​](#table-creation-and-data-distribution "Direct link to Table Creation and Data Distribution") * Users must specify Distribution Key when creating a colocate table using CTAS. [#45537](https://github.com/StarRocks/starrocks/pull/45537) * When users create a non-partitioned table without specifying the bucket number, the minimum bucket number the system sets for the table is `16` (instead of `2` based on the formula `2*BE or CN count`). If users want to set a smaller bucket number when creating a small table, they must set it explicitly. [#47005](https://github.com/StarRocks/starrocks/pull/47005) ###### Loading and Unloading[​](#loading-and-unloading "Direct link to Loading and Unloading") * `__op` is reserved by StarRocks for special purposes and creating columns with names prefixed by `__op` is forbidden by default. You can allow this such name format by setting FE configuration `allow_system_reserved_names` to `true`. Please note that creating such columns in Primary Key tables may result in undefined behaviors. [#46239](https://github.com/StarRocks/starrocks/pull/46239) * During Routine Load jobs, if the time duration that StarRocks cannot consume data exceeds the threshold specified in the FE configuration `routine_load_unstable_threshold_second` (Default value is `3600`, that is one hour), the status of the job will become `UNSTABLE`, but the job will continue. [#36222](https://github.com/StarRocks/starrocks/pull/36222) * The default value of the FE configuration `enable_automatic_bucket` is changed from `false` to `true`. When this item is set to `true`, the system will automatically set `bucket_size` for newly created tables, thus enabling automatic bucketing, which is the optimized random bucketing feature. However, in v3.2, setting `enable_automatic_bucket` to `true` will take effect. Instead, the system only enables automatic bucketing when `bucket_size` is specified. This will prevent risks when users downgrade StarRocks from v3.3 to v3.2. ###### Query and Semi-structured Data[​](#query-and-semi-structured-data "Direct link to Query and Semi-structured Data") * When a single query is executed within the Pipeline framework, the memory limit is no longer restricted by `exec_mem_limit` but is only limited by `query_mem_limit`. A value of `0` for `query_mem_limit` indicates no limit. [#34120](https://github.com/StarRocks/starrocks/pull/34120) * NULL values in JSON is treated as SQL NULL values when they are executed by IS NULL and IS NOT NULL operators. For example, `parse_json('{"a": null}') -> 'a' IS NULL` returns `1`, and `parse_json('{"a": null}') -> 'a' IS NOT NULL` returns `0`. [#42765](https://github.com/StarRocks/starrocks/pull/42765) [#42909](https://github.com/StarRocks/starrocks/pull/42909) * A new session variable `cbo_decimal_cast_string_strict` is added to control how CBO converts data from the DECIMAL type to the STRING type. If this variable is set to `true`, the logic built in v2.5.x and later versions prevails and the system implements strict conversion (namely, the system truncates the generated string and fills 0s based on the scale length). If this variable is set to `false`, the logic built in versions earlier than v2.5.x prevails and the system processes all valid digits to generate a string. The default value is `true`. [#34208](https://github.com/StarRocks/starrocks/pull/34208) * The default value of `cbo_eq_base_type` is changed from `varchar` to `decimal`, indicating that the system will compare the DECIMAL-type data with strings as numerical values instead of strings. [#43443](https://github.com/StarRocks/starrocks/pull/43443) ###### Others[​](#others "Direct link to Others") * JDK 11 or later is required from StarRocks v3.3.0 onwards. * The default value of the materialized view property `partition_refresh_num` has been changed from `-1` to `1`. When a partitioned materialized view needs to be refreshed, instead of refreshing all partitions in a single task, the new behavior will incrementally refresh one partition at a time. This change is intended to prevent excessive resource consumption caused by the original behavior. The default behavior can be adjusted using the FE configuration `default_mv_partition_refresh_number`. * Originally, the database consistency checker was scheduled based on GMT+8 time zone. Database consistency checker is scheduled based on the local time zone now. [#45748](https://github.com/StarRocks/starrocks/issues/45748) * By default, Data Cache is enabled to accelerate data lake queries. Users can manually disable it by executing `SET enable_scan_datacache = false`. * If users want to re-use the cached data in Data Cache after downgrading a shared-data cluster from v3.3 to v3.2.8 and earlier, they need to manually rename the Blockfile in the directory **starlet\_cache** by changing the file name format from `blockfile_{n}.{version}` to `blockfile_{n}`, that is, to remove the suffix of version information. For more information, refer to the [Data Cache Usage Notes](https://docs.starrocks.io/docs/using_starrocks/caching/block_cache/#usage-notes). v3.2.9 and later versions are compatible with the file name format in v3.3, so users do not need to perform this operation manually. * Supports dynamically modifying FE parameter `sys_log_level`. [#45062](https://github.com/StarRocks/starrocks/issues/45062) * The default value of the Hive Catalog property `metastore_cache_refresh_interval_sec` is changed from `7200` (two hours) to `60` (one minute). [#46681](https://github.com/StarRocks/starrocks/pull/46681) ##### Bug Fixes[​](#bug-fixes-21 "Direct link to Bug Fixes") Fixed the following issues: * Query results are incorrect when queries are rewritten to materialized views created by using UNION ALL. [#42949](https://github.com/StarRocks/starrocks/issues/42949) * Extra columns are read when queries with predicates are rewritten to materialized views during query execution. [#45272](https://github.com/StarRocks/starrocks/issues/45272) * The results of functions `next_day` and `previous_day` are incorrect. [#45343](https://github.com/StarRocks/starrocks/issues/45343) * Schema change fails because of replica migration. [#45384](https://github.com/StarRocks/starrocks/issues/45384) * Restoring a table with full-text inverted index causes BEs to crash. [#45010](https://github.com/StarRocks/starrocks/issues/45010) * Duplicate data rows are returned when an Iceberg catalog is used to query data. [#44753](https://github.com/StarRocks/starrocks/issues/44753) * Low cardinality dictionary optimization does not take effect on `ARRAY`-type columns in Aggregate tables. [#44702](https://github.com/StarRocks/starrocks/issues/44702) * Query results are incorrect when queries are rewritten to materialized views created by using UNION ALL. [#42949](https://github.com/StarRocks/starrocks/issues/42949) * If BEs are compiled with ASAN, BEs crash when the cluster is started and the `be.warning` log shows `dict_func_expr == nullptr`. [#44551](https://github.com/StarRocks/starrocks/issues/44551) * Query results are incorrect when aggregate queries are performed on single-replica tables. [#43223](https://github.com/StarRocks/starrocks/issues/43223) * View Delta Join rewrite fails. [#43788](https://github.com/StarRocks/starrocks/issues/43788) * BEs crash after the column type is modified from VARCHAR to DECIMAL. [#44406](https://github.com/StarRocks/starrocks/issues/44406) * When a table with List partitioning is queried by using a not-equal operator, partitions are incorrectly pruned, resulting in wrong query results. [#42907](https://github.com/StarRocks/starrocks/issues/42907) * Leader FE's heap size increases quickly as many Stream Load jobs using non-transactional interface finishes. [#43715](https://github.com/StarRocks/starrocks/issues/43715) ##### Downgrade notes[​](#downgrade-notes-3 "Direct link to Downgrade notes") To downgrade a cluster from v3.3.0 or later to v3.2, users must follow these steps: 1. Ensure that all ALTER TABLE SCHEMA CHANGE transactions initiated in the v3.3 cluster are either completed or canceled before downgrading. 2. Clear all transaction history by executing the following command: ```sql ADMIN SET FRONTEND CONFIG ("history_job_keep_max_second" = "0"); ``` 3. Verify that there are no remaining historical records by running the following command: ```sql SHOW PROC '/jobs//schema_change'; ``` 4. If you want to downgrade the cluster to a patch version earlier than v3.2.8 or v3.1.14, you must drop all asynchronous materialized views you have created using `PROPERTIES('compression' = 'lz4')`. 5. Execute the following command to create an image file for your metadata: ```sql ALTER SYSTEM CREATE IMAGE; ``` 6. After the new image file is transmitted to the directory **meta/image** of all FE nodes, you can first downgrade a Follower FE node. If no error is returned, you can then downgrade other nodes in the cluster. --- ## Release 3.4 ### StarRocks version 3.4 #### 3.4.10[​](#3410 "Direct link to 3.4.10") Release Date: January 12, 2026 ##### Improvements[​](#improvements "Direct link to Improvements") * Supports pushing down GROUP BY expressions to scan operators and rewriting through materialized views, further improving query performance [#66546](https://github.com/StarRocks/starrocks/pull/66546) * Added a configuration switch for the Hudi library internal metadata table, allowing users to disable it when encountering performance issues. [#67581](https://github.com/StarRocks/starrocks/pull/67581) ##### Bug Fixes[​](#bug-fixes "Direct link to Bug Fixes") The following issues have been fixed: * CVE-2025-12183 and CVE-2025-66566. [#66373](https://github.com/StarRocks/starrocks/pull/66373) [#66480](https://github.com/StarRocks/starrocks/pull/66480) * In multi-statement submission scenarios, the SQL/statement information recorded in the Profile may be incorrect, leading to unreliable troubleshooting and performance analysis data. [#67119](https://github.com/StarRocks/starrocks/pull/67119) * Java UDF/UDAF parameter conversion may take an abnormal path when the input column is "all NULL and nullable", causing Java heap memory to balloon abnormally and potentially triggering OOM. [#67105](https://github.com/StarRocks/starrocks/pull/67105) * When there is no `PARTITION BY`/`GROUP BY` and the window function is a ranking type (`row_number`/`rank`/`dense_rank`), the optimizer may generate an invalid execution plan (TOP-N with empty ORDER BY + MERGING-EXCHANGE), causing BE to crash. [#67085](https://github.com/StarRocks/starrocks/pull/67085) * After operations like resizing, deserialization, or filtering on `Object`/JSON columns, the internal pointer cache may still point to old addresses (dangling pointers), returning `nullptr` when reading object values and causing segmentation faults/data corruption. [#66990](https://github.com/StarRocks/starrocks/pull/66990) * `trim()` may trigger vector out-of-bounds operations (for example, underflow on empty slices) with specific Unicode whitespace characters/boundary inputs, causing BE internal errors or crashes. [#66484](https://github.com/StarRocks/starrocks/pull/66484) * `trim()` has improper buffer reservation length calculation, potentially causing insufficient reservation and frequent expansions, triggering exception paths, and leading to BE internal errors in severe cases. [#66489](https://github.com/StarRocks/starrocks/pull/66489) * After table value function rewrite (from `bitmap_to_array` to `unnest_bitmap`), the projection column type may be incorrectly inferred (bitmap mistaken as `ARRAY``<``BIGINT``>`), leading to type inconsistency risks in subsequent plans/execution. [#66986](https://github.com/StarRocks/starrocks/pull/66986) * When BE triggers crash handling on fatal signals (for example, SIGSEGV), the heartbeat service may still briefly return success, causing FE to mistakenly believe the BE is alive (until heartbeat timeout), potentially scheduling queries to the crashed node during this period. [#66250](https://github.com/StarRocks/starrocks/pull/66250) * Race condition exists between initial submission and dynamic driver addition (`submit_next_driver`) in Execution Groups, potentially causing "driver already blocked yet added to schedule" assertion failures and BE crashes. [#66111](https://github.com/StarRocks/starrocks/pull/66111) * When pushing down DISTINCT + LIMIT predicates, the global LIMIT may be incorrectly applied before the Exchange node, causing data to be truncated prematurely and result sets to miss some rows. [#66129](https://github.com/StarRocks/starrocks/pull/66129) * When using ExecutionGroup (group execution) mode, if JOIN is followed by window functions, data may be out of order or duplicated, leading to incorrect results. [#66458](https://github.com/StarRocks/starrocks/pull/66458) * In audit logs and query statistics, statistics like scan row count may be missing or inaccurate in some high-selectivity filtering scenarios, causing monitoring and troubleshooting information to be distorted. [#66422](https://github.com/StarRocks/starrocks/pull/66422) * When CASE-WHEN nesting is deep and each layer has many branches, the expression tree node count may explode exponentially, causing FE OOM. [#66379](https://github.com/StarRocks/starrocks/pull/66379) * The `percentile_approx_weighted` function may access the wrong parameter position when obtaining the compression factor from const parameters, causing BE crashes. [#65217](https://github.com/StarRocks/starrocks/pull/65217) * During BE startup, when loading tablet metadata, if RocksDB iteration times out, it may discard already-loaded tablets and retry from the beginning; in cross-disk migration scenarios, this may lead to version loss. [#65445](https://github.com/StarRocks/starrocks/pull/65445) [#65427](https://github.com/StarRocks/starrocks/pull/65427) * Stream Load may fail during transaction commit due to invalid tablet references (for example, deleted during ALTER). [#65986](https://github.com/StarRocks/starrocks/pull/65986) * When rowset COMMIT or Compaction COMMIT fails on Primary Key tables, the rowset ID is not released, causing files to not be garbage-collected and disk space to leak. [#66336](https://github.com/StarRocks/starrocks/pull/66336) * DELETE statements during partition pruning may attempt materialized view rewrite preparation, potentially blocking or failing DELETE due to table lock order or deadlock issues. [#65818](https://github.com/StarRocks/starrocks/pull/65818) [#65820](https://github.com/StarRocks/starrocks/pull/65820) * When the same table is referenced multiple times in a query, scan nodes may concurrently allocate partition IDs, potentially causing ID conflicts and partition mapping confusion, leading to incorrect query results. [#65608](https://github.com/StarRocks/starrocks/pull/65608) * When column-mode partial update is used together with conditional update, loading may fail with "invalid rssid" errors. [#66217](https://github.com/StarRocks/starrocks/pull/66217) * When concurrent transactions create temporary partitions with the same partition values but different transaction IDs, "Duplicate values" errors may cause automatic partition creation to fail. [#66203](https://github.com/StarRocks/starrocks/pull/66203) [#66398](https://github.com/StarRocks/starrocks/pull/66398) * Clone task checks the wrong status variable during cleanup after `_finish_clone_primary` failure, potentially preventing cleanup logic from executing correctly. [#65765](https://github.com/StarRocks/starrocks/pull/65765) * When DROP and CLONE tasks execute concurrently on the same tablet, the only replica may be deleted by DROP, causing query failures. [#66271](https://github.com/StarRocks/starrocks/pull/66271) * In Spilling scenarios, very large string encoding may cause BE crashes due to buffer reservation errors or type overflows. [#65373](https://github.com/StarRocks/starrocks/pull/65373) * When using CACHE SELECT functionality, if column iterators are not properly sought or the schema is incorrectly reordered, assertion failures or data confusion may occur, causing BE crashes. [#66276](https://github.com/StarRocks/starrocks/pull/66276) * External table (file format schema detection) sampling scans may experience range index out-of-bounds, causing BE crashes or reading incorrect data. [#65931](https://github.com/StarRocks/starrocks/pull/65931) * When materialized views are based on VIEW + JOIN scenarios and the view name matches a base table name (but in different databases), partition expression parsing may fail, causing materialized view creation errors. [#66315](https://github.com/StarRocks/starrocks/pull/66315) * When materialized views refresh on multi-level partitioned base tables, only parent partition metadata (ID/Version) is checked, not sub-partition changes, causing materialized view not to refresh after sub-partition data updates. [#66108](https://github.com/StarRocks/starrocks/pull/66108) * After the Iceberg table snapshot expiration, the partition `last_updated_at` may be null, causing materialized views depending on that table to fail to track partition changes correctly and skip refreshes. [#66044](https://github.com/StarRocks/starrocks/pull/66044) * When queries contain the same table multiple times with different partition predicates, materialized view compensation (MVCompensation) may confuse partition information, leading to incorrect rewrites. [#66416](https://github.com/StarRocks/starrocks/pull/66416) * Text-based materialized view rewrite after AST cache hits does not refresh metadata, potentially using outdated tablet information and causing query failures or data inconsistencies. [#66583](https://github.com/StarRocks/starrocks/pull/66583) * Low cardinality optimization has bugs in disabled column propagation logic, potentially causing incorrect column disabling and wrong query results. [#66771](https://github.com/StarRocks/starrocks/pull/66771) * PRIMARY KEY tables with low cardinality optimization enabled may crash or produce incorrect data due to incompatible global dictionary collection logic. [#66739](https://github.com/StarRocks/starrocks/pull/66739) * Nested CTEs with partial inlining and partial reuse scenarios may have overly strict optimizer checks, rejecting valid plans. [#66703](https://github.com/StarRocks/starrocks/pull/66703) * After merging UNION to constants (VALUES), output column nullability may be incorrectly set, causing downstream operators to crash or produce incorrect results. [#65454](https://github.com/StarRocks/starrocks/pull/65454) * Partition column min/max rewrite optimization may generate invalid TOP-N in scenarios without PARTITION BY/ORDER BY, causing BE crashes or incorrect results. [#66498](https://github.com/StarRocks/starrocks/pull/66498) * Non-deterministic functions (e.g., `now()`) are incorrectly pushed down to lower operators, potentially causing results to be inconsistent across different operators/shards. [#66391](https://github.com/StarRocks/starrocks/pull/66391) * Foreign key constraints are lost after FE restart because `MaterializedView. onCreate()` does not trigger constraint rebuilding and registration. [#66615](https://github.com/StarRocks/starrocks/pull/66615) * When materialized views contain `colocate_with` property, metadata is not written to Edit Log, causing follower FE to be unaware of colocate relationships and query performance degradation. [#65840](https://github.com/StarRocks/starrocks/pull/65840) [#65405](https://github.com/StarRocks/starrocks/pull/65405) * After a warehouse is deleted, `SHOW LOAD` or `information_schema.loads` queries may fail; original sessions cannot execute any SQL (including switching warehouses). [#66464](https://github.com/StarRocks/starrocks/pull/66464) * If the tablet statistics report is untimely, table cardinality estimation may incorrectly use sample-type statistics in some edge cases, with cardinality of 1, causing execution plans to deviate significantly. [#65655](https://github.com/StarRocks/starrocks/pull/65655) * Tablet statistics reporting timing issues may cause partition row counts to be 0, rendering table cardinality estimation completely ineffective. [#65266](https://github.com/StarRocks/starrocks/pull/65266) * FE's locking order on TransactionState during `createPartition` is opposite to Gson serialization, potentially causing deadlocks. [#65792](https://github.com/StarRocks/starrocks/pull/65792) * DELETE VECTOR CRC32 may fail validation in ABA upgrade/downgrade scenarios due to version incompatibilities, causing query errors. [#65436](https://github.com/StarRocks/starrocks/pull/65436) [#65421](https://github.com/StarRocks/starrocks/pull/65421) [#65475](https://github.com/StarRocks/starrocks/pull/65475) [#65483](https://github.com/StarRocks/starrocks/pull/65483) * `map_agg` aggregate function may trigger crashes on specific inputs. [#67460](https://github.com/StarRocks/starrocks/pull/67460) * When `flat_path` is empty, calling `substr(1)` triggers `std::out_of_range` exception, causing BE crashes. [#65386](https://github.com/StarRocks/starrocks/pull/65386) * In shared-data clusters, compression configurations do not take effect correctly during table creation and schema changes. [#65778](https://github.com/StarRocks/starrocks/pull/65778) * When adding columns with default values, concurrent INSERTs may fail due to invalid column references. [#66107](https://github.com/StarRocks/starrocks/pull/66107) [#65968](https://github.com/StarRocks/starrocks/pull/65968) * Segment iterator selection order is inconsistent during scan initialization between shared-data and shared-nothing scenarios, potentially causing inconsistent scan behavior. [#65782](https://github.com/StarRocks/starrocks/pull/65782) [#61171](https://github.com/StarRocks/starrocks/pull/61171) * merge\_condition is not supported when merge commit is enabled, causing partial update scenarios to fail. [#65278](https://github.com/StarRocks/starrocks/pull/65278) * Image journal ID retrieval logic is incorrect, potentially causing cluster snapshot functionality abnormalities. [#65989](https://github.com/StarRocks/starrocks/pull/65989) * LDAP users trigger NPE in TaskRun scenarios due to null `ConnectContext. get()`, causing task failures. [#65877](https://github.com/StarRocks/starrocks/pull/65877) * When ANALYZE statements execute on follower FE, RPC timeout still uses `query_timeout` instead of `statistic_collect_query_timeout`, potentially causing timeouts too early or too late. [#66785](https://github.com/StarRocks/starrocks/pull/66785) * `MemoryScratchSinkOperator` cannot properly finish pending\_finish after RecordBatchQueue shutdown, causing tasks to hang. [#66095](https://github.com/StarRocks/starrocks/pull/66095) * Query error rate metric calculations use the wrong variables, potentially producing negative values or inaccuracies. [#65901](https://github.com/StarRocks/starrocks/pull/65901) * Load profile counters may be updated repeatedly, causing inflated statistics. [#65352](https://github.com/StarRocks/starrocks/pull/65352) * In multi-task deployment (deploy more tasks) scenarios, the Profile collection thread context is not switched correctly, causing some metrics to be lost. [#65733](https://github.com/StarRocks/starrocks/pull/65733) * Local/Lake TabletsChannel lifecycle management has circular lock waiting risks: deadlocks may occur on specific close/deregister paths, affecting import/write task availability. [#66820](https://github.com/StarRocks/starrocks/pull/66820) * Filesystem instance cache (filesystem cache) causes query performance to drop significantly and cannot recover after capacity is set to 0 (due to cache key and instance mismatch). [#65979](https://github.com/StarRocks/starrocks/pull/65979) #### 3.4.9[​](#349 "Direct link to 3.4.9") Release Date: November 24, 2025 ##### Behavior Changes[​](#behavior-changes "Direct link to Behavior Changes") * Changed the return type of `json_extract` in the Trino dialect from STRING to JSON. This may cause incompatibility in CAST, UNNEST, and type check logic. [#59718](https://github.com/StarRocks/starrocks/pull/59718) * The metric that reports “connections per user” under `/metrics` now requires admin authentication. Without authentication, only total connection counts are exposed, preventing information leakage of all usernames via metrics. [#64635](https://github.com/StarRocks/starrocks/pull/64635) * Removed the deprecated system variable `analyze_mv`. Materialized view refresh no longer automatically triggers ANALYZE jobs, avoiding large numbers of background statistics tasks. This changes expectations for users relying on legacy behavior. [#64863](https://github.com/StarRocks/starrocks/pull/64863) * Changed the overflow detection logic of casting from LARGEINT to DECIMAL128 on x86. `INT128_MIN * 1` is no longer considered an overflow to ensure consistent casting semantics for extreme values. [#63559](https://github.com/StarRocks/starrocks/pull/63559) * Added a configurable table-level lock timeout to `finishTransaction`. If a table lock cannot be acquired within the timeout, finishing the transaction will fail for this round and be retried later, rather than blocking indefinitely. The final result is unchanged, but the lock behavior is more explicit. [#63981](https://github.com/StarRocks/starrocks/pull/63981) ##### Bug Fixes[​](#bug-fixes-1 "Direct link to Bug Fixes") The following issues have been fixed: * During BE start, if loading tablet metadata from RocksDB times out, RocksDB may restart loading from the beginning and accidentally pick up stale tablet entries, risking data version loss. [#65146](https://github.com/StarRocks/starrocks/pull/65146) * Data corruption issues related to CRC32C checksum for delete-vectors of Lake Primary Key tables. [#65006](https://github.com/StarRocks/starrocks/pull/65006) [#65354](https://github.com/StarRocks/starrocks/pull/65354) [#65442](https://github.com/StarRocks/starrocks/pull/65442) [#65354](https://github.com/StarRocks/starrocks/pull/65354) * When the internal `flat_path` string is empty because the JSON hyper extraction path is `$` or all paths are skipped, calling `substr` will throw an exception and cause BE crash. [#65260](https://github.com/StarRocks/starrocks/pull/65260) * When spilling large strings to disk, insufficient length checks, using 32‑bit attachment sizes, and issues in the BlockReader could cause crashes. [#65373](https://github.com/StarRocks/starrocks/pull/65373) * When multiple HTTP requests reuse the same TCP connection, if a non‑ExecuteSQL request arrives after an ExecuteSQL request, the `HttpConnectContext` cannot be unregistered at channel close, causing HTTP context leaks. [#65203](https://github.com/StarRocks/starrocks/pull/65203) * Primitive value loss issue under certain circumstances when JSON data is being flattened. [#64939](https://github.com/StarRocks/starrocks/pull/64939) [#64703](https://github.com/StarRocks/starrocks/pull/64703) * Crash in `ChunkAccumulator` when chunks are appended with incompatible JSON schemas. [#64894](https://github.com/StarRocks/starrocks/pull/64894) * In `AsyncFlushOutputStream`, asynchronous I/O tasks may attempt to access a destroyed `MemTracker`, resulting in use‑after‑free crashes. [#64735](https://github.com/StarRocks/starrocks/pull/64735) * Concurrent Compaction tasks against the same Lake Primary Key table lack integrity checks, which could leave metadata in an inconsistent state after a failed publish. [#65005](https://github.com/StarRocks/starrocks/pull/65005) * When spilling Hash Joins, if the build side’s `set_finishing` task failed, it only recorded the status in the spiller, allowing the Probe side to continue, and eventually causing a crash or an indefinite loop. [#65027](https://github.com/StarRocks/starrocks/pull/65027) * During tablet migration, if the only newest replica is marked as DECOMMISSION, the version of the target replica is outdated and stuck at VERSION\_INCOMPLETE. [#62942](https://github.com/StarRocks/starrocks/pull/62942) * Use-after-free issue because the relevant Block Group is not released when `PartitionedSpillerWriter` is removing partitions. [#63903](https://github.com/StarRocks/starrocks/pull/63903) [#63825](https://github.com/StarRocks/starrocks/pull/63825) * BE crash caused by MorselQueue's failure to get splits. [#62753](https://github.com/StarRocks/starrocks/pull/62753) * In shared-data clusters, Sorted-by-key Scans with multiple I/O tasks could produce wrong results in sort-based aggregations. [#63849](https://github.com/StarRocks/starrocks/pull/63849) * On ARM, reading Parquet columns for certain Hive external tables could crash in LZ4 conversion when copying NULL bitmaps because the destination null buffer pointer was stale due to out-of-order execution. [#63294](https://github.com/StarRocks/starrocks/pull/63294) #### 3.4.8[​](#348 "Direct link to 3.4.8") Release Date: September 30, 2025 ##### Behavior Changes[​](#behavior-changes-1 "Direct link to Behavior Changes") * By setting the default value of `enable_lake_tablet_internal_parallel` to `true`, Parallel Scan for Cloud-native tables in shared-data clusters is enabled by default to increase per‑query internal parallelism. It may raise peak resource usage. [#62159](https://github.com/StarRocks/starrocks/pull/62159) ##### Bug Fixes[​](#bug-fixes-2 "Direct link to Bug Fixes") The following issues have been fixed: * Delta Lake partition column names were forcibly converted to lowercase, causing a mismatch with the actual column names. [#62953](https://github.com/StarRocks/starrocks/pull/62953) * The Iceberg manifest cache eviction race could trigger a NullPointerException (NPE). [#](https://github.com/StarRocks/starrocks/pull/63052)[#63043](https://github.com/StarRocks/starrocks/pull/63043) * Uncaught generic exceptions during the Iceberg scan phase interrupted scan range submission and produced no metrics. [#62994](https://github.com/StarRocks/starrocks/pull/62994) * Complex multi-layer projected views used in materialized view rewrite produced invalid plans or missing column statistics. [#62918](https://github.com/StarRocks/starrocks/pull/62918) [#62198](https://github.com/StarRocks/starrocks/pull/62198) * Case mismatch of partition columns in the Hive table-based materialized view was incorrectly rejected. [#62598](https://github.com/StarRocks/starrocks/pull/62598) * Materialized view refresh used only the creator’s default role, causing an insufficient privilege issue. [#62396](https://github.com/StarRocks/starrocks/pull/62396) * Case-insensitive conflicts in partition names of list-partitioned materialized views led to duplicate name errors. [#62389](https://github.com/StarRocks/starrocks/pull/62389) * Residual version mapping after failed materialized view restores caused subsequent incremental refresh to be skipped, returning empty results. [#62634](https://github.com/StarRocks/starrocks/pull/62634) * Abnormal partitions after materialized view restores caused FE restart NullPointerException. [#62563](https://github.com/StarRocks/starrocks/pull/62563) * Non-global aggregation queries incorrectly applied the aggregation pushdown rewrite, producing invalid plans. [#63060](https://github.com/StarRocks/starrocks/pull/63060) * The tablet deletion state was only updated in memory and not persisted, so GC still treated it as running and skipped reclamation. [#63623](https://github.com/StarRocks/starrocks/pull/63623) * Concurrent query and drop tablet led to early delvec cleanup and "no delete vector found" errors. [#63291](https://github.com/StarRocks/starrocks/pull/63291) * An issue with base and cumulative compaction for the Primary Key index sharing the same `max_rss_rowid`. [#63277](https://github.com/StarRocks/starrocks/pull/63277) * Possible BE crash when LakePersistentIndex destructor runs after a failed initialization. [#62279](https://github.com/StarRocks/starrocks/pull/62279) * Graceful shutdown of Publish thread pool silently discarded queued tasks without marking failures, creating version holes and a false "all succeeded" impression. [#62417](https://github.com/StarRocks/starrocks/pull/62417) * The newly cloned replica on a newly added BE during rebalance was immediately judged redundant and removed, preventing data migration to the new node. [#62542](https://github.com/StarRocks/starrocks/pull/62542) * Missing lock when reading the tablet's maximum version caused inconsistent replication transaction decisions. [#62238](https://github.com/StarRocks/starrocks/pull/62238) * A combination of `date_trunc` equality and raw column range predicate was reduced to a point interval, returning empty result sets (for example, `date_trunc('month', dt)='2025-09-01' AND dt>'2025-09-23'`). [#63464](https://github.com/StarRocks/starrocks/pull/63464) * Pushdown of non-deterministic predicates (random/time functions) produced inconsistent results. [#63495](https://github.com/StarRocks/starrocks/pull/63495) * Missing consumer node after CTE reuse decision produced incomplete execution plans. [#62784](https://github.com/StarRocks/starrocks/pull/62784) * Type mismatch crashes when table functions and low-cardinality dictionary encoding coexist. [#62466](https://github.com/StarRocks/starrocks/pull/62466) [#62292](https://github.com/StarRocks/starrocks/pull/62292) * Oversized CSV split into parallel fragments caused every fragment to skip header rows, leading to data loss. [#62719](https://github.com/StarRocks/starrocks/pull/62719) * `SHOW CREATE ROUTINE LOAD` without explicit DB returned job from another database with the same name. [#62745](https://github.com/StarRocks/starrocks/pull/62745) * NullPointerException when `sameLabelJobs` became null during concurrent load job cleanup. [#63042](https://github.com/StarRocks/starrocks/pull/63042) * BE decommission blocked even when all tablets were already in the recycle bin. [#62781](https://github.com/StarRocks/starrocks/pull/62781) * `OPTIMIZE TABLE` task stuck in PENDING after thread pool rejection. [#62300](https://github.com/StarRocks/starrocks/pull/62300) * Dirty tablet metadata cleanup used GTID arguments in the wrong order. [62275](https://github.com/StarRocks/starrocks/pull/62275) #### 3.4.7[​](#347 "Direct link to 3.4.7") Release Date: September 1, 2025 ##### Bug Fixes[​](#bug-fixes-3 "Direct link to Bug Fixes") The following issues have been fixed: * Routine Load jobs did not serialize `max_filter_ratio`. [#61755](https://github.com/StarRocks/starrocks/pull/61755) * In Stream Load, the `now(precision)` function lost the precision parameter. [#61721](https://github.com/StarRocks/starrocks/pull/61721) * In Audit Log, the Scan Rows result for `INSERT INTO SELECT` statements was inaccurate. [#61381](https://github.com/StarRocks/starrocks/pull/61381) * After upgrading the cluster to v3.4.5, the `fslib read iops` metric increased compared to before the upgrade. [#61724](https://github.com/StarRocks/starrocks/pull/61724) * Queries against SQLServer using JDBC Catalog often got stuck. [#61719](https://github.com/StarRocks/starrocks/pull/61719) #### 3.4.6[​](#346 "Direct link to 3.4.6") Release Date: August 7, 2025 ##### Improvements[​](#improvements-1 "Direct link to Improvements") * When exporting data to Parquet files using `INSERT INTO FILES`, you can now specify the Parquet version via the [`parquet.version`](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files.md#parquetversion) property to improve compatibility with other tools when reading the exported files. [#60843](https://github.com/StarRocks/starrocks/pull/60843) ##### Bug Fixes[​](#bug-fixes-4 "Direct link to Bug Fixes") The following issues have been fixed: * Loading jobs failed due to overly coarse lock granularity in `TableMetricsManager`. [#58911](https://github.com/StarRocks/starrocks/pull/58911) * Case sensitivity issue in column names when loading Parquet data via `FILES()`. [#61059](https://github.com/StarRocks/starrocks/pull/61059) * Cache did not take effect after upgrading a shared-data cluster from v3.3 to v3.4 or later. [#60973](https://github.com/StarRocks/starrocks/pull/60973) * A division-by-zero error occurred when the partition ID was null, causing a BE crash. [#60842](https://github.com/StarRocks/starrocks/pull/60842) * Broker Load jobs failed during BE scaling. [#60224](https://github.com/StarRocks/starrocks/pull/60224) ##### Behavior Changes[​](#behavior-changes-2 "Direct link to Behavior Changes") * The `keyword` column in the `information_schema.keywords` view has been renamed to `word` to align with the MySQL definition. [#60863](https://github.com/StarRocks/starrocks/pull/60863) #### 3.4.5[​](#345 "Direct link to 3.4.5") Release Date: July 10, 2025 ##### Improvements[​](#improvements-2 "Direct link to Improvements") * Enhanced observability of loading job execution: Unified the runtime information of loading tasks into the `information_schema.loads` view. Users can view the execution details of all INSERT, Broker Load, Stream Load, and Routine Load subtasks in this view. Additional fields have been added to help users better understand the status of loading tasks and the association with parent jobs (PIPES, Routine Load Jobs). * Support modifying `kafka_broker_list` via the `ALTER ROUTINE LOAD` statement. ##### Bug Fixes[​](#bug-fixes-5 "Direct link to Bug Fixes") The following issues have been fixed: * Under high-frequency loading scenarios, Compaction could be delayed. [#59998](https://github.com/StarRocks/starrocks/pull/59998) * Querying Iceberg external tables via Unified Catalog would throw an error: `not support getting unified metadata table factory`. [#59412](https://github.com/StarRocks/starrocks/pull/59412) * When using `DESC FILES()` to view CSV files in remote storage, incorrect results were returned because the system mistakenly inferred `xinf` as the FLOAT type. [#59574](https://github.com/StarRocks/starrocks/pull/59574) * `INSERT INTO` could cause BE to crash when encountering empty partitions. [#59553](https://github.com/StarRocks/starrocks/pull/59553) * When StarRocks reads Equality Delete files in Iceberg, it could still access deleted data if the data had already been removed from the Iceberg table. [#59709](https://github.com/StarRocks/starrocks/pull/59709) * Query failures caused by renaming columns. [#59178](https://github.com/StarRocks/starrocks/pull/59178) ##### Behavior Changes[​](#behavior-changes-3 "Direct link to Behavior Changes") * The default value of the BE configuration item `skip_pk_preload` has been changed from `false` to `true`. As a result, the system will skip preloading Primary Key Indexes for Primary Key tables to reduce the likelihood of `Reached Timeout` errors. This change may increase query latency for operations that require loading Primary Key Indexes. #### 3.4.4[​](#344 "Direct link to 3.4.4") Release Date: June 10, 2025 ##### Improvements[​](#improvements-3 "Direct link to Improvements") * Storage Volume now supports ADLS2 using Managed Identity as the credential. [#58454](https://github.com/StarRocks/starrocks/pull/58454) * For [partitions based on complex time function expressions](https://docs.starrocks.io/docs/table_design/data_distribution/expression_partitioning/#partitioning-based-on-a-complex-time-function-expression-since-v34), partition pruning works well for partitions based on most DATETIME-related functions * Supports loading Avro data files from Azure using the `FILES` function. [#58131](https://github.com/StarRocks/starrocks/pull/58131) * When Routine Load encounters invalid JSON data, the consumed partition and offset information is logged in the error log to facilitate troubleshooting. [#55772](https://github.com/StarRocks/starrocks/pull/55772) ##### Bug Fixes[​](#bug-fixes-6 "Direct link to Bug Fixes") The following issues have been fixed: * Concurrent queries accessing the same partition in a partitioned table caused Hive Metastore to hang. [#58089](https://github.com/StarRocks/starrocks/pull/58089) * Abnormal termination of `INSERT` tasks caused the job to remain in the `QUEUEING` state. [#58603](https://github.com/StarRocks/starrocks/pull/58603) * After upgrading the cluster from v3.4.0 to v3.4.2, a large number of tablet replicas encounter exceptions. [#58518](https://github.com/StarRocks/starrocks/pull/58518) * FE OOM caused by incorrect `UNION` execution plans. [#59040](https://github.com/StarRocks/starrocks/pull/59040) * Invalid database IDs during partition recycling could cause FE startup to fail. [#59666](https://github.com/StarRocks/starrocks/pull/59666) * After a failed FE CheckPoint operation, the process could not exit properly, resulting in blocking. [#58602](https://github.com/StarRocks/starrocks/pull/58602) #### 3.4.3[​](#343 "Direct link to 3.4.3") Release Date: April 30, 2025 ##### Improvements[​](#improvements-4 "Direct link to Improvements") * Routine Load and Stream Load support the use of Lambda expressions in the `columns` parameter for complex column data extraction. `array_filter`/`map_filter` can be used to filter and extract ARRAY/MAP data. Complex filtering and extraction of JSON data can be achieved by combining the `cast` function to convert JSON array/JSON object to ARRAY and MAP types. For example, `COLUMNS (js, col=array_filter(i -> json_query(i, '$.type')=='t1', cast(js as Array))[1])` can extract the first JSON object from the JSON array `js` where `type` is `t1`. [#58149](https://github.com/StarRocks/starrocks/pull/58149) * Supports converting JSON objects to MAP type using the `cast` function, combined with `map_filter` to extract items from the JSON object that meet specific conditions. For example, `map_filter((k, v) -> json_query(v, '$.type') == 't1', cast(js AS MAP))` can extract the JSON object from `js` where `type` is `t1`. [#58045](https://github.com/StarRocks/starrocks/pull/58045) * LIMIT is now supported when querying the `information_schema.task_runs` view. [#57404](https://github.com/StarRocks/starrocks/pull/57404) ##### Bug Fixes[​](#bug-fixes-7 "Direct link to Bug Fixes") The following issues have been fixed: * Queries against ORC format Hive tables are returned with an error `OrcChunkReader::lazy_seek_to failed. reason = bad read in RleDecoderV2: :readByte`. [#57454](https://github.com/StarRocks/starrocks/pull/57454) * RuntimeFilter from the upper layer could not be pushed down when querying Iceberg tables that contain Equality Delete files. [#57651](https://github.com/StarRocks/starrocks/pull/57651) * Enabling the spill-to-disk pre-aggregation strategy causes queries to fail. [#58022](https://github.com/StarRocks/starrocks/pull/58022) * Queries are returned with an error `ConstantRef-cmp-ConstantRef not supported here, null != 111 should be eliminated earlier`. [#57735](https://github.com/StarRocks/starrocks/pull/57735) * Query timeout with the `query_queue_pending_timeout_second` parameter while the Query Queue feature is not enabled. [#57719](https://github.com/StarRocks/starrocks/pull/57719) #### 3.4.2[​](#342 "Direct link to 3.4.2") Release Date: April 10, 2025 ##### Improvements[​](#improvements-5 "Direct link to Improvements") * FE supports graceful shutdown to improve system availability. When exiting FE via `./stop_fe.sh -g`, FE will first return a 500 status code to the front-end Load Balancer via the `/api/health` API to indicate that it is preparing to shut down, allowing the Load Balancer to switch to other available FE nodes. Meanwhile, FE will continue to run ongoing queries until they finish or timeout (default timeout: 60 seconds). [#56823](https://github.com/StarRocks/starrocks/pull/56823) ##### Bug Fixes[​](#bug-fixes-8 "Direct link to Bug Fixes") The following issues have been fixed: * Partition pruning might not work if the partition column is a generated column. [#54543](https://github.com/StarRocks/starrocks/pull/54543) * Incorrect parameter handling in the `concat` function could cause a BE crash during query execution. [#57522](https://github.com/StarRocks/starrocks/pull/57522) * The `ssl_enable` property did not take effect when using Broker Load to load data. [#57229](https://github.com/StarRocks/starrocks/pull/57229) * When NULL values exist, querying subfields of STRUCT-type columns could cause a BE crash. [#56496](https://github.com/StarRocks/starrocks/pull/56496) * When modifying the bucket distribution of a table with the statement `ALTER TABLE {table} PARTITIONS (p1, p1) DISTRIBUTED BY ...`, specifying duplicate partition names could result in failure to delete internally generated temporary partitions. [#57005](https://github.com/StarRocks/starrocks/pull/57005) * In a shared-data cluster, running `SHOW PROC '/current_queries'` resulted in the error "Error 1064 (HY000): Sending collect query statistics request fails". [#56597](https://github.com/StarRocks/starrocks/pull/56597) * Running `INSERT OVERWRITE` loading tasks in parallel caused the error "ConcurrentModificationException: null", resulting in loading failure. [#56557](https://github.com/StarRocks/starrocks/pull/56557) * After upgrading from v2.5.21 to v3.1.17, running multiple Broker Load tasks concurrently could cause exceptions. [#56512](https://github.com/StarRocks/starrocks/pull/56512) ##### Behavior Changes[​](#behavior-changes-4 "Direct link to Behavior Changes") * The default value of the BE configuration item `avro_ignore_union_type_tag` has been changed to `true`, enabling the direct parsing of `["NULL", "STRING"]` as STRING type data, which better aligns with typical user requirements. [#57553](https://github.com/StarRocks/starrocks/pull/57553) * The default value of the session variable `big_query_profile_threshold` has been changed from 0 to 30 (seconds). [#57177](https://github.com/StarRocks/starrocks/pull/57177) * A new FE configuration item `enable_mv_refresh_collect_profile` has been added to control whether to collect Profile information during materialized view refresh. The default value is `false` (previously, the system collected Profile by default). [#56971](https://github.com/StarRocks/starrocks/pull/56971) #### 3.4.1 (Yanked)[​](#341-yanked "Direct link to 3.4.1 (Yanked)") Release Date: March 12, 2025 tip This version has been taken offline due to metadata loss issues in **shared-data clusters**. * **Problem**: When there are committed compaction transactions that are not yet been published during a shift of Leader FE node in a shared-data cluster, metadata loss may occur after the shift. * **Impact scope**: This problem only affects shared-data clusters. Shared-nothing clusters are unaffected. * **Temporary workaround**: When the Publish task is returned with an error, you can execute `SHOW PROC 'compactions'` to check if there are any partitions that have two compaction transactions with empty `FinishTime`. You can execute `ALTER TABLE DROP PARTITION FORCE` to drop the partitions to avoid Publish tasks getting hang. ##### New Features and Enhancements[​](#new-features-and-enhancements "Direct link to New Features and Enhancements") * Data lake analytics supports Deletion Vector in Delta Lake. * Supports secure views. By creating a secure view, you can prevent users without the SELECT privilege on the referenced base tables from querying the view (even if they have the SELECT privilege on the view). * Supports for Sketch HLL ([`ds_hll_count_distinct`](https://docs.starrocks.io/docs/sql-reference/sql-functions/aggregate-functions/ds_hll_count_distinct/)). Compared to `approx_count_distinct`, this function provides higher-precision approximate deduplication. * Shared-data clusters support automatic snapshot creation for cluster recovery. * Storage Volume in the shared-data clusters supports Azure Data Lake Storage Gen2. * Supports SSL authentication for connections to StarRocks via the MySQL protocol, ensuring that data transmitted between the client and the StarRocks cluster cannot be read by unauthorized users. ##### Bug Fixes[​](#bug-fixes-9 "Direct link to Bug Fixes") The following issues have been fixed: * An issue where OLAP views affected the materialized view processing logic. [#52989](https://github.com/StarRocks/starrocks/pull/52989) * Write transactions would fail if one replica was not found, regardless of how many replicas had successfully committed. (After the fix, the transaction succeeds as long as the majority replicas succeed. [#55212](https://github.com/StarRocks/starrocks/pull/55212) * Stream Load fails when a node with an Alive status of false was scheduled. [#55371](https://github.com/StarRocks/starrocks/pull/55371) * Files in cluster snapshots were mistakenly deleted. [#56338](https://github.com/StarRocks/starrocks/pull/56338) ##### Behavior Changes[​](#behavior-changes-5 "Direct link to Behavior Changes") * Graceful shutdown is now enabled by default (previously it was disabled). The default value of the related BE/CN parameter `loop_count_wait_fragments_finish` has been changed to `2`, meaning that the system will wait up to 20 seconds for running queries to complete. [#56002](https://github.com/StarRocks/starrocks/pull/56002) #### 3.4.0[​](#340 "Direct link to 3.4.0") Release date: January 24, 2025 ##### Data Lake Analytics[​](#data-lake-analytics "Direct link to Data Lake Analytics") * Optimized Iceberg V2 query performance and lowered memory usage by reducing repeated reads of delete-files. * Supports column mapping for Delta Lake tables, allowing queries against data after Delta Schema Evolution. For more information, see [Delta Lake catalog - Feature support](https://docs.starrocks.io/docs/data_source/catalog/deltalake_catalog/#feature-support). * Data Cache related improvements: * Introduces a Segmented LRU (SLRU) Cache eviction strategy, which significantly defends against cache pollution from occasional large queries, improves cache hit rate, and reduces fluctuations in query performance. In simulated test cases with large queries, SLRU-based query performance can be improved by 70% or even higher. For more information, see [Data Cache - Cache replacement policies](https://docs.starrocks.io/docs/data_source/data_cache/#cache-replacement-policies). * Unified the Data Cache instance used in both shared-data architecture and data lake query scenarios to simplify the configuration and improve resource utilization. For more information, see [Data Cache](https://docs.starrocks.io/docs/using_starrocks/caching/block_cache/). * Provides an adaptive I/O strategy optimization for Data Cache, which flexibly routes some query requests to remote storage based on the cache disk's load and performance, thereby enhancing overall access throughput. * Supports persistence of data in Data Cache in Data Lake query scenarios. The previously cached data can be reused after BE restarts to reduce query performance fluctuations. * Supports automatic collection of external table statistics through automatic ANALYZE tasks triggered by queries. It can provide more accurate NDV information compared to metadata files, thereby optimizing the query plan and improving query performance. For more information, see [Query-triggered collection](https://docs.starrocks.io/docs/using_starrocks/Cost_based_optimizer/#query-triggered-collection). * Provides Time Travel query capability for Iceberg, allowing data to be read from a specified BRANCH or TAG by specifying TIMESTAMP or VERSION. * Supports asynchronous delivery of query fragments for data lake queries. It avoids the restriction that FE must obtain all files to be queried before BE can execute a query, thus allowing FE to fetch query files and BE to execute queries in parallel, and reducing the overall latency of data lake queries involving a large number of files that are not in the cache. Meanwhile, it reduces the memory load on FE due to caching the file list and improves query stability. (Currently, the optimization for Hudi and Delta Lake is implemented, while the optimization for Iceberg is still under development.) ##### Performance Improvement and Query Optimization[​](#performance-improvement-and-query-optimization "Direct link to Performance Improvement and Query Optimization") * \[Experimental] Offers a preliminary Query Feedback feature for automatic optimization of slow queries. The system will collect the execution details of slow queries, automatically analyze its query plan for potential opportunities for optimization, and generate a tailored optimization guide for the query. If CBO generates the same bad plan for subsequent identical queries, the system will locally optimize this query plan based on the guide. For more information, see [Query Feedback](https://docs.starrocks.io/docs/using_starrocks/query_feedback/). * \[Experimental] Supports Python UDFs, offering more convenient function customization compared to Java UDFs. For more information, see [Python UDF](https://docs.starrocks.io/docs/sql-reference/sql-functions/Python_UDF/). * Enables the pushdown of multi-column OR predicates, allowing queries with multi-column OR conditions (for example, `a = xxx OR b = yyy`) to utilize certain column indexes, thus reducing data read volume and improving query performance. * Optimized TPC-DS query performance by roughly 20% under the TPC-DS 1TB Iceberg dataset. Optimization methods include table pruning and aggregated column pruning using primary and foreign keys, and aggregation pushdown. ##### Shared-data Enhancements[​](#shared-data-enhancements "Direct link to Shared-data Enhancements") * Supports Query Cache, aligning the shared-nothing architecture. * Supports synchronous materialized views, aligning the shared-nothing architecture. ##### Storage Engine[​](#storage-engine "Direct link to Storage Engine") * Unified all partitioning methods into the expression partitioning and supported multi-level partitioning, where each level can be any expression. For more information, see [Expression Partitioning](https://docs.starrocks.io/docs/table_design/data_distribution/expression_partitioning/). * \[Preview] Supports all native aggregate functions in Aggregate tables. By introducing a generic aggregate function state storage framework, all native aggregate functions supported by StarRocks can be used to define an Aggregate table. * Supports vector indexes, enabling fast approximate nearest neighbor searches (ANNS) of large-scale, high-dimensional vectors, which are commonly required in deep learning and machine learning scenarios. Currently, StarRocks supports two types of vector indexes: IVFPQ and HNSW. ##### Loading[​](#loading "Direct link to Loading") * INSERT OVERWRITE now supports a new semantic - Dynamic Overwrite. When this semantic is enabled, the ingested data will either create new partitions or overwrite existing partitions that correspond to the new data records. Partitions not involved will not be truncated or deleted. This semantic is especially useful when users want to recover data in specific partitions without specifying the partition names. For more information, see [Dynamic Overwrite](https://docs.starrocks.io/docs/loading/InsertInto/#dynamic-overwrite). * Optimized the data ingestion with INSERT from FILES to replace Broker Load as the preferred loading method: * FILES now supports listing files in remote storage, and providing basic statistics of the files. For more information, see [FILES - list\_files\_only](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files/#list_files_only). * INSERT now supports matching columns by name, which is especially useful when users load data from numerous columns with identical names. (The default behavior matches columns by their position.) For more information, see [Match column by name](https://docs.starrocks.io/docs/loading/InsertInto/#match-column-by-name). * INSERT supports specifying PROPERTIES, aligning with other loading methods. Users can specify `strict_mode`, `max_filter_ratio`, and `timeout` for INSERT operations to control and behavior and quality of the data ingestion. For more information, see [INSERT - PROPERTIES](https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/INSERT/#properties). * INSERT from FILES supports pushing down the target table schema check to the Scan stage of FILES to infer a more accurate source data schema. For more information, see see [Push down target table schema check](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files/#push-down-target-table-schema-check). * FILES supports unionizing files with different schema. The schema of Parquet and ORC files are unionized based on the column names, and that of CSV files are unionized based on the position (order) of the columns. When there are mismatched columns, users can choose to fill the columns with NULL or return an error by specifying the property `fill_mismatch_column_with`. For more information, see [Union files with different schema](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files/#union-files-with-different-schema). * FILES supports inferring the STRUCT type data from Parquet files. (In earlier versions, STRUCT data is inferred as STRING type.) For more information, see [Infer STRUCT type from Parquet](https://docs.starrocks.io/docs/sql-reference/sql-functions/table-functions/files/#infer-struct-type-from-parquet). * Supports merging multiple concurrent Stream Load requests into a single transaction and committing data in a batch, thus improving the throughput of real-time data ingestion. It is designed for high concurrency, small-batch (from KB to tens of MB) real-time loading scenarios. It can reduce the excessive data versions caused by frequent loading operations, resource consumption during Compaction, and IOPS and I/O latency brought by excessive small files. ##### Others[​](#others "Direct link to Others") * Optimized the graceful exit process of BE and CN by accurately displaying the status of BE or CN nodes during a graceful exit as `SHUTDOWN`. * Optimized log printing to avoid excessive disk space being occupied. * Shared-nothing clusters now support backing up and restoring more objects: logical view, external catalog metadata, and partitions created with expression partitioning and list partitioning strategies. * \[Preview] Supports CheckPoint on Follower FE to avoid excessive memory on Leader FE during CheckPoint, thereby improving the stability of Leader FE. ##### Behavior Changes[​](#behavior-changes-6 "Direct link to Behavior Changes") * Because the Data Cache instance used in both shared-data architecture and data lake query scenarios is now unified, there will be the following behavior changes after the upgrade to v3.4.0: * BE configuration item `datacache_disk_path` is now deprecated. The data will be cached under the directory `${storage_root_path}/datacache`. If you want to allocate a dedicated disk for data cache, you can manually point the directory to the directory mentioned above using a symlink. * Cached data in the shared-data cluster will be automatically migrated to `${storage_root_path}/datacache` and can be re-used after the upgrade. * The behavior changes of `datacache_disk_size`: * When `datacache_disk_size` is `0` (Default), the automatic adjustment of cache capacity is enabled (consistent with the behavior before the upgrade). * When `datacache_disk_size` is set to a value greater than `0`, the system will pick a larger value between `datacache_disk_size` and `starlet_star_cache_disk_size_percent` as the cache capacity. * From v3.4.0 onwards, `insert_timeout` applies to operations involved INSERT (for example, UPDATE, DELETE, CTAS, materialized view refresh, statistics collection, and PIPE), replacing `query_timeout`. * From v3.4.0 onwards, the default value of `mysql_server_version` is changed to `8.0.33`. ##### Downgrade Notes[​](#downgrade-notes "Direct link to Downgrade Notes") * Clusters can be downgraded from v3.4.0 only to v3.3.9 and later. --- ## Release 3.5 ### StarRocks version 3.5 warning **Upgrade Notes** * JDK 17 or later is required from StarRocks v3.5.0 onwards. * To upgrade a cluster from v3.4 or earlier, you must upgrade the version of JDK that StarRocks depends, and remove the options that are incompatible with JDK 17 in the configuration item `JAVA_OPTS` in the FE configuration file **fe.conf**, for example, options that involve CMS and GC. The default value of `JAVA_OPTS` in the v3.5 configuration file is recommended. * For clusters using external catalogs, you need to add `--add-opens=java.base/java.util=ALL-UNNAMED` to the `JAVA_OPTS` configuration item in the BE configuration file **be.conf**. * For clusters using Java UDFs, you need to add `--add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED` to the `JAVA_OPTS` configuration item in the BE configuration file **be.conf**. * In addition, as of v3.5.0, StarRocks no longer provides JVM configurations for specific JDK versions. All versions of JDK use `JAVA_OPTS`. * It is recommended to upgrade the cluster to v3.4.10 or later before upgrading it to v3.5. Otherwise, you must manually disable low cardinality optimization during the gray-scale upgrade by executing the following statement: ```sql SET GLOBAL cbo_enable_low_cardinality_optimize=false; ``` **Downgrade Notes** * After upgrading StarRocks to v3.5, DO NOT downgrade it directly to v3.4.0 ～ v3.4.5, otherwise it will cause metadata incompatibility. You must downgrade the cluster to v3.4.6 or later to prevent the issue. * After upgrading StarRocks to v3.5.2 or later, DO NOT downgrade it to v3.5.0 & v3.5.1, otherwise it will cause FE crash. #### 3.5.20[​](#3520 "Direct link to 3.5.20") Release date: July 23, 2026 ##### Behavior Changes[​](#behavior-changes "Direct link to Behavior Changes") * `CREATE DATABASE IF NOT EXISTS` on Iceberg REST catalogs now succeeds silently when the database already exists, instead of raising an error. [#75017](https://github.com/StarRocks/starrocks/pull/75017) * Iceberg REST catalogs with vended credentials now cache `Table` objects and keep their credentials refreshed on access, instead of bypassing the cache and re-fetching from the REST catalog/Lake Formation on every `getTable()` call, which could trigger AWS `Rate exceeded` errors. [#75431](https://github.com/StarRocks/starrocks/pull/75431) * GIN inverted-index-accelerated `NOT MATCH` predicates no longer return rows with a `NULL` value, matching SQL three-valued-logic semantics. [#75578](https://github.com/StarRocks/starrocks/pull/75578) ##### Improvements[​](#improvements "Direct link to Improvements") * Added the FE metric `txn_max_committed_pending_publish_ms`, a per-database gauge reporting the longest time a committed transaction has been pending publish, to help diagnose stuck or lagging version publishing. [#75025](https://github.com/StarRocks/starrocks/pull/75025) * Enforced the query memory limit when a column is upgraded (widened) during window-function aggregation in `Analytor`, instead of letting it grow unbounded. [#75821](https://github.com/StarRocks/starrocks/pull/75821) * Removed useless per-rowid seeks in the array-column offsets-only read path used by `array_length()`/`cardinality()`. [#75861](https://github.com/StarRocks/starrocks/pull/75861) ##### Bug fixes[​](#bug-fixes "Direct link to Bug fixes") The following issues have been fixed: * Several wrong-result issues: `EliminateSortColumnWithEqualityPredicateRule` dropping the global `LIMIT` under concurrency; `SplitJoinORToUnionRule` producing duplicate rows for a null-safe-equal (`<=>`) `JOIN ON p1 OR p2`; JIT codegen truncating `LARGEINT` literals `>= 2^64` to 64 bits; `array_map`/`transform` silently dropping `NULL` rows when all non-null input arrays were empty; nested dictionary expressions rebuilt inconsistently across exchange fragments causing dict-decode failures; and a `LIKE` pattern with the `_` wildcard returning wrong rows on a GIN inverted index. [#74983](https://github.com/StarRocks/starrocks/pull/74983) [#75038](https://github.com/StarRocks/starrocks/pull/75038) [#75137](https://github.com/StarRocks/starrocks/pull/75137) [#75141](https://github.com/StarRocks/starrocks/pull/75141) [#75246](https://github.com/StarRocks/starrocks/pull/75246) [#75551](https://github.com/StarRocks/starrocks/pull/75551) * Join-reorder column pruning could drop a column still referenced by a predicate, causing a `missing statistic of col` planning error, and `JoinTuningGuide` could lose `predicateCommonOperators` when rebuilding a join, failing plan validation. [#74791](https://github.com/StarRocks/starrocks/pull/74791) [#75773](https://github.com/StarRocks/starrocks/pull/75773) * Sync materialized view/rollup rewrite could lose a rollup column when a query aggregated the same base column twice (e.g. `min(c)` and `max(c)`), and async materialized view rewrite could serve stale results after an Iceberg base table's `rollback_to_snapshot`. [#75528](https://github.com/StarRocks/starrocks/pull/75528) [#75924](https://github.com/StarRocks/starrocks/pull/75924) * `PARTITION-TOP-N` could rewrite its partition-by column to a dictionary slot that no longer existed, failing with a `slot_id not found` error. [#75956](https://github.com/StarRocks/starrocks/pull/75956) * An NPE collecting view tables when a `SECURITY INVOKER` view's stored definition contains a CTE. [#74813](https://github.com/StarRocks/starrocks/pull/74813) * Three FE metadata-lock correctness races around `DROP PERSISTENT INDEX`, `RestoreJob` post-restore handling, and related unlocked paths. [#74968](https://github.com/StarRocks/starrocks/pull/74968) * A race between FE EOS-cancel and BE stage-2 deploy could mark a fully successful query as canceled. [#75009](https://github.com/StarRocks/starrocks/pull/75009) * `ApplyTuningGuideRule` could throw `UnsupportedOperationException` when an earlier rewrite produced an `OptExpression` with an immutable input list. [#70785](https://github.com/StarRocks/starrocks/pull/70785) * BE/CN crashes: a null `driver_executor` when a cancel RPC arrives before pipeline start; a use-after-free in the spill partition-sort-sink cancel path; a heap-use-after-free in `OrderedPartitionExchanger` for a skew-hinted window function at DOP>1; an `NLJoin` crash from a build-side column nullability mismatch; a `StructColumn` field-count mismatch in `UNNEST` output; a crash loop reading a flat-JSON column that changed from `NOT NULL` to nullable during compaction; an uncaught memory-allocation exception in `NLJoinProbeOperator`; a crash in primary-key auto-increment partial-update apply; and a crash rewriting predicates inside an `array_map` lambda during scan-predicate pushdown. [#75030](https://github.com/StarRocks/starrocks/pull/75030) [#75140](https://github.com/StarRocks/starrocks/pull/75140) [#75279](https://github.com/StarRocks/starrocks/pull/75279) [#75343](https://github.com/StarRocks/starrocks/pull/75343) [#75445](https://github.com/StarRocks/starrocks/pull/75445) [#75680](https://github.com/StarRocks/starrocks/pull/75680) [#75788](https://github.com/StarRocks/starrocks/pull/75788) [#76119](https://github.com/StarRocks/starrocks/pull/76119) [#76380](https://github.com/StarRocks/starrocks/pull/76380) * `histogram()` crashed (or silently mis-bucketed values) on a non-positive `bucket_num` instead of raising a clear error, and `bar()` could grow an unbounded string for a negative or huge `width` argument, exhausting BE memory. [#75041](https://github.com/StarRocks/starrocks/pull/75041) [#75143](https://github.com/StarRocks/starrocks/pull/75143) * A query using `unnest` over array columns could exceed `query_mem_limit` and get the BE OOM-killed instead of failing just that query. [#75179](https://github.com/StarRocks/starrocks/pull/75179) * A second-order SQL injection in the `information_schema.task_runs` `TASK_NAME`/`QUERY_ID` predicate lookup. [#75520](https://github.com/StarRocks/starrocks/pull/75520) * `SHOW CREATE ROUTINE LOAD` could emit a spurious leading comma before the first load-desc clause, and an unescaped `jsonpaths` value, producing non-runnable DDL. [#75522](https://github.com/StarRocks/starrocks/pull/75522) [#75755](https://github.com/StarRocks/starrocks/pull/75755) * Shared-data (lake) `SHOW PARTITIONS` and `information_schema.partitions_meta` reported every physical partition's bucket count as the table-level default instead of its own bucket count. [#75734](https://github.com/StarRocks/starrocks/pull/75734) * Several dependency CVEs by upgrading `jackson-databind` and Netty. [#75373](https://github.com/StarRocks/starrocks/pull/75373) [#76555](https://github.com/StarRocks/starrocks/pull/76555) * Batched `TabletInvertedIndex` write-lock acquisition in `markTabletsForceDelete`, reducing lock churn when force-deleting many tablets at once. [#75616](https://github.com/StarRocks/starrocks/pull/75616) * Batched tablet inverted-index writes in the insert-overwrite path. [#75923](https://github.com/StarRocks/starrocks/pull/75923) * Skipped an unnecessary remote `clear_parent_path` call when a load spill never used remote storage. [#76224](https://github.com/StarRocks/starrocks/pull/76224) * A null-padding size mismatch for missing columns in `ParquetScanner` so padded rows match the actual per-batch chunk size instead of the whole Parquet/Arrow batch size. [#75981](https://github.com/StarRocks/starrocks/pull/75981) * Vulnerable, stale transitive dependencies (old BouncyCastle, OkHttp 2.x, Tomcat, and others) that previously shipped alongside their fixed counterparts. [#76097](https://github.com/StarRocks/starrocks/pull/76097) #### 3.5.19[​](#3519 "Direct link to 3.5.19") Release date: June 26, 2026 ##### Behavior Changes[​](#behavior-changes-1 "Direct link to Behavior Changes") * `parse_json` now respects `ALLOW_THROW_EXCEPTION`: malformed JSON fails the query instead of silently producing `NULL`, mirroring the earlier `get_json_string` change. [#74976](https://github.com/StarRocks/starrocks/pull/74976) * `FILES()` and Broker Load now honor the Parquet `isAdjustedToUTC=false` flag for `INT64` timestamps, so such timestamps are no longer shifted by the session time zone. [#73674](https://github.com/StarRocks/starrocks/pull/73674) * `SHOW FUNCTIONS` now surfaces the `isolation` property (`shared` or `isolated`) for Java UDFs and UDAFs. [#75255](https://github.com/StarRocks/starrocks/pull/75255) * The non-reserved keywords `FLOOR` and `CEIL` are now allowed as column names. [#75241](https://github.com/StarRocks/starrocks/pull/75241) ##### Improvements[​](#improvements-1 "Direct link to Improvements") * Supports loading Arrow `LARGE_LIST` and `FIXED_SIZE_LIST` columns into `JSON` columns via `FILES()` and Broker Load. [#73714](https://github.com/StarRocks/starrocks/pull/73714) [#73718](https://github.com/StarRocks/starrocks/pull/73718) * Added an opt-in `statistics_large_string_column_merge_threshold` to isolate wide `CHAR`/`VARCHAR` columns into dedicated statistics collection. [#73258](https://github.com/StarRocks/starrocks/pull/73258) * Optimized `base64_to_bitmap` for constant inputs and hardened it against invalid base64-encoded bitmaps. [#74684](https://github.com/StarRocks/starrocks/pull/74684) * Added new metrics for lake vacuum batch size and retry counts, and gauges for `CatalogRecycleBin` size. [#74112](https://github.com/StarRocks/starrocks/pull/74112) [#74440](https://github.com/StarRocks/starrocks/pull/74440) * Supports auditing a statement twice. [#73896](https://github.com/StarRocks/starrocks/pull/73896) ##### Bug fixes[​](#bug-fixes-1 "Direct link to Bug fixes") The following issues have been fixed: * Several wrong-result and planning issues: low-cardinality dictionary translation for expressions where `f(NULL)` is not `NULL`; a `Multiple entries with same key` error from common-subexpression elimination of commutative `AND`/`OR`; an `AGGREGATE has mismatch types` error; a compound predicate with an always-false nested branch under `UNION` returning no rows; and an off-by-one that dropped a row in `RANK` TopN at a chunk boundary. [#69376](https://github.com/StarRocks/starrocks/pull/69376) [#72823](https://github.com/StarRocks/starrocks/pull/72823) [#74159](https://github.com/StarRocks/starrocks/pull/74159) [#74218](https://github.com/StarRocks/starrocks/pull/74218) [#75045](https://github.com/StarRocks/starrocks/pull/75045) * Materialized view rewrite issues that could return incorrect results: aggregate MVs with a `HAVING` clause rewriting queries without (or with weaker) `HAVING`, and `avg(DISTINCT x)` being rewritten through a `sum`/`count` MV. [#73610](https://github.com/StarRocks/starrocks/pull/73610) [#75071](https://github.com/StarRocks/starrocks/pull/75071) * Incorrect window-function results when `enable_push_down_pre_agg_with_rank` split a window count into local pre-aggregation and global analytic merge, and an empty window operator generated after pushing down distinct aggregation. [#74453](https://github.com/StarRocks/starrocks/pull/74453) [#74810](https://github.com/StarRocks/starrocks/pull/74810) * Partition TopN losing a child operator's output column, and silently swallowing sort or pre-aggregation errors and returning wrong or partial results. [#72848](https://github.com/StarRocks/starrocks/pull/72848) [#74693](https://github.com/StarRocks/starrocks/pull/74693) * Iceberg equality-delete rows with `NULL` identity-column values were not applied. [#67321](https://github.com/StarRocks/starrocks/pull/67321) * A spurious strict-mode cast overflow error raised from undefined data in `NULL` rows. [#74903](https://github.com/StarRocks/starrocks/pull/74903) * Decimal scale could be lost when a column is entirely `NULL`. [#73789](https://github.com/StarRocks/starrocks/pull/73789) * BE crashes in `to_base64` (stack overflow), JSON load of nested types via partial append, local partition TopN with a non-nullable aggregate result, partitioned join (out-of-bounds from inaccurate memory accounting), runtime profile serialization (counter min/max race), JIT compilation failure (use-after-free of `LLVMContext`), invalid JIT IR for `CASE WHEN` with mixed float/integer types, and partial column updates under schema drift. [#70623](https://github.com/StarRocks/starrocks/pull/70623) [#73715](https://github.com/StarRocks/starrocks/pull/73715) [#69752](https://github.com/StarRocks/starrocks/pull/69752) [#74315](https://github.com/StarRocks/starrocks/pull/74315) [#72904](https://github.com/StarRocks/starrocks/pull/72904) [#74396](https://github.com/StarRocks/starrocks/pull/74396) [#74382](https://github.com/StarRocks/starrocks/pull/74382) [#74005](https://github.com/StarRocks/starrocks/pull/74005) * An out-of-bounds read and potential oversized allocation in `split`, `split_part`, and `str_to_map` when the input ends with a truncated UTF-8 byte. [#75068](https://github.com/StarRocks/starrocks/pull/75068) * A memory leak from the UDAF context cache and inflated query-pool memory accounting in `OlapTableSink`. [#74025](https://github.com/StarRocks/starrocks/pull/74025) [#73807](https://github.com/StarRocks/starrocks/pull/73807) * Unexpected backend process restarts. [#74424](https://github.com/StarRocks/starrocks/pull/74424) * Materialized view issues: a slot-nullability crash for MVs defined with `FULL OUTER JOIN` under late materialization, an NPE refreshing nested MVs, a duplicated warehouse property in `SHOW CREATE MATERIALIZED VIEW`, and a vector ANN query polluting a shared table schema and breaking unrelated statements. [#72621](https://github.com/StarRocks/starrocks/pull/72621) [#73644](https://github.com/StarRocks/starrocks/pull/73644) [#69418](https://github.com/StarRocks/starrocks/pull/69418) [#74785](https://github.com/StarRocks/starrocks/pull/74785) * Querying Paimon tables whose `DATE` partition column contains `NULL` values. [#73950](https://github.com/StarRocks/starrocks/pull/73950) * Reading Hudi MOR tables with `char`/`varchar` columns when `hudi_mor_force_jni_reader` is enabled. [#58521](https://github.com/StarRocks/starrocks/pull/58521) * Nested `INT96` timestamps (inside `ARRAY`, `MAP`, or `STRUCT`) were shifted by the session time zone during `FILES()`/Broker Load. [#74868](https://github.com/StarRocks/starrocks/pull/74868) * Incorrect bytes-read statistics in the audit log for connector scans, and incremental connector scan ranges being assigned to driver sequences absent from the deployed fragment, which could drop part of the scan. [#73799](https://github.com/StarRocks/starrocks/pull/73799) [#74674](https://github.com/StarRocks/starrocks/pull/74674) * Meta scan could fail after schema changes such as `ADD COLUMN`, which could fail background statistics collection. [#72901](https://github.com/StarRocks/starrocks/pull/72901) * Slow broker RPCs held the per-job Routine Load write lock and blocked admin RPCs and `SHOW ROUTINE LOAD`. [#73591](https://github.com/StarRocks/starrocks/pull/73591) * `ALTER ROUTINE LOAD` persisted an invalid statement for reserved-keyword table names, which could drop the load clause on FE restart. [#74188](https://github.com/StarRocks/starrocks/pull/74188) * `GRANT`/`REVOKE` on the `public` role did not invalidate cached merged privileges, leaving stale authorization. [#73717](https://github.com/StarRocks/starrocks/pull/73717) * A race allowing concurrent operations to observe torn state during table and materialized-view `RENAME` and `SWAP`, and a data race on `MaterializedIndexMeta` schema-update tracking. [#74100](https://github.com/StarRocks/starrocks/pull/74100) [#74412](https://github.com/StarRocks/starrocks/pull/74412) * Database-level UDFs were missing on FE followers after `RESTORE ... AS `. [#74313](https://github.com/StarRocks/starrocks/pull/74313) * Queries could become unkillable when a coordinator held its lock during external resource cleanup. [#72830](https://github.com/StarRocks/starrocks/pull/72830) * A permanent version hole on non-primary-key replicas could cause queries to fail with `version not found`. [#74408](https://github.com/StarRocks/starrocks/pull/74408) * Force-killed `SUBMIT TASK` runs disappeared from task-run history (and session-prefixed task-run timeouts are now honored), and an illegal running-to-running edit log could wedge subsequent task runs. [#74146](https://github.com/StarRocks/starrocks/pull/74146) [#73882](https://github.com/StarRocks/starrocks/pull/73882) * `ADMIN SHOW REPLICA STATUS` emitted a misaligned row for missing replicas, which could hang or disconnect the client. [#74393](https://github.com/StarRocks/starrocks/pull/74393) * `CatalogRecycleBin` halted all deletions in shared-data mode when cluster snapshots kept failing, causing unbounded FE memory growth. [#74379](https://github.com/StarRocks/starrocks/pull/74379) * An NPE in statistics calculation when a partition is dropped concurrently, and zero row counts written into partition statistics after `INSERT OVERWRITE` corrupting cardinality estimates. [#73711](https://github.com/StarRocks/starrocks/pull/73711) [#74801](https://github.com/StarRocks/starrocks/pull/74801) * Colocate tablets with all replicas on dead BEs were reported as healthy when `tablet_sched_disable_colocate_balance` is enabled. [#73550](https://github.com/StarRocks/starrocks/pull/73550) * An `IllegalMonitorStateException` from a lock mismatch in the tablet checker could abort a checker round. [#74596](https://github.com/StarRocks/starrocks/pull/74596) * Reduced lock contention by narrowing several full-database `WRITE` locks to table-scoped locks in shared-nothing mode, and skipped unnecessary locking in `TabletInvertedIndex.deleteTablets` for empty input. [#74523](https://github.com/StarRocks/starrocks/pull/74523) [#73955](https://github.com/StarRocks/starrocks/pull/73955) * A race between transaction begin and autovacuum could delete a still-needed transaction log and permanently wedge publishing in shared-data mode; decorrelated jitter was also added to lake vacuum retry backoff. [#74906](https://github.com/StarRocks/starrocks/pull/74906) [#74108](https://github.com/StarRocks/starrocks/pull/74108) * Added `lake_vacuum_enable_task_timeout` to abort BE vacuum tasks once the FE caller's timeout elapses. [#74694](https://github.com/StarRocks/starrocks/pull/74694) * A crash from a missing null check when reading `gtid` during a data-rewriting schema change. [#74855](https://github.com/StarRocks/starrocks/pull/74855) * A thread-name race produced noisy warnings during BE data directory load. [#73862](https://github.com/StarRocks/starrocks/pull/73862) * An `IllegalStateException` during parallel profile collection for external-table queries when `enable_profile` is on. [#74746](https://github.com/StarRocks/starrocks/pull/74746) * `ALTER TABLE ... MODIFY COLUMN ... AFTER` a nonexistent column raised an internal NPE instead of a clean error. [#75073](https://github.com/StarRocks/starrocks/pull/75073) * Query hangs and operator stalls from missing notifications when a distinct aggregate source finishes and on missed operator state transitions, and sort merge provider errors now propagate to the fragment context. [#74055](https://github.com/StarRocks/starrocks/pull/74055) [#74557](https://github.com/StarRocks/starrocks/pull/74557) [#73337](https://github.com/StarRocks/starrocks/pull/73337) * FE dropped the connection (client `ERROR 2013`) instead of returning a proper error when connecting to a database the user lacks privileges on. [#70072](https://github.com/StarRocks/starrocks/pull/70072) * Prepared statements with a microsecond `DATETIME` parameter failed with `Invalid date type: DECIMAL(6,0)`. [#74141](https://github.com/StarRocks/starrocks/pull/74141) * The audit log recorded `ReturnRows=0` for `SELECT ... INTO OUTFILE`. [#74467](https://github.com/StarRocks/starrocks/pull/74467) * `DATETIME_PRECISION` was always `NULL` in `information_schema.COLUMNS`, which broke type mapping for some MySQL-protocol clients. [#74623](https://github.com/StarRocks/starrocks/pull/74623) * CTAS did not preserve an explicitly declared `VARCHAR(N)` length in the new table's schema. [#73498](https://github.com/StarRocks/starrocks/pull/73498) * `enable_statistic_collect_on_first_load` now allows a table-level setting to override the global configuration. [#74794](https://github.com/StarRocks/starrocks/pull/74794) * A typo in the `azure_adls2_oauth2_client_endpoint` configuration field name. [#74581](https://github.com/StarRocks/starrocks/pull/74581) * Assertion name lookup in assert-num-rows. [#74178](https://github.com/StarRocks/starrocks/pull/74178) * Several dependency CVEs by upgrading libthrift, Tomcat, and Netty, and excluding a vulnerable jline transitive dependency. [#73243](https://github.com/StarRocks/starrocks/pull/73243) [#73797](https://github.com/StarRocks/starrocks/pull/73797) [#74668](https://github.com/StarRocks/starrocks/pull/74668) [#75066](https://github.com/StarRocks/starrocks/pull/75066) #### 3.5.18[​](#3518 "Direct link to 3.5.18") Release date: June 5, 2026 ##### Behavior Changes[​](#behavior-changes-2 "Direct link to Behavior Changes") * `SHOW` statements are now allowed inside explicit transactions. [#72954](https://github.com/StarRocks/starrocks/pull/72954) * `get_json_string` now respects `ALLOW_THROW_EXCEPTION` when handling JSON parsing errors. [#73199](https://github.com/StarRocks/starrocks/pull/73199) * `IGNORE NULLS` is now preserved in view definitions when the window function argument is an expression. [#69971](https://github.com/StarRocks/starrocks/pull/69971) * Ranger row filter and masking policies are now correctly applied to Hive views and to base tables expanded from Hive view definitions. [#73265](https://github.com/StarRocks/starrocks/pull/73265) * Hive partition statistics are no longer automatically refreshed per partition. Existing cached stats are preserved while a table-level asynchronous refresh updates the cache in batches. [#73563](https://github.com/StarRocks/starrocks/pull/73563) ##### Improvements[​](#improvements-2 "Direct link to Improvements") * Supports caching Java UDAF class-level initialization so shared UDAFs can reuse loaded classes and generated stubs across aggregator and window-function instances. [#72038](https://github.com/StarRocks/starrocks/pull/72038) * Supports Paimon time types and improves Paimon materialized view handling. [#58292](https://github.com/StarRocks/starrocks/pull/58292) * Added an Avro schema cache for shadowed `PartitionData` during partition load. [#72215](https://github.com/StarRocks/starrocks/pull/72215) * Added a configurable FE write timeout `mysql_send_packet_timeout_ms` for the MySQL result send path to prevent indefinitely blocked result sending to slow clients. [#73646](https://github.com/StarRocks/starrocks/pull/73646) * Optimized `CatalogRecycleBin` adjusted recycle timestamp lookup. [#72128](https://github.com/StarRocks/starrocks/pull/72128) * Reduced metadata and lock overhead in load balancing, compaction scheduling, consistency checks, and StarMgr metadata synchronization paths. [#73555](https://github.com/StarRocks/starrocks/pull/73555) [#72218](https://github.com/StarRocks/starrocks/pull/72218) [#72178](https://github.com/StarRocks/starrocks/pull/72178) [#72108](https://github.com/StarRocks/starrocks/pull/72108) * Improved diagnostics for filesystem copy failures and Parquet broker load errors by surfacing the underlying cause and file/column/row context. [#73414](https://github.com/StarRocks/starrocks/pull/73414) [#73236](https://github.com/StarRocks/starrocks/pull/73236) * Reduced external catalog and information schema metadata overhead by deferring JDBC REMARKS fetching, avoiding redundant Paimon snapshot lookups, and pushing down `table_name` predicates for `information_schema.tables_config`. [#73488](https://github.com/StarRocks/starrocks/pull/73488) [#72892](https://github.com/StarRocks/starrocks/pull/72892) [#73210](https://github.com/StarRocks/starrocks/pull/73210) * Simplified the scalar-function merge implementation by using `merge()` directly. [#69575](https://github.com/StarRocks/starrocks/pull/69575) ##### Bug fixes[​](#bug-fixes-2 "Direct link to Bug fixes") The following issues have been fixed: * Empty `ALTER TABLE` statements could be parsed as OPTIMIZE clauses, and replaying malformed OPTIMIZE jobs could clear a table's default distribution. [#73352](https://github.com/StarRocks/starrocks/pull/73352) * Decimal-valued unit counters in runtime profiles could cause query progress parsing failures and noisy FE warnings. [#73683](https://github.com/StarRocks/starrocks/pull/73683) * Concurrent `SegmentFlushTask` race in `DeltaWriter::commit()` and loss of `merge_condition` during normal rowset commit. [#73371](https://github.com/StarRocks/starrocks/pull/73371) [#72542](https://github.com/StarRocks/starrocks/pull/72542) * Crashes, hangs, or unsafe cleanup in `SinkBuffer` graceful exit, `PipelineTimerTask`, runtime filter workers, spillable hash join probe, `information_schema.warehouse_queries`, lake vacuum, HTTP connection unregister paths, and query queue timeout handling. [#73202](https://github.com/StarRocks/starrocks/pull/73202) [#73082](https://github.com/StarRocks/starrocks/pull/73082) [#72058](https://github.com/StarRocks/starrocks/pull/72058) [#72626](https://github.com/StarRocks/starrocks/pull/72626) [#72397](https://github.com/StarRocks/starrocks/pull/72397) [#72019](https://github.com/StarRocks/starrocks/pull/72019) [#73088](https://github.com/StarRocks/starrocks/pull/73088) [#72006](https://github.com/StarRocks/starrocks/pull/72006) [#65802](https://github.com/StarRocks/starrocks/pull/65802) * Materialized view issues involving JDBC SQL Server tables, lost index properties, cached plan context memory leaks, Paimon tables, and incorrect shuffle distribution after MV rewrite. [#72962](https://github.com/StarRocks/starrocks/pull/72962) [#69187](https://github.com/StarRocks/starrocks/pull/69187) [#72300](https://github.com/StarRocks/starrocks/pull/72300) [#58292](https://github.com/StarRocks/starrocks/pull/58292) [#71075](https://github.com/StarRocks/starrocks/pull/71075) * Query planning and rewrite issues in Spark connector external scans, `INSERT OVERWRITE` re-planning, aggregation spill with small LIMIT, and generated columns produced by `UNNEST`. [#73225](https://github.com/StarRocks/starrocks/pull/73225) [#72832](https://github.com/StarRocks/starrocks/pull/72832) [#72705](https://github.com/StarRocks/starrocks/pull/72705) [#72027](https://github.com/StarRocks/starrocks/pull/72027) * Paimon Primary Key columns could be incorrectly marked as non-nullable when querying external catalogs. [#71660](https://github.com/StarRocks/starrocks/pull/71660) * Primary Key and tablet metadata issues including partial tablet schema short-key mismatch, rowset metadata cache warmup deadlock, disk data cache expansion failure, Azure filesystem client cache issues in Starlet, and colocate-heavy cluster-balance performance issues in StarOS. [#70586](https://github.com/StarRocks/starrocks/pull/70586) [#71459](https://github.com/StarRocks/starrocks/pull/71459) [#58206](https://github.com/StarRocks/starrocks/pull/58206) [#73145](https://github.com/StarRocks/starrocks/pull/73145) [#72391](https://github.com/StarRocks/starrocks/pull/72391) * Locker rollback and unlock-order issues during partial intensive-lock acquisition. [#72789](https://github.com/StarRocks/starrocks/pull/72789) [#72423](https://github.com/StarRocks/starrocks/pull/72423) * Dependency CVEs and broker dependency regressions. [#72905](https://github.com/StarRocks/starrocks/pull/72905) [#72797](https://github.com/StarRocks/starrocks/pull/72797) [#72184](https://github.com/StarRocks/starrocks/pull/72184) [#72191](https://github.com/StarRocks/starrocks/pull/72191) * JNI local-reference leaks in JDBC scanner initialization. [#72913](https://github.com/StarRocks/starrocks/pull/72913) * Arrow dictionary values in Parquet scanner and Apache Parquet namespace ambiguity during scanner builds. [#71855](https://github.com/StarRocks/starrocks/pull/71855) [#72284](https://github.com/StarRocks/starrocks/pull/72284) * NPE in Iceberg `getPartitionLastUpdatedTime` when the snapshot is expired. [#68925](https://github.com/StarRocks/starrocks/pull/68925) #### 3.5.17[​](#3517 "Direct link to 3.5.17") Release date: May 13, 2026 ##### Behavior Changes[​](#behavior-changes-3 "Direct link to Behavior Changes") * `SHOW CREATE TABLE` and `DESC` now show Primary Keys for Paimon tables. [#70535](https://github.com/StarRocks/starrocks/pull/70535) * Disallowed INSERT into insert-only ACID Hive tables in Hive catalogs. [#71460](https://github.com/StarRocks/starrocks/pull/71460) * `START_TIME` and `END_TIME` in Profile are now displayed using the session time zone. [#71429](https://github.com/StarRocks/starrocks/pull/71429) ##### Improvements[​](#improvements-3 "Direct link to Improvements") * Supports `csv.enclose` and `csv.escape` in `INSERT INTO FILES` CSV export. [#71589](https://github.com/StarRocks/starrocks/pull/71589) * Added query relation information (directly queried tables and viewa) to audit logs. [#71596](https://github.com/StarRocks/starrocks/pull/71596) * Made the FE configuration `star_mgr_meta_sync_interval_sec` runtime mutable. [#71675](https://github.com/StarRocks/starrocks/pull/71675) * Reduced metadata and lock overhead in table metadata and row-count paths. [#72053](https://github.com/StarRocks/starrocks/pull/72053) [#72042](https://github.com/StarRocks/starrocks/pull/72042) [#71672](https://github.com/StarRocks/starrocks/pull/71672) * Improved build and dependency hygiene by merging the broker builder into the FE build and removing WildFly OpenSSL. [#71823](https://github.com/StarRocks/starrocks/pull/71823) [#71908](https://github.com/StarRocks/starrocks/pull/71908) ##### Bug fixes[​](#bug-fixes-3 "Direct link to Bug fixes") The following issues have been fixed: * Wrong results for local-shuffle aggregate queries with OFFSET. [#71997](https://github.com/StarRocks/starrocks/pull/71997) * Incorrect Join output properties after Exchange shuffle columns are pruned. [#72003](https://github.com/StarRocks/starrocks/pull/72003) * Several dependency CVE issues. [#71762](https://github.com/StarRocks/starrocks/pull/71762) [#71914](https://github.com/StarRocks/starrocks/pull/71914) * Oracle JDBC NLS format handling issue. [#71412](https://github.com/StarRocks/starrocks/pull/71412) * Missing Iceberg column statistics in manifest data file cache. [#71913](https://github.com/StarRocks/starrocks/pull/71913) * Missing Hive partition directory before INSERT OVERWRITE commit. [#71810](https://github.com/StarRocks/starrocks/pull/71810) * Aggregate-join-pushdown materialized view rewrite and min/max optimization issues on Iceberg base tables. [#71856](https://github.com/StarRocks/starrocks/pull/71856) [#71863](https://github.com/StarRocks/starrocks/pull/71863) * Race conditions in `ConnectorSinkPassthroughExchanger` and `LoadChannel::get_load_replica_status`. [#71848](https://github.com/StarRocks/starrocks/pull/71848) [#71843](https://github.com/StarRocks/starrocks/pull/71843) * Credential redaction issue in INSERT FILES operations. [#71245](https://github.com/StarRocks/starrocks/pull/71245) * Incorrect `reverse(DecimalV3)` results. [#71834](https://github.com/StarRocks/starrocks/pull/71834) * Missing JNI exception handling checks in Java UDF code. [#71734](https://github.com/StarRocks/starrocks/pull/71734) * Incorrect short-circuit checks in `EventScheduler`. [#71740](https://github.com/StarRocks/starrocks/pull/71740) * Incorrect Arrow Flight column name for empty result sets. [#71534](https://github.com/StarRocks/starrocks/pull/71534) * Batch publish deadlock caused by partition version gaps. [#71483](https://github.com/StarRocks/starrocks/pull/71483) * Repeated Apply attachments in scalar-subquery plans. [#71155](https://github.com/StarRocks/starrocks/pull/71155) #### 3.5.16[​](#3516 "Direct link to 3.5.16") Release date: April 20, 2026 ##### Improvements[​](#improvements-4 "Direct link to Improvements") * Added clearer warning logs for swallowed exceptions in `WarehouseManager`. [#71215](https://github.com/StarRocks/starrocks/pull/71215) * Supports pausing Routine Load jobs on non-retryable errors. [#71161](https://github.com/StarRocks/starrocks/pull/71161) * Added thread names to the utility that prints all thread stacks. [#69366](https://github.com/StarRocks/starrocks/pull/69366) * Supports constant folding for `regexp_replace` in FE. [#70804](https://github.com/StarRocks/starrocks/pull/70804) * Supports showing column comments for PostgreSQL external tables, and added an option to let `information_schema.tables` fetch full metadata such as comments from external catalogs. [#70520](https://github.com/StarRocks/starrocks/pull/70520) [#70197](https://github.com/StarRocks/starrocks/pull/70197) * Added automatic query plan dumping on query exceptions. [#70387](https://github.com/StarRocks/starrocks/pull/70387) * Improved cloud-native tablet metadata fetch and repair efficiency. [#70492](https://github.com/StarRocks/starrocks/pull/70492) [#70386](https://github.com/StarRocks/starrocks/pull/70386) * Added batch tablet deletion in FE to reduce write lock contention. [#70052](https://github.com/StarRocks/starrocks/pull/70052) * Added logs for update compaction suspension, and added Iceberg metadata-table and time-travel query metrics. [#70538](https://github.com/StarRocks/starrocks/pull/70538) [#70825](https://github.com/StarRocks/starrocks/pull/70825) [#70788](https://github.com/StarRocks/starrocks/pull/70788) ##### Bug fixes[​](#bug-fixes-4 "Direct link to Bug fixes") The following issues have been fixed: * `be_tablets.DATA_SIZE` reports rowset column data bytes inaccurately. [#70735](https://github.com/StarRocks/starrocks/pull/70735) * An outdated Maven repository for broker builds. [#71533](https://github.com/StarRocks/starrocks/pull/71533) * Incorrect thread-pool resizing when updating `parallel_clone_task_per_path`. [#71484](https://github.com/StarRocks/starrocks/pull/71484) * Several use-after-free issues. [#71513](https://github.com/StarRocks/starrocks/pull/71513) [#71276](https://github.com/StarRocks/starrocks/pull/71276) [#71083](https://github.com/StarRocks/starrocks/pull/71083) [#62917](https://github.com/StarRocks/starrocks/pull/62917) [#69926](https://github.com/StarRocks/starrocks/pull/69926) [#69968](https://github.com/StarRocks/starrocks/pull/69968) * Resource group user classifier validation is not aligned with `CREATE USER`. [#71470](https://github.com/StarRocks/starrocks/pull/71470) * “no queryable replica” issues on follower FEs by syncing StarMgr journal replay. [#71263](https://github.com/StarRocks/starrocks/pull/71263) * Multiple dependency CVEs. [#71256](https://github.com/StarRocks/starrocks/pull/71256) [#71017](https://github.com/StarRocks/starrocks/pull/71017) [#70862](https://github.com/StarRocks/starrocks/pull/70862) * `VARCHAR` length is not preserved after reduce-cast with global variables. [#70269](https://github.com/StarRocks/starrocks/pull/70269) * Special-character escaping in equality predicates on `information_schema.tables`. [#71273](https://github.com/StarRocks/starrocks/pull/71273) * `UpdateTabletSchemaTask` signature collisions across alter jobs. [#71242](https://github.com/StarRocks/starrocks/pull/71242) * Issue with shared-object mutation in `PushDownAggregateRewriter` for `CASE WHEN` and `IF` expressions. [#71309](https://github.com/StarRocks/starrocks/pull/71309) * Stopped inactive materialized view schedulers correctly and added missing leader checks in TaskManager scheduler callbacks. [#71265](https://github.com/StarRocks/starrocks/pull/71265) [#71156](https://github.com/StarRocks/starrocks/pull/71156) * `NaN` row-count estimation for MCV-only histograms. [#71241](https://github.com/StarRocks/starrocks/pull/71241) * Packaging issues caused by a missing `s3-transfer-manager` dependency in the AWS SDK. [#71230](https://github.com/StarRocks/starrocks/pull/71230) * Thread-local `ConnectContext` pollution after leader forwarding. [#71141](https://github.com/StarRocks/starrocks/pull/71141) * Orphaned delvec entries when write and compaction transactions are published in the same batch. [#71001](https://github.com/StarRocks/starrocks/pull/71001) [#71049](https://github.com/StarRocks/starrocks/pull/71049) [#71107](https://github.com/StarRocks/starrocks/pull/71107) * Missing partition predicates in short-circuit point lookups. [#71124](https://github.com/StarRocks/starrocks/pull/71124) * Potential hash-table data loss during aggregation spill `set_finishing`. [#70851](https://github.com/StarRocks/starrocks/pull/70851) * Query-progress HTTP loopback records from `current_queries`. [#71032](https://github.com/StarRocks/starrocks/pull/71032) * Primary Key tablet rowset metadata loss caused by a GC race during disk re-migration (A→B→A). [#70727](https://github.com/StarRocks/starrocks/pull/70727) * DB read-lock leaks in `SharedDataStorageVolumeMgr`. [#70987](https://github.com/StarRocks/starrocks/pull/70987) * Incorrect `NullColumn` sharing in `NullableColumn`, `BinaryColumn`, and `locate()`. [#66037](https://github.com/StarRocks/starrocks/pull/66037) [#70957](https://github.com/StarRocks/starrocks/pull/70957) * Race conditions in global runtime-filter readiness checks and corrected metric overflow caused by `ACCUMULATED` macro truncation. [#70920](https://github.com/StarRocks/starrocks/pull/70920) [#70889](https://github.com/StarRocks/starrocks/pull/70889) * Generated-column is not displayed in `DESC` and `SHOW CREATE TABLE`. [#70037](https://github.com/StarRocks/starrocks/pull/70037) * An issue with load spill directory cleanup timing, an ASAN crash in memory table spiller workgroup handling, and CN crashes when scanning empty tablets with physical split enabled. [#70778](https://github.com/StarRocks/starrocks/pull/70778) [#64379](https://github.com/StarRocks/starrocks/pull/64379) [#70281](https://github.com/StarRocks/starrocks/pull/70281) * Incorrect `Content-Length` handling when `proxy_pass_request_body` is off. [#70821](https://github.com/StarRocks/starrocks/pull/70821) * Issues with connector scan retry state handling and multiple schema pushdown issues for `INSERT INTO BY NAME ... FROM FILES()`. [#70762](https://github.com/StarRocks/starrocks/pull/70762) [#70774](https://github.com/StarRocks/starrocks/pull/70774) [#70622](https://github.com/StarRocks/starrocks/pull/70622) [#70621](https://github.com/StarRocks/starrocks/pull/70621) * Invalid conjunct pushdown in MySQL and JDBC scan nodes that caused BE predicate type errors. [#70694](https://github.com/StarRocks/starrocks/pull/70694) * Incomplete and partially written Iceberg manifest cache entries, and bypassed catalog caching when vended credentials are enabled. [#70675](https://github.com/StarRocks/starrocks/pull/70675) [#70652](https://github.com/StarRocks/starrocks/pull/70652) [#69434](https://github.com/StarRocks/starrocks/pull/69434) * Ubuntu runtime dependency issues by installing `libssl-dev`. [#70688](https://github.com/StarRocks/starrocks/pull/70688) * User authentication strings are not masked in audit logs and SQL redaction. [#70360](https://github.com/StarRocks/starrocks/pull/70360) * External materialized view refresh issues for Iceberg-like connectors. [#70589](https://github.com/StarRocks/starrocks/pull/70589) [#70523](https://github.com/StarRocks/starrocks/pull/70523) * `array_map` crashes on null literal arrays and BE crashes when a child iterator is exhausted in `MaskMergeIterator`. [#70629](https://github.com/StarRocks/starrocks/pull/70629) [#70539](https://github.com/StarRocks/starrocks/pull/70539) * `starlet` configuration updates were incorrectly captured through `std::call_once`. [#70482](https://github.com/StarRocks/starrocks/pull/70482) * Robustness issue with Iceberg materialized view refresh when snapshot timestamps are non-monotonic. [#70382](https://github.com/StarRocks/starrocks/pull/70382) * Issues that forced materialized view refresh is not supported, and duplicated partition names during materialized view refresh. [#70381](https://github.com/StarRocks/starrocks/pull/70381) [#70354](https://github.com/StarRocks/starrocks/pull/70354) * Incorrect distribution-column handling across partitions in `OlapTableSink`. [#70310](https://github.com/StarRocks/starrocks/pull/70310) * File-existence checks are not cached across tablet metadata versions during missing-file validation. [#70364](https://github.com/StarRocks/starrocks/pull/70364) * Issue with `dataVersion` handling during RESTORE, and incorrect materialized view rewrite logic. [#70373](https://github.com/StarRocks/starrocks/pull/70373) [#69751](https://github.com/StarRocks/starrocks/pull/69751) * Negative `query_pool` memory accounting during ingestion, and high FE OOM risk under high concurrency. [#70228](https://github.com/StarRocks/starrocks/pull/70228) [#68444](https://github.com/StarRocks/starrocks/pull/68444) * Incorrect SLF4J parameterized logging. [#70330](https://github.com/StarRocks/starrocks/pull/70330) * `AuditEventProcessor` exiting on `OutOfMemoryException`. [#70206](https://github.com/StarRocks/starrocks/pull/70206) * Adjusted handling of column-mode partial updates for Primary Key tables; the initial corruption fix was reverted in this release cycle for follow-up work. [#69652](https://github.com/StarRocks/starrocks/pull/69652) * Equality and deduplication issues in `InformationFunction`. [#70464](https://github.com/StarRocks/starrocks/pull/70464) * `brpc` connection retries when exceptions are wrapped in `NoSuchElementException`. [#70203](https://github.com/StarRocks/starrocks/pull/70203) * Lock-free materialized view rewrite does not fallback to live metadata. [#70475](https://github.com/StarRocks/starrocks/pull/70475) * Issue with `JoinHashTable::merge_ht()` that it does not skip dummy rows for expression-based join-key columns. [#70465](https://github.com/StarRocks/starrocks/pull/70465) #### 3.5.15[​](#3515 "Direct link to 3.5.15") Release Date: March 26, 2026 ##### Behavior Changes[​](#behavior-changes-4 "Direct link to Behavior Changes") * Improved `sql_mode` handling: when `DIVISION_BY_ZERO` or `FAIL_PARSE_DATE` mode is set, division by zero and date parse failures in `str_to_date`/`str2date` now return an error instead of being silently ignored. [#70004](https://github.com/StarRocks/starrocks/pull/70004) * When `sql_mode` is set to `FORBID_INVALID_DATE`, invalid dates in `INSERT VALUES` clauses are now correctly rejected instead of being bypassed. [#69803](https://github.com/StarRocks/starrocks/pull/69803) * Expression partition generated columns are now hidden from `DESC` and `SHOW CREATE TABLE` output. [#69793](https://github.com/StarRocks/starrocks/pull/69793) * Client ID is no longer included in audit logs. [#69383](https://github.com/StarRocks/starrocks/pull/69383) * The `FORCE` option for `REFRESH EXTERNAL TABLE` has been reverted and is no longer supported. [#70428](https://github.com/StarRocks/starrocks/pull/70428) * Aligned the backslash escape sequence usage in LIKE predicates with MySQL. Users need to specify four backslashes `\\\\` in sequence in a LIKE predicate to match a literal backslash `\` in the data, and eight backslashes `\\\\\\\\` in the predicate to match two consecutive literal backslashes `\\`. ##### Improvements[​](#improvements-5 "Direct link to Improvements") * Allowed disabling split and reverse scan ranges for descending TopN by setting `desc_hint_split_range` to `0` or less. [#70307](https://github.com/StarRocks/starrocks/pull/70307) * `information_schema` now shows comments for external catalog tables. [#70197](https://github.com/StarRocks/starrocks/pull/70197) * Added `EXPLAIN` and `EXPLAIN ANALYZE` support for `INSERT` statements in Trino dialect. [#70174](https://github.com/StarRocks/starrocks/pull/70174) * Added configurable parameters for `CatalogRecycleBin` to control recycle bin behavior. [#69838](https://github.com/StarRocks/starrocks/pull/69838) * Improved `ADMIN REPAIR TABLE` and `SHOW TABLET STATUS` to provide better repair and status information. [#69656](https://github.com/StarRocks/starrocks/pull/69656) * Blacklisted queries are now excluded from error metrics. [#69621](https://github.com/StarRocks/starrocks/pull/69621) * Added support for `SHOW TABLET STATUS` for cloud-native tablets in shared-data deployments. [#69616](https://github.com/StarRocks/starrocks/pull/69616) * Reduced overhead of Primary Key tablet statistics collection in shared-data clusters. [#69548](https://github.com/StarRocks/starrocks/pull/69548) * Added support for dynamic configuration of the execution state report thread pool size. [#69142](https://github.com/StarRocks/starrocks/pull/69142) ##### Bug Fixes[​](#bug-fixes-5 "Direct link to Bug Fixes") Fixed the following bugs: * Data version not set when restoring a tablet. [#70373](https://github.com/StarRocks/starrocks/pull/70373) * Table comment not set when creating a Hive table. [#70318](https://github.com/StarRocks/starrocks/pull/70318) * Constant folding with double precision arithmetic producing `INF` instead of returning an error. [#70309](https://github.com/StarRocks/starrocks/pull/70309) * Iceberg materialized view refresh failing when snapshot timestamps are non-monotonic. [#70382](https://github.com/StarRocks/starrocks/pull/70382) * `toIcebergTable` function using `common` instead of `comment` in property mapping. [#70267](https://github.com/StarRocks/starrocks/pull/70267) * Root user not correctly bypassing Ranger permission checks in all scenarios. [#70254](https://github.com/StarRocks/starrocks/pull/70254) * `AuditEventProcessor` thread exiting unexpectedly when an `OutOfMemoryException` occurs. [#70206](https://github.com/StarRocks/starrocks/pull/70206) * Out-of-bounds access in `cal_new_base_version` during schema change publish. [#70132](https://github.com/StarRocks/starrocks/pull/70132) * Partition predicates pruned unexpectedly due to type mismatch in boundary comparison. [#70097](https://github.com/StarRocks/starrocks/pull/70097) * `str_to_date` losing microsecond precision in BE runtime. [#70068](https://github.com/StarRocks/starrocks/pull/70068) * Crash in join spill process when `set_callback_function` is called. [#70030](https://github.com/StarRocks/starrocks/pull/70030) * DCHECK failure in `DeltaWriter::close()` when called from a bthread context. [#69960](https://github.com/StarRocks/starrocks/pull/69960) * Use-after-free race condition in `AsyncDeltaWriter` close/finish lifecycle. [#69940](https://github.com/StarRocks/starrocks/pull/69940) * Journal replay not awaited in `changeCatalogDb` on follower FE, causing consistency issues. [#69834](https://github.com/StarRocks/starrocks/pull/69834) * Race condition causing missed write transaction finished editlog. [#69899](https://github.com/StarRocks/starrocks/pull/69899) * Several known CVEs addressed. [#69863](https://github.com/StarRocks/starrocks/pull/69863) * Incorrect LIKE pattern matching with backslash escape sequences. [#69775](https://github.com/StarRocks/starrocks/pull/69775) * Expression analysis failing after renaming a partition column. [#69771](https://github.com/StarRocks/starrocks/pull/69771) * Use-after-free crash in `AsyncDeltaWriter::close`. [#69770](https://github.com/StarRocks/starrocks/pull/69770) * Potential bugs in `PartitionColumnMinMaxRewriteRule` caused by incorrect `Partition.hasStorageData` results. [#69751](https://github.com/StarRocks/starrocks/pull/69751) * Duplicated CSV compression suffix in file sink output file names. [#69749](https://github.com/StarRocks/starrocks/pull/69749) * Lake `capture_tablet_and_rowsets` operation accessible without experimental config flag. [#69748](https://github.com/StarRocks/starrocks/pull/69748) * Corrupted cache for Primary Key SST tables. [#69693](https://github.com/StarRocks/starrocks/pull/69693) * Use-after-free in `AsyncFlushOutputStream`. [#69688](https://github.com/StarRocks/starrocks/pull/69688) * Incorrect retention clock reset and incomplete scan in `disableRecoverPartitionWithSameName`. [#69677](https://github.com/StarRocks/starrocks/pull/69677) * Tablet info not fetched correctly based on run mode in `SchemaBeTabletsScanner`. [#69645](https://github.com/StarRocks/starrocks/pull/69645) * Incorrect minimum partition pruning with shadow partitions. [#69641](https://github.com/StarRocks/starrocks/pull/69641) * Different transactions publishing the same version after graceful exit. [#69639](https://github.com/StarRocks/starrocks/pull/69639) * Iterator undefined behavior in `get_column_values` when `rssid` is not found. [#69617](https://github.com/StarRocks/starrocks/pull/69617) * `KILL ANALYZE` statement sometimes not stopping a running `ANALYZE TABLE` operation. [#69592](https://github.com/StarRocks/starrocks/pull/69592) * Materialized view force refresh bugs for partition tables. [#69488](https://github.com/StarRocks/starrocks/pull/69488) #### 3.5.14[​](#3514 "Direct link to 3.5.14") Release Date: March 5, 2026 ##### Improvements[​](#improvements-6 "Direct link to Improvements") * Added SST read/write failure metrics for Primary Key index in Lake tables. [#69513](https://github.com/StarRocks/starrocks/pull/69513) * Added a counter metric for "segment file not found" errors. [#69543](https://github.com/StarRocks/starrocks/pull/69543) * Extracted range predicates from scalar-subquery containing `convert_tz`. [#69055](https://github.com/StarRocks/starrocks/pull/69055) * Supports complex type for Paimon tables. [#66784](https://github.com/StarRocks/starrocks/pull/66784) * Deferred remote load Spill Directory removal. [#68803](https://github.com/StarRocks/starrocks/pull/68803) * Supports repairing cloud-native tables. [#67108](https://github.com/StarRocks/starrocks/pull/67108) * Supports inserting ARRAY type to Hive table in CSV format. [#67355](https://github.com/StarRocks/starrocks/pull/67355) ##### Bug Fixes[​](#bug-fixes-6 "Direct link to Bug Fixes") The following issues have been fixed: * Unexpected behavior caused by exceptions of `RowGroupWriter`. [#69568](https://github.com/StarRocks/starrocks/pull/69568) * Sort key not including newly added key columns after schema change on Aggregate Key/Unique Key tables. [#69529](https://github.com/StarRocks/starrocks/pull/69529) * Mertic value `g_publish_version_failed_tasks` does not reflect the real situation during the `resource_busy` state. [#69526](https://github.com/StarRocks/starrocks/pull/69526) * Rowset files are removed when moving Primary Key tablets to trash. [#69438](https://github.com/StarRocks/starrocks/pull/69438) * Lock leak in `addPartitions` caused by name-based table lookup after concurrent SWAP. [#69284](https://github.com/StarRocks/starrocks/pull/69284) * `isInternalCancelError` used `equals` instead of `startsWith`. [#69523](https://github.com/StarRocks/starrocks/pull/69523) * Pipeline blocks or crashes when `_writer->Close()` throws an exception other than `ParquetStatusException`. [#69492](https://github.com/StarRocks/starrocks/pull/69492) * A Hadoop-client lib bug. [#69503](https://github.com/StarRocks/starrocks/pull/69503) * Success is mistakenly returned while write operations fails. [#69473](https://github.com/StarRocks/starrocks/pull/69473) * CVE-2025-67721. [#69138](https://github.com/StarRocks/starrocks/pull/69138) * Issue with RuntimeFilter with low-cardinality optimization in share-data clusters. [#64669](https://github.com/StarRocks/starrocks/pull/64669) * Materialized view tablet meta inconsistency between FE leader and follower. [#69428](https://github.com/StarRocks/starrocks/pull/69428) * Rollup handler's active transaction ID was not considered in `computeMinActiveTxnId`. [#69285](https://github.com/StarRocks/starrocks/pull/69285) * Arrow Flight Proxy issue with multiple FE. [#68300](https://github.com/StarRocks/starrocks/pull/68300) * Concurrency bug of function field. [#69315](https://github.com/StarRocks/starrocks/pull/69315) * `DROP FUNCTION IF EXISTS` ignored `ifExists` flag. [#69216](https://github.com/StarRocks/starrocks/pull/69216) * Lacking case-insensitive username normalization for LDAP authentication. [#67966](https://github.com/StarRocks/starrocks/pull/67966) * Certain kinds of partitions cannot be written. [#68221](https://github.com/StarRocks/starrocks/pull/68221) * Projection loss in materialized view rewrite due to shared mutable state. [#69063](https://github.com/StarRocks/starrocks/pull/69063) * Issue with case-insensitive partition lookup in query table copy. [#69173](https://github.com/StarRocks/starrocks/pull/69173) * All-null value handling bug in synchronous materialized views. [#69136](https://github.com/StarRocks/starrocks/pull/69136) * `mv onReload` issues when visiting external catalogs. [#68926](https://github.com/StarRocks/starrocks/pull/68926) * DISTINCT ORDER BY alias issues for duplicated constants. [#69014](https://github.com/StarRocks/starrocks/pull/69014) * Incorrect query results after modifying CHAR column length in shared-data clusters. [#68808](https://github.com/StarRocks/starrocks/pull/68808) * Issue with Azure ABFS/WASB FileSystem cache key. [#68901](https://github.com/StarRocks/starrocks/pull/68901) * Incorrect predicate rewrite for OUTER JOIN with constant-side column reference. [#67072](https://github.com/StarRocks/starrocks/pull/67072) * `IllegalArgumentException` comparator transitivity violation. [#68743](https://github.com/StarRocks/starrocks/pull/68743) * Issue caused by the query lifetime being shorter than the fragment in `report_fragment`. [#67219](https://github.com/StarRocks/starrocks/pull/67219) * Low-cardinality rewrite NPE caused by shared `DecodeInfo`. [#68799](https://github.com/StarRocks/starrocks/pull/68799) * Missing `pcu_upt_cnt` metric. [#68845](https://github.com/StarRocks/starrocks/pull/68845) * JSON-flatten array/object conflict on identical paths. [#68804](https://github.com/StarRocks/starrocks/pull/68804) * `ClonExpr` nullable bug. [#68800](https://github.com/StarRocks/starrocks/pull/68800) #### 3.5.13[​](#3513 "Direct link to 3.5.13") Release Date: February 13, 2026 ##### Improvements[​](#improvements-7 "Direct link to Improvements") * Added an FE configuration `enable_table_metrics_collect` to control the collection of table-level metrics. [#68691](https://github.com/StarRocks/starrocks/pull/68691) * Supports setting the default Warehouse for Merge Commit at user level. [#68616](https://github.com/StarRocks/starrocks/pull/68616) ##### Bug fixes[​](#bug-fixes-7 "Direct link to Bug fixes") The following issues have been fixed: * Issue with source partition checking in replication transactions. [#68883](https://github.com/StarRocks/starrocks/pull/68883) * Used labels were not identified when labels were specified in BEGIN TRANSACTION. [#68660](https://github.com/StarRocks/starrocks/pull/68660) * JOIN ON clause bug with CTE scope. [#68809](https://github.com/StarRocks/starrocks/pull/68809) * Overlapping range partitions can be created when an explicit lower bound is provided. [#68255](https://github.com/StarRocks/starrocks/pull/68255) * Incorrect parser logic when SQL dialect downgrades from Trino to StarRocks. [#68725](https://github.com/StarRocks/starrocks/pull/68725) * Issue with pruning projection columns. [#68242](https://github.com/StarRocks/starrocks/pull/68242) * Issue with subquery scope check. [#68415](https://github.com/StarRocks/starrocks/pull/68415) * Unmatched type cast in the function analyzer. [#66749](https://github.com/StarRocks/starrocks/pull/66749) * Incorrect candidate materialized view selection logic. [#68571](https://github.com/StarRocks/starrocks/pull/68571) * The Thrift `accept` thread exits on exception. [#68644](https://github.com/StarRocks/starrocks/pull/68644) * Inaccurate Iceberg data file size estimation. [#68787](https://github.com/StarRocks/starrocks/pull/68787) * Lake table memory leak issue. [#68678](https://github.com/StarRocks/starrocks/pull/68678) * Deadlock when the HMS connection pool is full. [#68033](https://github.com/StarRocks/starrocks/pull/68033) * Iceberg delete column nullability issue. [#68649](https://github.com/StarRocks/starrocks/pull/68649) * Materialized views hold large external tables. [#68171](https://github.com/StarRocks/starrocks/pull/68171) * Iceberg table cache memory limit issue. [#67769](https://github.com/StarRocks/starrocks/pull/67769) * Wrong timeout parameter is used for PocoHttpClient. [#68765](https://github.com/StarRocks/starrocks/pull/68765) * BE compile failure with Clang. [#68805](https://github.com/StarRocks/starrocks/pull/68805) * Materialized view was reloaded multiple times during startup. [#62351](https://github.com/StarRocks/starrocks/pull/62351) * CVE-2025-27821. [#68529](https://github.com/StarRocks/starrocks/pull/68529) * Variadic functions return incorrect date values in certain scenarios. [#67947](https://github.com/StarRocks/starrocks/pull/67947) #### 3.5.12[​](#3512 "Direct link to 3.5.12") Release Date: January 22, 2026 ##### Improvements[​](#improvements-8 "Direct link to Improvements") * Added a cleaner for BrpcStubCache to clean up unused connections. [#61417](https://github.com/StarRocks/starrocks/pull/61417) * Supports batch processing for statistics delete (for dropped tables) and Edit Log write requests. [#67896](https://github.com/StarRocks/starrocks/pull/67896) * Preserves SQL comments in Audit Logs when encryption is required. [#63298](https://github.com/StarRocks/starrocks/pull/63298) * Added the `warehouse_name` label to the materialized view metrics. [#67715](https://github.com/StarRocks/starrocks/pull/67715) * Improved identifier wrapping for JDBC table and column names. [#67853](https://github.com/StarRocks/starrocks/pull/67853) * Added the `CLIENT_FACTORY` property to the Iceberg JDBC catalog. [#67613](https://github.com/StarRocks/starrocks/pull/67613) ##### Bug fixes[​](#bug-fixes-8 "Direct link to Bug fixes") The following issues have been fixed: * Variadic functions return wrong dates when mixing DATE and DATETIME types. [#67947](https://github.com/StarRocks/starrocks/pull/67947) * `NormalizePredicateRule` oscillation on non-deterministic expressions. [#67923](https://github.com/StarRocks/starrocks/pull/67923) * Low cardinality bugs with the Lambda function. [#67843](https://github.com/StarRocks/starrocks/pull/67843) * Subfield expression does not collect children subfields. [#67850](https://github.com/StarRocks/starrocks/pull/67850) * NPE in RBO Join reorder when child statistics are missing. [#67693](https://github.com/StarRocks/starrocks/pull/67693) * BE crash due to MemTable finalize failed. [#67787](https://github.com/StarRocks/starrocks/pull/67787) * Temporary partitions are not cleaned up after FE restart for dynamic overwrite. [#67629](https://github.com/StarRocks/starrocks/pull/67629) * Inaccurate I/O statistics of Compaction. [#67524](https://github.com/StarRocks/starrocks/pull/67524) * Incorrect logic in physical partition comparison across clusters during replication transaction. [#67616](https://github.com/StarRocks/starrocks/pull/67616) * Issue with SQL Server and Oracle identifier symbol handling. [#67965](https://github.com/StarRocks/starrocks/pull/67965) * NPE in the Iceberg metadata table query due to missing configuration propagation. [#67151](https://github.com/StarRocks/starrocks/pull/67151) * Issue with the `f``iles()` schema detection for empty Parquet or ORC files. [#67762](https://github.com/StarRocks/starrocks/pull/67762) * Inaccurate value of metrics in Profile caused by UNION ALL on Hive tables. [#67912](https://github.com/StarRocks/starrocks/pull/67912) * Lacking support for data retrieval from Arrow Flight proxy for FE queries. [#67794](https://github.com/StarRocks/starrocks/pull/67794) * SIGSEGV crash during automatic partition creation caused by a race condition in `OlapTableSink::is_full()`. [#67566](https://github.com/StarRocks/starrocks/pull/67566) #### 3.5.11[​](#3511 "Direct link to 3.5.11") Release date: January 5, 2026 ##### Improvements[​](#improvements-9 "Direct link to Improvements") * Supports Arrow Flight data retrieval from inaccessible nodes. [#66348](https://github.com/StarRocks/starrocks/pull/66348) * Logs the cause (including the triggering process information) in the SIGTERM handler. [#66737](https://github.com/StarRocks/starrocks/pull/66737) * Added an FE configuration `enable_statistic_collect_on_update` to control whether UPDATE statements can trigger automatic statistics collection. [#66794](https://github.com/StarRocks/starrocks/pull/66794) * Supports configuring `networkaddress.cache.ttl`. [#66723](https://github.com/StarRocks/starrocks/pull/66723) * Improve the “no rows imported” error message. [#66624](https://github.com/StarRocks/starrocks/pull/66624) [#66535](https://github.com/StarRocks/starrocks/pull/66535) * Optimized `deltaRows` with lazy evaluation for large partition tables. [#66381](https://github.com/StarRocks/starrocks/pull/66381) * Optimized materialized view rewrite performance. [#66623](https://github.com/StarRocks/starrocks/pull/66623) * Supports single-tablet `ResultSink` optimization in shared-data clusters. [#66517](https://github.com/StarRocks/starrocks/pull/66517) * `rewrite``_``simple``_``agg``_``to``_``meta``_``scan` is enabled by default. [#64698](https://github.com/StarRocks/starrocks/pull/64698) * Supports pushing down GROUP BY expressions and materialized view rewrite. [#66507](https://github.com/StarRocks/starrocks/pull/66507) * Add overloaded `newMessage` methods to improve materialized view logs. [#66367](https://github.com/StarRocks/starrocks/pull/66367) ##### Bug Fixes[​](#bug-fixes-9 "Direct link to Bug Fixes") The following issues have been fixed: * A Publish Compaction crash when the input rowset is not found. [#67154](https://github.com/StarRocks/starrocks/pull/67154) * Significant CPU overhead and lock contention caused by repetitive invocation of `update_segment_cache_size` when querying tables with a large number of columns. [#66714](https://github.com/StarRocks/starrocks/pull/66714) * `MulticastSinkOperator` stuck in the `OUTPUT_FULL` state. [#67153](https://github.com/StarRocks/starrocks/pull/67153) * A “column not found” issue in the skew join hint. [#66929](https://github.com/StarRocks/starrocks/pull/66929) * The growth of all tablets continues unabated, and the sum of pending and running tablets is not the total number of tablets. [#66718](https://github.com/StarRocks/starrocks/pull/66718) * Transactions in the Compaction map built during Leader startup cannot be accessed by CompactionScheduler and will never be removed from the map. [#66533](https://github.com/StarRocks/starrocks/pull/66533) * Delta Lake table refresh does not take effect. [#67156](https://github.com/StarRocks/starrocks/pull/67156) * CN crash at queries against non-partitioned Iceberg tables with DATE predicates. [#66864](https://github.com/StarRocks/starrocks/pull/66864) * Statements in Profiles cannot be correctly displayed when multiple statements are submitted. [#67097](https://github.com/StarRocks/starrocks/pull/67097) * Missing dictionary information during collection because Meta Reader does not support reading from Delta column group files. [#66995](https://github.com/StarRocks/starrocks/pull/66995) * Potential Java heap OOM in Java UDAF. [#67025](https://github.com/StarRocks/starrocks/pull/67025) * BE crash due to the incorrect logic of ranking window optimization without PARTITION BY and ORDER BY. [#67081](https://github.com/StarRocks/starrocks/pull/67081) * Misleading log level for timezone cache miss. [#66817](https://github.com/StarRocks/starrocks/pull/66817) * Crash and incorrect results caused by the incorrect `can_use_bf` checking when merging runtime filters. [#67021](https://github.com/StarRocks/starrocks/pull/67021) * Issue about pushing down runtime bitset filter with other OR predicates. [#66996](https://github.com/StarRocks/starrocks/pull/66996) * Patch critical fix from lz4. [#67053](https://github.com/StarRocks/starrocks/pull/67053) * AsyncTaskQueue deadlock issue. [#66791](https://github.com/StarRocks/starrocks/pull/66791) * Cache inconsistency in ObjectColumn. [#66957](https://github.com/StarRocks/starrocks/pull/66957) * RewriteUnnestBitmapRule causes wrong output column types. [#66855](https://github.com/StarRocks/starrocks/pull/66855) * Data races and data loss when there are WRITE or FLUSH tasks after FINISH tasks in the Delta Writer. [#66943](https://github.com/StarRocks/starrocks/pull/66943) * Invalid load channel and misleading internal errors caused by reopened load channels that were previously aborted. [#66793](https://github.com/StarRocks/starrocks/pull/66793) * Bugs of Arrow Flight SQL. [#65889](https://github.com/StarRocks/starrocks/pull/65889) * Issues when querying renamed columns with MetaScan. [#66819](https://github.com/StarRocks/starrocks/pull/66819) * Hash column is not removed before flushing chunk in partitionwise spillable aggregation when skew elimination is off. [#66839](https://github.com/StarRocks/starrocks/pull/66839) * BOOLEAN type default values were not correctly handled when stored as string literals. [#66818](https://github.com/StarRocks/starrocks/pull/66818) * decimal2decimal cast unexpectedly returns the input column as the result directly. [#66773](https://github.com/StarRocks/starrocks/pull/66773) * NPE in query planning during schema change. [#66811](https://github.com/StarRocks/starrocks/pull/66811) * LocalTabletsChannel and LakeTabletsChannel deadlock. [#66748](https://github.com/StarRocks/starrocks/pull/66748) * `publish_version` log shows empty `txn_ids` with new FE. [#66732](https://github.com/StarRocks/starrocks/pull/66732) * Incorrect behavior of the FE configuration `statistic_collect_query_timeout`. [#66363](https://github.com/StarRocks/starrocks/pull/66363) * UPDATE statements do not support statistics collection. [#66443](https://github.com/StarRocks/starrocks/pull/66443) * Case rewrite errors related to low cardinality. [#66724](https://github.com/StarRocks/starrocks/pull/66724) * Statistics query failure when the column list is empty. [#66138](https://github.com/StarRocks/starrocks/pull/66138) * Usage/record mismatch when switching warehouse via hint. [#66677](https://github.com/StarRocks/starrocks/pull/66677) * `ANALYZE TABLE` statements lack ExecTimeout. [#66361](https://github.com/StarRocks/starrocks/pull/66361) * `array_map` returns wrong results from constant unary expressions. [#66514](https://github.com/StarRocks/starrocks/pull/66514) * Foreign key constraints are lost after FE restart. [#66474](https://github.com/StarRocks/starrocks/pull/66474) * `max(not null string)` on empty table throws `std::length_error`. [#66554](https://github.com/StarRocks/starrocks/pull/66554) * Concurrency issue between Primary Key index Compaction and Apply. [#66282](https://github.com/StarRocks/starrocks/pull/66282) * Improper behavior of `EXPLAIN `. [#66542](https://github.com/StarRocks/starrocks/pull/66542) * Issue when sinking DECIMAL128 to Iceberg table column. [#66071](https://github.com/StarRocks/starrocks/pull/66071) * JSON length check issue for JSON → CHAR/VARCHAR when the target length equals the minimum. [#66628](https://github.com/StarRocks/starrocks/pull/66628) * An expression children count error. [#66511](https://github.com/StarRocks/starrocks/pull/66511) #### 3.5.10[​](#3510 "Direct link to 3.5.10") Release date: December 15, 2025 ##### Improvements[​](#improvements-10 "Direct link to Improvements") * Supports dumping plan node IDs in BE crash logs to speed up locating problematic operators. [#66454](https://github.com/StarRocks/starrocks/pull/66454) * Optimized scans on the system views in `information_schema` to reduce the overhead. [#66200](https://github.com/StarRocks/starrocks/pull/66200) * Added two histogram metrics (`slow_lock_held_time_ms` and `slow_lock_wait_time_ms`) to provide better observability for slow lock scenarios and distinguish between long-held locks and high lock contention. [#66027](https://github.com/StarRocks/starrocks/pull/66027) * Optimized replica lock handling in tablet report and clone flows by switching the lock from database level to table level, reducing lock contention and improving scheduling efficiency. [#61939](https://github.com/StarRocks/starrocks/pull/61939) * Avoided outputting columns in BE storage, and pushed down predicate computation to BE storage. [#60462](https://github.com/StarRocks/starrocks/pull/60462) * Improved query profile accuracy when deploying scan ranges in background threads. [#62223](https://github.com/StarRocks/starrocks/pull/62223) * Improved profile accounting when deploying additional tasks, so CPU time is not repetitively counted. [#62186](https://github.com/StarRocks/starrocks/pull/62186) * Added more detailed error messages when a referenced partition does not exist, making failures easier to diagnose. [#65674](https://github.com/StarRocks/starrocks/pull/65674) * Made sample-type cardinality estimation more robust in corner cases to improve row-count estimates. [#65599](https://github.com/StarRocks/starrocks/pull/65599) * Added a partition filter when loading statistics to prevent INSERT OVERWRITE from reading stale partition statistics. [#65578](https://github.com/StarRocks/starrocks/pull/65578) * Splited pipeline CPU `execution_time` metrics into separate series for queries and loads, improving observability by workload type. [#65535](https://github.com/StarRocks/starrocks/pull/65535) * Supported `enable_statistic_collect_on_first_load` at table granularity for finer-grained control over statistics collection on the first load. [#65463](https://github.com/StarRocks/starrocks/pull/65463) * Renamed the S3-dependent unit test from `PocoClientTest` to an S3-specific name to better reflect its dependency and intent. [#65524](https://github.com/StarRocks/starrocks/pull/65524) ##### Bug Fixes[​](#bug-fixes-10 "Direct link to Bug Fixes") The following issues have been fixed: * libhdfs crashes when StarRocks is started with an incompatible JDK. [#65882](https://github.com/StarRocks/starrocks/pull/65882) * Incorrect query results caused by `PartitionColumnMinMaxRewriteRule`. [#66356](https://github.com/StarRocks/starrocks/pull/66356) * Rewrite issues due to the materialized view metadata not refreshed when resolving materialized views by AST keys. [#66472](https://github.com/StarRocks/starrocks/pull/66472) * The trim function crashes or produces wrong results when trimming specific Unicode whitespace characters. [#66428](https://github.com/StarRocks/starrocks/pull/66428), [#66477](https://github.com/StarRocks/starrocks/pull/66477) * Failures in load metadata and SQL execution that still referenced a deleted warehouse. [#66436](https://github.com/StarRocks/starrocks/pull/66436) * Wrong results when group execution Join is combined with window functions. [#66441](https://github.com/StarRocks/starrocks/pull/66441) * A possible FE null pointer in `resetDecommStatForSingleReplicaTabletUnlocked`. [#66034](https://github.com/StarRocks/starrocks/pull/66034) * Missing Join runtime filter pushdown optimization in shared-data clusters for `LakeDataSource`. [#66354](https://github.com/StarRocks/starrocks/pull/66354) * Parameters are inconsistent for runtime filter transmit options (timeouts, HTTP RPC limits, etc.) because they are not forwarded to receivers. [#66393](https://github.com/StarRocks/starrocks/pull/66393) * Automatic partition creation fails when partition values already exist. [#66167](https://github.com/StarRocks/starrocks/pull/66167) * Inaccurate scan statistics in audit logs when predicates have high selectivity. [#66280](https://github.com/StarRocks/starrocks/pull/66280) * Incorrect query results because non-deterministic functions are pushed below operators. [#66323](https://github.com/StarRocks/starrocks/pull/66323) * Exponential growth in the number of expressions caused by CASE WHEN. [#66324](https://github.com/StarRocks/starrocks/pull/66324) * Materialized view compensation bugs when the same table appears multiple times in a query with different partition predicates. [#66369](https://github.com/StarRocks/starrocks/pull/66369) * BE becomes unresponsive when using fork in subprocesses. [#66334](https://github.com/StarRocks/starrocks/pull/66334) * CVE-2025-66566 and CVE-2025-12183. [#66453](https://github.com/StarRocks/starrocks/pull/66453), [#66362](https://github.com/StarRocks/starrocks/pull/66362) * Errors caused by nested CTE reuse. [#65800](https://github.com/StarRocks/starrocks/pull/65800) * Issues due to the lack of validation on conflicting schema-change clauses. [#66208](https://github.com/StarRocks/starrocks/pull/66208) * Improper rowset GC behavior when rowset commit fails. [#66301](https://github.com/StarRocks/starrocks/pull/66301) * A potential use-after-free when counting down pipelines. [#65940](https://github.com/StarRocks/starrocks/pull/65940) * The `W``arehouse` field is NULL in `information_schema.loads` for Stream Load. [#66202](https://github.com/StarRocks/starrocks/pull/66202) * Issues with materialized view creation when the referenced view has the same name as its base table. [#66274](https://github.com/StarRocks/starrocks/pull/66274) * The global dictionary is not updated correctly under some cases. [#66194](https://github.com/StarRocks/starrocks/pull/66194) * Incorrect query profile logging for queries forwarded from Follower nodes. [#64395](https://github.com/StarRocks/starrocks/pull/64395) * BE crash when caching SELECT results and reordering schema. [#65850](https://github.com/StarRocks/starrocks/pull/65850) * Shadow partitions are dropped when dropping partitions by expression. [#66171](https://github.com/StarRocks/starrocks/pull/66171) * DROP tasks run when a CLONE task exists for the same tablet. [#65780](https://github.com/StarRocks/starrocks/pull/65780) * Stability and observability issues because RocksDB log file options were not properly set. [#66166](https://github.com/StarRocks/starrocks/pull/66166) * Incorrect materialized view compensation that could produce NULL results. [#66216](https://github.com/StarRocks/starrocks/pull/66216) * BE reported as alive even after receiving `SIGSEGV`. [#66212](https://github.com/StarRocks/starrocks/pull/66212) * Bugs in Iceberg scans. [#65658](https://github.com/StarRocks/starrocks/pull/65658) * Regression coverage and stability issues for Iceberg view SQL test cases. [#66126](https://github.com/StarRocks/starrocks/pull/66126) * Unexpected behavior because `set_collector` is invoked repetitively. [#66199](https://github.com/StarRocks/starrocks/pull/66199) * Ingestion failures when column-mode partial updates are used together with conditional updates. [#66139](https://github.com/StarRocks/starrocks/pull/66139) * Temporary partition value conflicts under concurrent transactions. [#66025](https://github.com/StarRocks/starrocks/pull/66025) * An Iceberg table cache bug where Guava LocalCache could retain stale entries even when `cache.size() == 0`, causing refresh to be ineffective and queries to return outdated tables. [#65917](https://github.com/StarRocks/starrocks/pull/65917) * Incorrect format placeholder in `LargeInPredicateException`, causing the actual number of LargeInPredicate occurrences to be incorrectly reported in the error message. [#66152](https://github.com/StarRocks/starrocks/pull/66152) * NullPointerException in `ConnectScheduler’s` timeout checker when connectContext is null. [#66136](https://github.com/StarRocks/starrocks/pull/66136) * Crashes caused by unhandled exceptions thrown from threadpool tasks. [#66114](https://github.com/StarRocks/starrocks/pull/66114) * Data loss when pushing down DISTINCT LIMIT in certain plans. [#66109](https://github.com/StarRocks/starrocks/pull/66109) * `multi_distinct_count` not updating `distinct_size` after the underlying hash set is converted to a two-level hash set, which could lead to incorrect distinct counts. [#65916](https://github.com/StarRocks/starrocks/pull/65916) * A race condition when an exec group submits the next driver which could trigger `Check failed: !driver->is_in_blocked() `and abort the BE process. [#66099](https://github.com/StarRocks/starrocks/pull/66099) * INSERT failures when running ALTER TABLE ADD COLUMN with a default value concurrently with INSERT, due to mismatched types for the newly added column’s default expression. [#65968](https://github.com/StarRocks/starrocks/pull/65968) * An issue where `MemoryScratchSinkOperator` could remain in `pending_finish` after `RecordBatchQueue` was shut down when SparkSQL exited early, causing the pipeline to hang. [#66041](https://github.com/StarRocks/starrocks/pull/66041) * A core dump when reading Parquet files that contain empty row groups. [#65928](https://github.com/StarRocks/starrocks/pull/65928) * Recursive calls and potential stack overflow at high DOP because the event scheduler’s readiness check is complicated. [#66016](https://github.com/StarRocks/starrocks/pull/66016) * Asynchronous materialized view refresh skips updates when the Iceberg base table contains expired snapshots. [#65969](https://github.com/StarRocks/starrocks/pull/65969) * Potential issues in predicate reuse and rewrite because the optimizer relies solely on hashCode to distinguish differences in predicates. [#65999](https://github.com/StarRocks/starrocks/pull/65999) * In the refresh of an asynchronous materialized view with multi-level partitioned base tables, only the parent partition metadata was checked, while sub-partition updates were skipped. [#65596](https://github.com/StarRocks/starrocks/pull/65596) * Statistics collection issues where AVG(ARRAY\_LENGTH(...)) could return NULL for empty result sets. [#65788](https://github.com/StarRocks/starrocks/pull/65788) * Runtime profile counters are not correctly updating or clearing their min/max values during incremental updates on both BE and FE. [#65869](https://github.com/StarRocks/starrocks/pull/65869) * Incorrect logic to obtain the image journal ID when creating a cluster snapshot to ensure the snapshot uses the correct log position. [#65970](https://github.com/StarRocks/starrocks/pull/65970) * Results are misreported when the cleanup fails due to incorrect status checking logic in file cleanup error handling. [#65709](https://github.com/StarRocks/starrocks/pull/65709) * A possible infinite loop in certain plans isVariable() in DictMappingOperator. [#65743](https://github.com/StarRocks/starrocks/pull/65743) * Failures and missing audit/profile data because ConnectContext is not passed into scan-range deployment threads. [#63544](https://github.com/StarRocks/starrocks/pull/63544) * A use-after-free issue in the local Primary Key index manager when the storage engine is stopped. [#65534](https://github.com/StarRocks/starrocks/pull/65534) * Statistics collection issues for INSERT OVERWRITE with dynamic overwrite. [#65657](https://github.com/StarRocks/starrocks/pull/65657) * Concurrency issues caused by coarse-grained locks in DiskAndTabletLoadReBalancer. [#65557](https://github.com/StarRocks/starrocks/pull/65557) * Slow locks cannot be detected and reported correctly for the lack of slow-lock detection for critical locks. [#65559](https://github.com/StarRocks/starrocks/pull/65559) * NullPointerException when replaying upsert transaction state after the target database has been dropped. [#65595](https://github.com/StarRocks/starrocks/pull/65595) * Stale statistics were used because outdated partition statistics are not dropped after statistics collection triggered by INSERT OVERWRITE. [#65586](https://github.com/StarRocks/starrocks/pull/65586) * Data race in partition ID allocation that could lead to ID conflicts under concurrency. [#65608](https://github.com/StarRocks/starrocks/pull/65608) * Missing tablet IDs when retrieving initial tablet metadata. [#65550](https://github.com/StarRocks/starrocks/pull/65550) * Incorrect record information for PREPARE/EXECUTE statements in audit and profile logs. [#65448](https://github.com/StarRocks/starrocks/pull/65448) * Potential crashes because the non–thread-safe has\_output function is called from multiple threads. [#65514](https://github.com/StarRocks/starrocks/pull/65514) * MemTable finalize tasks cannot be properly tracked because the `memtable_finalize_task_total` counter metric is lacking. [#65548](https://github.com/StarRocks/starrocks/pull/65548) * Query ID collisions in Arrow Flight, causing multiple queries no longer share the same query ID. [#65558](https://github.com/StarRocks/starrocks/pull/65558) * Lock conflicts for `TabletChecker.doCheck()` with other operations. [#65237](https://github.com/StarRocks/starrocks/pull/65237) * Scan behavior is inconsistent between shared-data and shared-nothing clusters, causing query semantics to differ. [#61100](https://github.com/StarRocks/starrocks/pull/61100) #### 3.5.9[​](#359 "Direct link to 3.5.9") Release date: November 26, 2025 ##### Improvements[​](#improvements-11 "Direct link to Improvements") * Added transaction latency metrics to FE for observing timing across transaction stages. [#64948](https://github.com/StarRocks/starrocks/pull/64948) * Supports overwriting S3 unpartitioned Hive tables to simplify full-table rewrites in data lake scenarios. [#65340](https://github.com/StarRocks/starrocks/pull/65340) * Introduced CacheOptions to provide finer-grained control over tablet metadata caching. [#65222](https://github.com/StarRocks/starrocks/pull/65222) * Supports sample statistics collection for INSERT OVERWRITE to ensure statistics stay consistent with the latest data. [#65363](https://github.com/StarRocks/starrocks/pull/65363) * Optimized the statistics collection strategy after INSERT OVERWRITE to avoid missing or incorrect statistics due to asynchronous tablet reports. [#65327](https://github.com/StarRocks/starrocks/pull/65327) * Introduced a retention period for partitions dropped or replaced by INSERT OVERWRITE or materialized view refresh operations, keeping them in the recycle bin for a while to improve recoverability. [#64779](https://github.com/StarRocks/starrocks/pull/64779) ##### Bug Fixes[​](#bug-fixes-11 "Direct link to Bug Fixes") The following issues have been fixed: * Lock contention and concurrency issues related to `LocalMetastore.truncateTable()`. [#65191](https://github.com/StarRocks/starrocks/pull/65191) * Lock contention and replica check performance issues related to TabletChecker. [#65312](https://github.com/StarRocks/starrocks/pull/65312) * Incorrect error logging when changing user via HTTP SQL. [#65371](https://github.com/StarRocks/starrocks/pull/65371) * Checksum failures caused by DelVec CRC32 upgrade compatibility issues. [#65442](https://github.com/StarRocks/starrocks/pull/65442) * Tablet metadata load failures caused by RocksDB iteration timeout. [#65146](https://github.com/StarRocks/starrocks/pull/65146) * When the internal `flat_path` string is empty because the JSON hyper extraction path is `$` or all paths are skipped, calling `substr` will throw an exception and cause BE crash. [#65260](https://github.com/StarRocks/starrocks/pull/65260) * The PREPARED flag in fragment execution is not correctly set. [#65423](https://github.com/StarRocks/starrocks/pull/65423) * Inaccurate write and flush metrics caused by duplicated load profile counters. [#65252](https://github.com/StarRocks/starrocks/pull/65252) * When multiple HTTP requests reuse the same TCP connection, if a non‑ExecuteSQL request arrives after an ExecuteSQL request, the `HttpConnectContext` cannot be unregistered at channel close, causing HTTP context leaks. [#65203](https://github.com/StarRocks/starrocks/pull/65203) * MySQL 8.0 schema introspection errors (Fixed by adding session variables `default_authentication_plugin` and `authentication_policy`). [#65330](https://github.com/StarRocks/starrocks/pull/65330) * SHOW ANALYZE STATUS errors caused by unnecessary statistics collection for temporary partitions created after partition overwrite operations. [#65298](https://github.com/StarRocks/starrocks/pull/65298) * Global Runtime Filter race in the Event Scheduler. [#65200](https://github.com/StarRocks/starrocks/pull/65200) * Data Cache is aggressively disabled because the minimum Data Cache disk size constraint is too large. [#64909](https://github.com/StarRocks/starrocks/pull/64909) * An aarch64 build issue related to the `gold` linker automatic fallback. [#65156](https://github.com/StarRocks/starrocks/pull/65156) #### 3.5.8[​](#358 "Direct link to 3.5.8") Release date: November 10, 2025 ##### Improvements[​](#improvements-12 "Direct link to Improvements") * Upgraded Arrow to 19.0.1 to support the Parquet legacy list to include nested, complex files. [#64238](https://github.com/StarRocks/starrocks/pull/64238) * FILES() supports legacy Parquet LIST encodings. [#64160](https://github.com/StarRocks/starrocks/pull/64160) * Automatically determine the Partial Update mode based on the session variable and the number of inserted columns. [#62091](https://github.com/StarRocks/starrocks/pull/62091) * Applied low-cardinality optimization on analytic operators above table functions. [#63378](https://github.com/StarRocks/starrocks/pull/63378) * Added configurable table lock timeout to `finishTransaction` to avoid blocking. [#63981](https://github.com/StarRocks/starrocks/pull/63981) * Shared-data clusters support table-level scan metrics attribution. [#62832](https://github.com/StarRocks/starrocks/pull/62832) * Window functions LEAD/LAG/FIRST\_VALUE/LAST\_VALUE now accept ARRAY type arguments. [#63547](https://github.com/StarRocks/starrocks/pull/63547) * Supports constant folding for several array functions to improve predicate pushdown and join simplification. [#63692](https://github.com/StarRocks/starrocks/pull/63692) * Supports batched API to optimize `tabletNum` retrieval for a given node via `SHOW PROC /backends/{id}`. Added an FE configuration item `enable_collect_tablet_num_in_show_proc_backend_disk_path` (Default: `true`). [#64013](https://github.com/StarRocks/starrocks/pull/64013) * Ensured `INSERT ... SELECT` reads the freshest metadata by refreshing external tables before planning. [#64026](https://github.com/StarRocks/starrocks/pull/64026) * Added `capacity_limit_reached` checks to table functions, NL-join probe, and hash-join probe to avoid constructing overflowing columns. [#64009](https://github.com/StarRocks/starrocks/pull/64009) * Added FE configuration item `collect_stats_io_tasks_per_connector_operator` (Default: `4`) for setting the maximum number of tasks to collect statistics for external tables. [#64016](https://github.com/StarRocks/starrocks/pull/64016) * Updated the default partition size for sample collection from 1000 to 300. [#64022](https://github.com/StarRocks/starrocks/pull/64022) * Increased lock table slots to 256 and added `rid` to slow-lock logs. [#63945](https://github.com/StarRocks/starrocks/pull/63945) * Improved robustness of Gson deserialization in the presence of legacy data. [#63555](https://github.com/StarRocks/starrocks/pull/63555) * Reduced metadata lock scope for FILES() schema pushdown to cut lock contention and planning latency. [#63796](https://github.com/StarRocks/starrocks/pull/63796) * Added Task Run execute timeout checker by introducing an FE configuration item `task_runs_timeout_second`, and refined cancellation logics for overdue runs. [#63842](https://github.com/StarRocks/starrocks/pull/63842) * Ensured `REFRESH MATERIALIZED VIEW ... FORCE` always refreshes target partitions (even in inconsistent or corrupted cases). [#63844](https://github.com/StarRocks/starrocks/pull/63844) ##### Bug Fixes[​](#bug-fixes-12 "Direct link to Bug Fixes") The following issues have been fixed: * An exception when parsing the Nullable (Decimal) type of ClickHouse. [#64195](https://github.com/StarRocks/starrocks/pull/64195) * An issue with tablet migration and Primary Key index lookup concurrency. [#64164](https://github.com/StarRocks/starrocks/pull/64164) * Lack of FINISHED status in materialized view refresh. [#64191](https://github.com/StarRocks/starrocks/pull/64191) * Schema Change Publish does not retry in shared-data clusters. [#64093](https://github.com/StarRocks/starrocks/pull/64093) * Wrong row count statistics on Primary Key tables in Data Lake. [#64007](https://github.com/StarRocks/starrocks/pull/64007) * When tablet creation times out in shared-data clusters, node information cannot be returned. [#63963](https://github.com/StarRocks/starrocks/pull/63963) * Corrupted Lake DataCache cannot be cleared. [#63182](https://github.com/StarRocks/starrocks/pull/63182) * Window function with IGNORE NULLS flags can not be consolidated with its counterpart without iIGNORE NULLS flag. [#63958](https://github.com/StarRocks/starrocks/pull/63958) * Table compaction cannot be scheduled again after FE restart if the compaction was previously aborted. [#63881](https://github.com/StarRocks/starrocks/pull/63881) * Tasks fail to be scheduled if FE restarts frequently. [#63966](https://github.com/StarRocks/starrocks/pull/63966) * An issue with GCS error codes. [#64066](https://github.com/StarRocks/starrocks/pull/64066) * Instability issue with StarMgr gRPC executor. [#63828](https://github.com/StarRocks/starrocks/pull/63828) * Deadlock when creating an exclusive work group. [#63893](https://github.com/StarRocks/starrocks/pull/63893) * Cache for Iceberg tables is not properly invalidated. [#63971](https://github.com/StarRocks/starrocks/pull/63971) * Wrong results for sorted aggregation in shared-data clusters. [#63849](https://github.com/StarRocks/starrocks/pull/63849) * ASAN error in `PartitionedSpillerWriter::_remove_partition`. [#63903](https://github.com/StarRocks/starrocks/pull/63903) * BE crash when failing to get splits from morsel queue. [#62753](https://github.com/StarRocks/starrocks/pull/62753) * A bug with aggregate push-down type cast in materialized view rewrite. [#63875](https://github.com/StarRocks/starrocks/pull/63875) * NPE when removing expired load jobs in FE. [#63820](https://github.com/StarRocks/starrocks/pull/63820) * Partitioned Spill crash when removing partitions. [#63825](https://github.com/StarRocks/starrocks/pull/63825) * Materialized view rewrite throws `IllegalStateException` under certain plans. [#63655](https://github.com/StarRocks/starrocks/pull/63655) * NPE when creating a partitioned materialized view. [#63830](https://github.com/StarRocks/starrocks/pull/63830) #### 3.5.7[​](#357 "Direct link to 3.5.7") Release date: October 21, 2025 ##### Improvements[​](#improvements-13 "Direct link to Improvements") * Improved memory statistics accuracy for Scan operators by introducing retry backoff under heavy memory contention scenarios. [#63788](https://github.com/StarRocks/starrocks/pull/63788) * Optimized materialized view bucketing inference by leveraging existing tablet distribution to prevent excessive bucket creation. [#63367](https://github.com/StarRocks/starrocks/pull/63367) * Revised the Iceberg table caching mechanism to enhance consistency and reduce cache invalidation risks during frequent metadata updates. [#63388](https://github.com/StarRocks/starrocks/pull/63388) * Added the `querySource` field to `QueryDetail` and `AuditEvent` for better traceability of query origins across APIs and schedulers. [#63480](https://github.com/StarRocks/starrocks/pull/63480) * Enhanced Persistent Index diagnostics by printing detailed context when duplicate keys are detected in MemTable writes. [#63560](https://github.com/StarRocks/starrocks/pull/63560) * Reduced lock contention in materialized view operations by refining lock granularity and sequencing in concurrent scenarios. [#63481](https://github.com/StarRocks/starrocks/pull/63481) ##### Bug Fixes[​](#bug-fixes-13 "Direct link to Bug Fixes") The following issues have been fixed: * Materialized view rewrite failures caused by type mismatch. [#63659](https://github.com/StarRocks/starrocks/pull/63659) * `regexp_extract_all` has wrong behavior and lacks support for `pos=0`. [#63626](https://github.com/StarRocks/starrocks/pull/63626) * Degraded scan performance caused by the profitless simplification of CASE WHEN with complex functions. [#63732](https://github.com/StarRocks/starrocks/pull/63732) * Incorrect DCG data reading when partial updates switch from column mode to row mode. [#61529](https://github.com/StarRocks/starrocks/pull/61529) * A potential deadlock during initialization of `ExceptionStackContext`. [#63776](https://github.com/StarRocks/starrocks/pull/63776) * Crashes in Parquet numeric conversion for ARM architecture machines. [#63294](https://github.com/StarRocks/starrocks/pull/63294) * An issue caused by the aggregate intermediate type uses `ARRAY`. [#63371](https://github.com/StarRocks/starrocks/pull/63371) * Stability issue caused by incorrect overflow detection when casting LARGEINT to DECIMAL128 at sign-edge cases (for example, INT128\_MIN) [#63559](https://github.com/StarRocks/starrocks/pull/63559) * LZ4 compression and decompression errors cannot be perceived. [#63629](https://github.com/StarRocks/starrocks/pull/63629) * `ClassCastException` when querying tables partitioned by `FROM_UNIXTIME` on INT-type columns. [#63684](https://github.com/StarRocks/starrocks/pull/63684) * Tablets cannot be repaired after a balance-triggered migration when the only valid source replica is marked `DECOMMISSION`. [#62942](https://github.com/StarRocks/starrocks/pull/62942) * Profiles lost SQL statements and Planner Trace when the PREPARE statement is used. [#63519](https://github.com/StarRocks/starrocks/pull/63519) * The `extract_number`, `extract_bool`, and `extract_string` functions are not exception-safe. [#63575](https://github.com/StarRocks/starrocks/pull/63575) * Shutdown tablets cannot be garbage-collected properly. [#63595](https://github.com/StarRocks/starrocks/pull/63595) * Profiles showing SQL as `omit` for returns of the PREPARE/EXECUTE statements. [#62988](https://github.com/StarRocks/starrocks/pull/62988) * `date_trunc` partition pruning with combined predicates that mistakenly produced EMPTYSET. [#63464](https://github.com/StarRocks/starrocks/pull/63464) * Crashes in release builds due to the CHECK in NullableColumn. [#63553](https://github.com/StarRocks/starrocks/pull/63553) #### 3.5.6[​](#356 "Direct link to 3.5.6") Release date: September 22, 2025 ##### Improvements[​](#improvements-14 "Direct link to Improvements") * A decommissioned BE will be forcibly dropped when all its tablets are in the recycle bin, to avoid the decommission being blocked by those tablets. [#62781](https://github.com/StarRocks/starrocks/pull/62781) * Vacuum metrics will be updated when Vacuum succeeds. [#62540](https://github.com/StarRocks/starrocks/pull/62540) * Added thread pool metrics to the fragment instance execution state report, including active threads, queue count, and running threads. [#63067](https://github.com/StarRocks/starrocks/pull/63067) * Supports S3 path-style access in shared-data clusters to improve compatibility with MinIO and other S3-compatible storage systems. You can enable this feature by setting `aws.s3.enable_path_style_access` to `true` when creating a storage volume. [#62591](https://github.com/StarRocks/starrocks/pull/62591) * Supports resetting the starting point of the AUTO\_INCREMENT value via `ALTER TABLE`` `` AUTO_INCREMENT`` = 10000;`. [#62767](https://github.com/StarRocks/starrocks/pull/62767) * Supports using Distinguished Name (DN) in Group Provider for group matching, improving the user group solution for LDAP/Microsoft Active Directory environments. [#62711](https://github.com/StarRocks/starrocks/pull/62711) * Supports Azure Workload Identity authentication for Azure Data Lake Storage Gen2. [#62754](https://github.com/StarRocks/starrocks/pull/62754) * Added transaction error messages to the `information_schema.``loads` view to aid failure diagnosis. [#61364](https://github.com/StarRocks/starrocks/pull/61364) * Supports reusing common expressions for complex CASE WHEN expressions in Scan predicates to reduce repetitive computation. [#62779](https://github.com/StarRocks/starrocks/pull/62779) * Uses the REFRESH (instead of ALTER) privilege on the materialized view to execute REFRESH statements. [#62636](https://github.com/StarRocks/starrocks/pull/62636) * Disabled low-cardinality optimization on Lake tables by default to avoid potential issues. [#62586](https://github.com/StarRocks/starrocks/pull/62586) * Enabled tablet balancing between workers by default in shared-data clusters. [#62661](https://github.com/StarRocks/starrocks/pull/62661) * Supports reusing expressions in outer-join WHERE predicates to reduce repetitive computation. [#62139](https://github.com/StarRocks/starrocks/pull/62139) * Added Clone metrics in FE. [#62421](https://github.com/StarRocks/starrocks/pull/62421) * Added Clone metrics in BE. [#62479](https://github.com/StarRocks/starrocks/pull/62479) * Added an FE configuration item `enable_statistic_cache_refresh_after_write` to disable statistics-cache lazy refresh by default. [#62518](https://github.com/StarRocks/starrocks/pull/62518) * Masked credential information in SUBMIT TASK for better security. [#62311](https://github.com/StarRocks/starrocks/pull/62311) * `json_extract` in the Trino dialect returns a JSON type. [#59718](https://github.com/StarRocks/starrocks/pull/59718) * Supports ARRAY type in `null_or_empty`. [#62207](https://github.com/StarRocks/starrocks/pull/62207) * Adjusted the size limit for the Iceberg manifest cache. [#61966](https://github.com/StarRocks/starrocks/pull/61966) * Added a remote file-cache limit for Hive. [#62288](https://github.com/StarRocks/starrocks/pull/62288) ##### Bug Fixes[​](#bug-fixes-14 "Direct link to Bug Fixes") The following issues have been fixed: * Secondary replicas hang indefinitely due to negative timeout values, which cause incorrect timestamp comparisons. [#62805](https://github.com/StarRocks/starrocks/pull/62805) * PublishTask may be blocked when TransactionState is REPLICATION. [#61664](https://github.com/StarRocks/starrocks/pull/61664) * Incorrect repair mechanism for Hive tables that have been dropped and recreated during materialized view refresh. [#63072](https://github.com/StarRocks/starrocks/pull/63072) * Incorrect execution plans were generated after the materialized view aggregation push‑down rewrite. [#63060](https://github.com/StarRocks/starrocks/pull/63060) * ANALYZE PROFILE failures caused by PlanTuningGuide producing unrecognized strings (null explainString) in the query profiles. [#63024](https://github.com/StarRocks/starrocks/pull/63024) * Inappropriate return type of `hour_from_unixtime` and incorrect rewrite rule of `CAST`. [#63006](https://github.com/StarRocks/starrocks/pull/63006) * NPE in Iceberg manifest cache under data races. [#63043](https://github.com/StarRocks/starrocks/pull/63043) * Shared-data clusters lack support for colocation in materialized views. [#62941](https://github.com/StarRocks/starrocks/pull/62941) * Iceberg table Scan Exception during Scan Range deployment.[ #62994](https://github.com/StarRocks/starrocks/pull/62994) * Incorrect execution plans were generated for view-based rewrite. [#62918](https://github.com/StarRocks/starrocks/pull/62918) * Errors and disrupted tasks due to Compute Nodes are not gracefully shut down on exit. [#62916](https://github.com/StarRocks/starrocks/pull/62916) * NPE when Stream Load execution status updates. [#62921](https://github.com/StarRocks/starrocks/pull/62921) * An issue with statistics when the column name and the name in the PARTITION BY clause differ in case. [#62953](https://github.com/StarRocks/starrocks/pull/62953) * Wrong results are returned when the `LEAST` function is used as a predicate. [#62826](https://github.com/StarRocks/starrocks/pull/62826) * Invalid ProjectOperator above the table-pruning frontier CTEConsumer. [#62914](https://github.com/StarRocks/starrocks/pull/62914) * Redundant replica handling after Clone. [#62542](https://github.com/StarRocks/starrocks/pull/62542) * Failed to collect Stream Load profiles. [#62802](https://github.com/StarRocks/starrocks/pull/62802) * Ineffective disk rebalancing caused by improper BE selection. [#62776](https://github.com/StarRocks/starrocks/pull/62776) * A potential NPE cra